Introduction

Most SEO teams can tell you exactly how Googlebot behaves on their site. Crawl budget, index coverage, render queue — it’s second nature. Ask the same team which AI crawlers touched their site last week, and the answer is usually a shrug.

That gap matters more every month. AI-referred traffic is still a small slice of most sites’ totals, but it’s the fastest-growing slice, and it runs on a completely different set of rules than traditional search.

The problem is that “AI bot” isn’t one thing. Every major AI company runs multiple crawlers with different jobs, different consequences, and different rules. Understanding those differences is essential before making changes to your robots.txt file.

Stop Treating “AI Bot” as One Category

Every major AI company now runs at least two, and often three, distinct crawlers, each tied to a different product function:

• A training crawler that harvests content to improve future model versions.
• A search/indexing crawler that builds a retrieval index so the AI can cite live sources when answering questions.
• A user-triggered fetcher that grabs a specific page because a person asked the assistant to read it.

Blocking one doesn’t block the others. Each user-agent requires its own rule in robots.txt, and each decision comes with a different tradeoff.

Meet the Crawlers That Actually Matter

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

• GPTBot collects web content that may be used to train future versions of OpenAI’s models. Blocking it only opts your content out of training.
• OAI-SearchBot powers ChatGPT’s live search and determines whether your pages can be surfaced and cited in search answers.
• ChatGPT-User fetches a specific page when a user asks ChatGPT to access it. It performs a one-time fetch rather than a full crawl.

OpenAI allows publishers to block GPTBot while keeping OAI-SearchBot available for citations.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic follows the same three-tier structure.

• ClaudeBot handles training collection.
• Claude-SearchBot indexes content for Claude’s search-based answers.
• Claude-User retrieves a page when a user requests it.

Anthropic states that all three crawlers respect robots.txt. Blocking Claude-SearchBot may reduce your site’s visibility in Claude’s search results, while blocking ClaudeBot only affects model training.

Perplexity: PerplexityBot and Perplexity-User

Perplexity’s declared crawlers also separate training and retrieval. However, it has faced public criticism regarding crawler behavior.

In August 2025, Cloudflare reported that Perplexity was accessing websites that had blocked its bots through robots.txt and firewall rules. Cloudflare claimed it detected undeclared crawler traffic using browser-like user-agent strings on test domains.

Perplexity disputed those claims, arguing that fetching pages in response to user requests is different from automated crawling. The disagreement remains unresolved because robots.txt is a voluntary standard with no formal enforcement.

For most AI companies, robots.txt is generally a reliable control. For Perplexity, site owners who require stronger protection may also consider IP-based blocking or web application firewall rules.

Check Your Logs Before You Touch Robots.txt

Before changing any robots.txt directives, review your server logs.

• Pull recent access logs from your web server, CDN, or Cloudflare.
• Search for each crawler individually:
  • GPTBot
  • OAI-SearchBot
  • ChatGPT-User
  • ClaudeBot
  • Claude-SearchBot
  • Claude-User
  • PerplexityBot
  • Perplexity-User
• Review crawl frequency and requested paths.
• Verify crawler IP addresses using published IP ranges where available.
• Watch for unknown browser-style user-agent strings making systematic requests.

This audit often changes the decision. Many websites discover that most AI traffic comes from search and citation bots rather than training crawlers.

A Robots.txt Decision Framework

Decide your robots.txt policy based on your goals.

If your goal is AI visibility

Allow:

• OAI-SearchBot
• Claude-SearchBot
• ChatGPT-User
• Claude-User

Blocking these reduces your visibility in AI-generated answers and citations.

If your goal is opting out of model training

Block:

• GPTBot
• ClaudeBot

Keep the search bots enabled if you still want AI citation visibility.

If your goal is blocking AI completely

Block every declared user-agent for that company.

Keep in mind that this only stops crawlers that respect robots.txt. It cannot guarantee protection from crawlers that ignore those rules.

Regardless of your AI policy, continue blocking sensitive areas such as admin pages, account pages, checkout flows, and internal search results using a wildcard User-agent: * rule.

After making changes, allow time for search bots to process the update, and review your robots.txt after CMS migrations or website redesigns to ensure the rules haven’t been lost.

Conclusion

The sites getting this right aren’t the ones with the biggest block list or the most permissive one. They’re the ones that review their logs, understand what each crawler does, and create a robots.txt policy that matches their goals for model training, citation visibility, and referral traffic.

As AI search continues to grow, making these decisions deliberately is far better than relying on default settings or copy-pasting someone else’s robots.txt file.