AI Crawler Management: 2026 robots.txt Policy That Works

GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Applebot-Extended, Google-Extended, CCBot, Meta-ExternalAgent, Amazonbot, cohere-ai — the AI crawler list keeps growing, and on a growing share of sites the combined bot traffic from these user-agents now exceeds Googlebot. Most robots.txt files do not mention a single one of them. That gap is the post.

The reason it matters: an AI bot is not a search engine. It can scrape for training, fetch live for retrieval, or both, and the user-agent string alone does not tell you which. Allowing the wrong bot burns crawl budget, leaks unpublished drafts, and feeds your competitors’ models. Blocking the wrong one makes you invisible inside ChatGPT, Perplexity, and Gemini — surfaces an increasing share of discovery now happens on. A robots.txt policy is the only file that decides both, and almost nobody audits it.

This is the framework we use, the policy we ship, and the evidence we look at to decide which AI bots deserve which treatment on a given site. The honest summary up front: there is no universal right answer, but there is a working decision tree, and the robots.txt you wrote in 2022 is no longer it.

Key takeaways

• AI crawler traffic now rivals search-engine traffic on many sites. Content-heavy blogs see 30–60% of crawler hits from non-search bots.
• One user-agent often masks two products. OpenAI’s GPTBot crawls for training; OAI-SearchBot fetches live for retrieval. Same vendor, opposite intent.
• Google-Extended is not Googlebot. Blocking it does not affect Google Search rankings — only Gemini.
• Default policy is wrong on both sides. Blanket block cuts you out of every AI assistant; blanket allow feeds training you may not want.
• The framework is a 5-tier ladder: training-only blocked, retrieval-only allowed, search-affiliated allowed with citation rules, scraper-class blocked at edge, unidentified rate-limited.
• Trust logs, not press releases. Most “AI search” bots ship real traffic through different user-agents.

The new bot landscape

In 2022 a robots.txt named three or four bots worth caring about. In 2026 the file needs to handle a bot census that reads like a tech-conference badge list, and the rules are not written down in one place.

OpenAI ships three:

• GPTBot — crawls for training.
• OAI-SearchBot — fetches live for ChatGPT search results.
• ChatGPT-User — fires when a ChatGPT user clicks through.

Same vendor, three intents. Blocking GPTBot but allowing OAI-SearchBot keeps you out of training while staying in ChatGPT search. Blocking all three cuts you out of both.

Anthropic follows the same pattern: ClaudeBot for training, Claude-Web for retrieval, anthropic-ai legacy label. Perplexity ships PerplexityBot for retrieval (no training today) and Perplexity-User for clicks. Google splits Google-Extended for Gemini opt-out without affecting Search. Apple ships Applebot-Extended for the same reason. Meta uses meta-externalagent; Amazon uses Amazonbot.

The pattern is consistent: the search-indexing crawler stays under one brand, and a separate -Extended or training-specific user-agent handles the AI-training use case. Knowing which is which is the entire policy.

What each AI bot actually does with your content

Marketing copy from these vendors is consistent in tone and inconsistent in mechanism. The practical summary:

• GPTBot — crawls for OpenAI training. Honours robots.txt.
• OAI-SearchBot — fetches live for ChatGPT search. Independent of training.
• ChatGPT-User — fires only when a ChatGPT user clicks a citation. Treat as a real user.
• ClaudeBot — crawls for Anthropic training.
• Claude-Web — live retrieval for Claude search.
• PerplexityBot — live retrieval and citation; no training as of 2026.
• Google-Extended — controls Gemini training. Disabling does not affect Google Search rankings. The most misunderstood user-agent.
• Applebot-Extended — controls Apple Intelligence training.
• CCBot — Common Crawl. Used by most major trainers. Honoured by everyone downstream.
• meta-externalagent — Meta AI training and retrieval.
• Amazonbot — Amazon’s AI surfaces. Mixed role.

A site that wants to be cited in AI answers but not used as training fuel blocks GPTBot/ClaudeBot/meta-externalagent/CCBot and allows OAI-SearchBot/Claude-Web/PerplexityBot. The opposite policy is pointless in practice.

The 5-tier policy framework

Four questions, in order:

1. Does this bot train a model on my content? Most sites say no to closed-source training and yes to retrieval.
2. Does this bot retrieve live content and cite me? Citation is the AI-era equivalent of ranking.
3. Is this bot a search engine in disguise? Honour indexability signals, verify before trusting.
4. Is this bot unidentified or scraper-class? Block at the edge.

The five tiers:

• Tier 1 — Search-engine bots. Full allow with rich-snippet signals.
• Tier 2 — AI retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot, Perplexity-User, DuckAssistBot). Full allow. Drives AI-era discovery.
• Tier 3 — AI training bots (GPTBot, ClaudeBot, meta-externalagent, cohere-training-data-crawler, Google-Extended, Applebot-Extended). Commercial choice.
• Tier 4 — Aggregator crawlers (CCBot). Treat as Tier 3; its data ends up in dozens of models.
• Tier 5 — Unknown, scraper-class bots. Block at edge with rate limits and DNS verification.

Every bot is either a search engine, a citation source, a training pipeline, or noise. Decide which, then route accordingly.

The robots.txt we ship

Here is the policy SEOKit24 runs in production, abridged from the live file. The full thing lists every named bot individually so that any future bot not yet on the list falls into the catch-all block — the inverse of the typical “allow all by default” stance.

# AI crawlers — explicitly allowed for training + retrieval (cite-with-link)
# These bots power ChatGPT search, Perplexity, Claude, Gemini, Bing Chat.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

# Default rule for any other crawler
User-agent: *
Allow: /

Three things to notice:

• No blanket User-agent: * at the top. The default rule sits at the bottom, after the named bots, so the explicit Allow entries take precedence by specificity (a rule most engines honour — but not all, which is why tier 5 still needs the edge layer).
• No Disallow other than internal paths. A blanket block is one mistake away from making the site invisible to a newly announced bot that has not been added to the named list yet.
• GPTBot and OAI-SearchBot are separate. Treating them as one is the single most common OpenAI mistake in robots.txt files.

The reasoning for the allow stance: this site is a free toolkit. Citation in an AI answer drives direct traffic and earns attribution; training-fuel use is acceptable cost. A publisher with paywalled content or proprietary data would flip tier 3 and 4 to a block. A publisher whose customers come through AI citations would lean the other way. The policy is the same shape, the direction is yours.

Common robots.txt mistakes that quietly cut you out of AI search

Treating all AI bots as one user-agent

The most common error. A team adds User-agent: GPTBot and assumes they have covered OpenAI — they have not, because OAI-SearchBot and ChatGPT-User are separate products with separate policies.

Adding Disallow: / under the wrong user-agent

A User-agent: * with Disallow: / blocks every crawler including future AI bots not yet named. Teams add it thinking they are blocking scrapers; they are also blocking their own AI-search citation.

Assuming Google-Extended affects Google Search

It does not. Google-Extended is a separate opt-out for Gemini training. Disabling it changes nothing about whether your pages rank in Google Search.

Confusing polite request with enforcement

A bot with a research-paper agenda and no commercial incentive to behave can ignore robots.txt. CCBot honours it; hundreds of smaller AI scrapers do not. robots.txt is a request, not a guard. Anything sensitive needs an edge-layer rule.

Using Noindex where they meant Disallow

Noindex is a page-level signal telling Google not to index a URL. Disallow is a robots-level signal telling compliant bots not to fetch it. They are not interchangeable.

Robots.txt is the file that decides which bots see your site. The AI era just made the bot list longer than anyone expected.

How to verify what AI bots actually hit your site

The press release says one thing. The logs say another.

1. Pull 14 days of access logs. A spreadsheet, a pivot table, a per-user-agent sort.
2. Filter by AI bot user-agents. Add new ones as they appear; undocumented bots go into an “unverified” bucket.
3. Verify IPs the same way as Googlebot. Forward and reverse DNS. Real GPTBot resolves to *.openai.com; real ClaudeBot resolves to *.anthropic.com. Anything else is spoofed.
4. Group by URL. AI bots disproportionately hit your most recent and most-linked content.
5. Compare against your policy. If you blocked GPTBot and logs show GPTBot/1.0 at volume, the bot is spoofing or routing around robots.txt.

On most sites, the volume split between training, retrieval, and spoofed scrapers is roughly 40/40/20, with the spoofed 20% doing most of the damage. The fix is at the edge.

When robots.txt is not enough

Three situations require more than a robots.txt:

• High-volume spoofed scrapers. They ignore robots.txt. The fix is at your edge layer or WAF — rate-limit by ASN, block by reverse-DNS failure, challenge by JA3 fingerprint.
• Authenticated content. Anything behind a login is invisible to compliant bots by default, but a leaked URL or a shared session token reaches AI bots through legitimate retrieval paths. robots.txt cannot fix leaked access.
• Content licensing. If you sell syndication or your data carries contractual use restrictions, robots.txt is a starting position, not a contract. The legal layer is your terms of service and your CDN access controls, not the file at /robots.txt.

For everything else, robots.txt is the right tool, and the framework above is the right policy shape.

How to ship a robots.txt change without breaking SEO

The decision to block an AI bot is a business decision; the rollout is a technical one. The order matters.

1. Stage the change. Deploy the new robots.txt to a staging origin first, or serve it via a X-Robots-Tag rewrite at the edge so you can flip back in a minute.
2. Watch the indexing layer. Search Console Performance for 7–14 days; sudden impression drops mean the new policy caught a search-engine bot you did not name.
3. Watch the AI citation layer. Track referrals from chat.openai.com, perplexity.ai, claude.ai, gemini.google.com, copilot.microsoft.com in your analytics. A citation drop is the AI-search equivalent of a ranking drop.
4. Re-verify after every CMS template change. robots.txt rot is real; a CMS migration that rewrites /robots.txt is how sites accidentally get blocked.

The full audit cadence lives in our broader technical SEO framework — the technical SEO guide walks the crawl-to-index pipeline end to end, and the log file analysis playbook shows you how to read what the bots actually did versus what they claim.

The decision most teams get wrong

The most expensive decision in this file is not which bot to block. It is treating robots.txt as a one-time deliverable. New AI bots ship every quarter — OpenAI’s searchbot family grew three-fold in 18 months — and the file you wrote last year is missing entries you have never heard of. The audit cadence for robots.txt should be quarterly, not annual, and every time you do it the work is the same: pull logs, verify the bot list, decide each tier, ship the change, watch the next 14 days.

The second-most expensive decision is treating the policy as a moral question. It is a commercial one. “Should AI be allowed to train on the open web” is a debate worth having. “Should this site be cited when a user asks ChatGPT about SEO audits” is the question robots.txt actually answers, and the answer for most content sites is yes.

Frequently asked questions

Does blocking GPTBot affect ChatGPT search rankings?

No. GPTBot is training, OAI-SearchBot is search. To be cited in ChatGPT search, allow OAI-SearchBot.

Does blocking Google-Extended hurt Google Search rankings?

No. Google-Extended covers Gemini training only. Search rankings are controlled by Googlebot.

Should I block all AI crawlers by default?

Probably not. A blanket User-agent: * block cuts you out of every AI assistant overnight. The economic cost of being uncited in AI answers is already larger than the cost of training inclusion.

How do I know if an AI bot is following my robots.txt?

Verify with reverse DNS. Real GPTBot resolves to *.openai.com; real ClaudeBot to *.anthropic.com. Anything else is spoofed.

Is robots.txt legally enforceable, and how often should I audit it?

Not on its own. Quarterly cadence. Pair with the Search Console hidden reports cadence.

The verdict

The 2026 robots.txt is a longer file than the 2022 one, and the decision is harder. No universal policy; a 5-tier framework, a verification workflow, a quarterly audit cadence. The teams getting this right treat the file as a living document and route answers through logs-not-press-releases.

The bots will keep multiplying. The framework will hold. Verify, route by tier, watch logs, ship quarterly.

Run your robots.txt through the Sitemap & Robots Validator — the file ships with a per-bot check. Then use the SERP Preview tool to verify that robots.txt changes have not inadvertently stripped your pages of their title tags or meta descriptions in Google’s rendered view.