Which AI bots to allow, by type of site

In short

Five kinds of site need different answers on four purposes: training, search, live fetches and task agents. Each gets a policy table and a robots.txt generated from the Crawler Index by the same builder as the Builder page. In the crawl, 11.1 percent of top-10,000 sites refuse training and keep AI search open, and 77.3 percent refuse neither.

Every crawler in the Crawler Index has one of six purposes, and operators now let a site treat them separately. The choice that matters is which purposes pay you back. This page is our reading of that trade for five kinds of site; the numbers and the operator statements are sourced, the recommendation is a judgement and says so.

The policy for each type

Robots.txt choice per purpose. Shopping agents are shown for completeness; 0 of the 3 shopping rows in the Index publish a token, so a robots.txt line cannot reach them.
Site type	Training	Search index	Live answer fetch	Task agent	Shopping agent
Publisher selling ads or subscriptions	Refuse	Allow	Allow	Allow	Allow
Documentation or developer platform	Allow	Allow	Allow	Allow	Allow
Online store	Refuse	Allow	Allow	Allow	Allow
Logged-in app or regulated service	Refuse	Allow	Allow	Refuse	Refuse
Site that wants no AI use	Refuse	Refuse	Refuse	Refuse	Refuse

Why each type gets its choices

Publisher selling ads or subscriptions: Refuse training and keep search and fetches open. OpenAI treats OAI-SearchBot and GPTBot as independent settings, and says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can still appear as navigational links. Cloudflare reports (2026-09-15) that fewer than 1 percent of its sites block search bots while 17 percent block training. The crawl agrees on the shape: 660 of 5,938 top-10,000 sites, 11.1 percent, refuse at least one of six training tokens and none of three AI search tokens.
Documentation or developer platform: Allow everything, because agents read docs. Ora's lab (vendor claim, 2026-10-19) found agents reached docs pages in 83 percent of traced runs and the home page in 69 percent. Refusing training is the only choice with a cost here: a model that never saw your docs cannot recall them. That is an inference, not a measurement, and a licensing concern can outweigh it.
Online store: The same file as a publisher, with the work moved to the WAF. 3 shopping rows exist in the Index and 0 publish a token, so decide signed commerce agents by signature: how to let Visa TAP and Mastercard Agent Pay agents through.
Logged-in app or regulated service: Refuse training, task agents and shopping agents, and keep every private path behind authentication, since robots.txt states a preference and enforces nothing. The Index has 20 task-agent rows and 1 of them publishes a token, so the refusal in the file can address only that one. The agentic browsers table shows how the rest identify.
Site that wants no AI use: Refuse every AI purpose and accept the cost: you leave the search products too. 292 top-10,000 sites, 4.9 percent, refuse all nine training and AI search tokens the crawl measured, and 168 of those name none of the nine, so a blanket Disallow: / in the * group is doing the work.

The files

Each file is the output of the robots.txt builder for the choices above, with Allow lines for every other purpose. Types that share a file share a block. The files name only agents that publish a token, so they cannot address browsing agents that run in a visitor's own browser.

Publisher selling ads or subscriptions and Online store: 68 lines

# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04.
# Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none.

# Training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Search index: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Storebot-Google
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: DuckDuckBot
User-agent: MistralAI-Index
User-agent: YouBot
User-agent: Timpibot
User-agent: PetalBot
User-agent: ExaSearchBot
Allow: /

# Live answer fetch: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /

# Task agents: allowed
User-agent: Claude-Agent
Allow: /

# Other: allowed
User-agent: OAI-AdsBot
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: MicrosoftPreview
User-agent: meta-externalads
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: FirecrawlAgent
User-agent: OraBot
User-agent: agent-readiness-crawl
Allow: /

# Everything else
User-agent: *
Allow: /

Documentation or developer platform: 68 lines

# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04.
# Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none.

# Training: allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Allow: /

# Search index: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Storebot-Google
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: DuckDuckBot
User-agent: MistralAI-Index
User-agent: YouBot
User-agent: Timpibot
User-agent: PetalBot
User-agent: ExaSearchBot
Allow: /

# Live answer fetch: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /

# Task agents: allowed
User-agent: Claude-Agent
Allow: /

# Other: allowed
User-agent: OAI-AdsBot
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: MicrosoftPreview
User-agent: meta-externalads
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: FirecrawlAgent
User-agent: OraBot
User-agent: agent-readiness-crawl
Allow: /

# Everything else
User-agent: *
Allow: /

Logged-in app or regulated service: 68 lines

# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04.
# Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none.

# Training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Search index: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Storebot-Google
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: DuckDuckBot
User-agent: MistralAI-Index
User-agent: YouBot
User-agent: Timpibot
User-agent: PetalBot
User-agent: ExaSearchBot
Allow: /

# Live answer fetch: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /

# Task agents: refused
User-agent: Claude-Agent
Disallow: /

# Other: allowed
User-agent: OAI-AdsBot
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: MicrosoftPreview
User-agent: meta-externalads
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: FirecrawlAgent
User-agent: OraBot
User-agent: agent-readiness-crawl
Allow: /

# Everything else
User-agent: *
Allow: /

Site that wants no AI use: 68 lines

# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04.
# Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none.

# Training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Search index: refused
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Storebot-Google
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: DuckDuckBot
User-agent: MistralAI-Index
User-agent: YouBot
User-agent: Timpibot
User-agent: PetalBot
User-agent: ExaSearchBot
Disallow: /

# Live answer fetch: refused
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
Disallow: /

# Task agents: refused
User-agent: Claude-Agent
Disallow: /

# Other: allowed
User-agent: OAI-AdsBot
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: MicrosoftPreview
User-agent: meta-externalads
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: FirecrawlAgent
User-agent: OraBot
User-agent: agent-readiness-crawl
Allow: /

# Everything else
User-agent: *
Allow: /

What no file can do

A robots.txt line is a request. OpenAI and Perplexity say the fetchers a person triggers may ignore it, and a crawler that does not identify itself never reads the group you wrote for it. Pair the file with a check of what your server answers: how to test whether AI agents can reach your site.

Sources

Overview of OpenAI Crawlers read 2026-11-04
Have it both ways: stay discoverable in search while disallowing AI training read 2026-11-04
RFC 9309: Robots Exclusion Protocol read 2026-11-04
Ora's lab: AgentReady launch (vendor) read 2026-11-05
Training and search policy (this site) read 2026-11-04

Last reviewed 2026-11-05