--- title: "Which AI bots to allow, by type of site | Agent Ready Crawlers" description: "Which AI bots to allow for a publisher, a docs site, a store, a logged-in app or a closed site, with a robots.txt generated from the Crawler Index." canonical: https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Guides" --- # Which AI bots to allow, by type of site | Agent Ready Crawlers > Which AI bots to allow for a publisher, a docs site, a store, a logged-in app or a closed site, with a robots.txt generated from the Crawler Index. Source: https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Which AI bots to allow, by type of site In short Five kinds of site need different answers on four purposes: training, search, live fetches and task agents. Each gets a policy table and a robots.txt generated from the Crawler Index by the same builder as the Builder page. In the crawl, 11.1 percent of top-10,000 sites refuse training and keep AI search open, and 77.3 percent refuse neither. Every crawler in the Crawler Index has one of six purposes, and operators now let a site treat them separately. The choice that matters is which purposes pay you back. This page is our reading of that trade for five kinds of site; the numbers and the operator statements are sourced, the recommendation is a judgement and says so. ## The policy for each type | Site type | Training | Search index | Live answer fetch | Task agent | Shopping agent | | --- | --- | --- | --- | --- | --- | | Publisher selling ads or subscriptions | Refuse | Allow | Allow | Allow | Allow | | Documentation or developer platform | Allow | Allow | Allow | Allow | Allow | | Online store | Refuse | Allow | Allow | Allow | Allow | | Logged-in app or regulated service | Refuse | Allow | Allow | Refuse | Refuse | | Site that wants no AI use | Refuse | Refuse | Refuse | Refuse | Refuse | ## Why each type gets its choices Publisher selling ads or subscriptions Refuse training and keep search and fetches open. OpenAI treats OAI-SearchBot and GPTBot as independent settings, and says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can still appear as navigational links. Cloudflare reports (2026-09-15) that fewer than 1 percent of its sites block search bots while 17 percent block training. The crawl agrees on the shape: 660 of 5,938 top-10,000 sites, 11.1 percent, refuse at least one of six training tokens and none of three AI search tokens. Documentation or developer platform Allow everything, because agents read docs. Ora's lab (vendor claim, 2026-10-19) found agents reached docs pages in 83 percent of traced runs and the home page in 69 percent. Refusing training is the only choice with a cost here: a model that never saw your docs cannot recall them. That is an inference, not a measurement, and a licensing concern can outweigh it. Online store The same file as a publisher, with the work moved to the WAF. 3 shopping rows exist in the Index and 0 publish a token, so decide signed commerce agents by signature: [how to let Visa TAP and Mastercard Agent Pay agents through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/). Logged-in app or regulated service Refuse training, task agents and shopping agents, and keep every private path behind authentication, since robots.txt states a preference and enforces nothing. The Index has 20 task-agent rows and 1 of them publishes a token, so the refusal in the file can address only that one. [The agentic browsers](https://agentreadycrawlers.com/crawlers/agentic-browsers/) table shows how the rest identify. Site that wants no AI use Refuse every AI purpose and accept the cost: you leave the search products too. 292 top-10,000 sites, 4.9 percent, refuse all nine training and AI search tokens the crawl measured, and 168 of those name none of the nine, so a blanket `Disallow: /` in the \* group is doing the work. ## The files Each file is the output of the [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) for the choices above, with Allow lines for every other purpose. Types that share a file share a block. The files name only agents that publish a token, so they cannot address browsing agents that run in a visitor's own browser. **Publisher selling ads or subscriptions and Online store**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` **Documentation or developer platform**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: allowed User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Allow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` **Logged-in app or regulated service**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: refused User-agent: Claude-Agent Disallow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` **Site that wants no AI use**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: refused User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Disallow: / # Live answer fetch: refused User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Disallow: / # Task agents: refused User-agent: Claude-Agent Disallow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` ## What no file can do A robots.txt line is a request. OpenAI and Perplexity say the fetchers a person triggers may ignore it, and a crawler that does not identify itself never reads the group you wrote for it. Pair the file with a check of what your server answers: [how to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Ora's lab: AgentReady launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Training and search policy (this site)](https://agentreadycrawlers.com/block-rates/training-and-search/) read 2026-11-04 Last reviewed 2026-11-05