--- title: "Training and search policy in robots.txt | Agent Ready Crawlers" description: "How many top sites refuse training crawlers but keep AI search open, how many refuse both, and how many refuse neither. Crawl of 2026-11-04." canonical: https://agentreadycrawlers.com/block-rates/training-and-search site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Block rates" --- # Training and search policy in robots.txt | Agent Ready Crawlers > How many top sites refuse training crawlers but keep AI search open, how many refuse both, and how many refuse neither. Crawl of 2026-11-04. Source: https://agentreadycrawlers.com/block-rates/training-and-search Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Training and search policy in robots.txt In short 11.1 percent of 5,938 top-10,000 sites refuse a training crawler and no AI search crawler, 11.4 percent refuse both, and 77.3 percent refuse neither. Only 12 sites refuse search alone. The groups are six training tokens and three AI search tokens taken from the Crawler Index. Crawl of 13,071 domains, 2026-11-04. Operators now let a site refuse training and still be found by AI search. The Block Rate Report counts refusals one token at a time; this page reads each site as a whole and asks which combination it chose. Training means GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. AI search means OAI-SearchBot, Claude-SearchBot and PerplexityBot. A site counts as refusing a token when RFC 9309 matching disallows the root for it, whether it named the token or a \* group refuses everything. ## The four combinations Refuses neitherRefuses training onlyRefuses training and searchRefuses search only Share of sites with a parsed robots.txt, crawl of 2026-11-04. Segments narrower than a few percent carry no label; the table has every number. | Combination | Top 10,000 | Share | All crawled | Share | | --- | --- | --- | --- | --- | | Refuses neither | 4,590 | 77.3% | 7,128 | 82.7% | | Refuses training only | 660 | 11.1% | 792 | 9.2% | | Refuses training and search | 676 | 11.4% | 692 | 8.0% | | Refuses search only | 12 | 0.2% | 12 | 0.1% | [Download the table as CSV](https://agentreadycrawlers.com/data/policy-mix-2026-11-04.csv). ## Refusing every training token Top-10,000 sites by the number of the six training tokens refused. The two ends hold most sites: 77.5 percent refuse none and 11.0 percent refuse all six. 297 top-10,000 sites, 5.0 percent, refuse all six training tokens and none of the three search tokens: the recipe in [how to allow search and refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/). 292 sites, 4.9 percent, refuse all nine. Of those, 168 name none of the nine, so their refusal is a blanket group and not a decision about AI. ## What refusing search costs 688 sites in the top 10,000 refuse at least one of the three search tokens. The operators say what that does. OpenAI: Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. ## Limits The counts read robots.txt and nothing else. A site that refuses a token here may still admit the agent at the WAF, and one that allows it may refuse it there. The six training tokens and the three search tokens are our grouping from the Index purposes; Amazonbot, Bytespider and other tokens that the Index files under training or Other are measured in the [Block Rate Report](https://agentreadycrawlers.com/block-rates/) and left out of this mix. Only the top-10,000 population has an upper rank bound, and the all-crawled population includes the Ora leaderboard's sites. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Allow search, refuse training (this site)](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) read 2026-11-04 Last reviewed 2026-11-05