Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training and search policy in robots.txt

In short

11.1 percent of 5,938 top-10,000 sites refuse a training crawler and no AI search crawler, 11.4 percent refuse both, and 77.3 percent refuse neither. Only 12 sites refuse search alone. The groups are six training tokens and three AI search tokens taken from the Crawler Index. Crawl of 13,071 domains, 2026-11-04.

Operators now let a site refuse training and still be found by AI search. The Block Rate Report counts refusals one token at a time; this page reads each site as a whole and asks which combination it chose. Training means GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. AI search means OAI-SearchBot, Claude-SearchBot and PerplexityBot. A site counts as refusing a token when RFC 9309 matching disallows the root for it, whether it named the token or a * group refuses everything.

The four combinations

How sites combine training and AI search refusals in robots.txtTop 10,000: refuses neither 77.3 percent, refuses training only 11.1 percent, refuses training and search 11.4 percent, refuses search only 0.2 percent.Top 10,000 (5,938 sites)77.3%11.1%11.4%All crawled (8,624 sites)82.7%How sites combine training and AI search refusals in robots.txtTop 10,000: refuses neither 77.3 percent, refuses training only 11.1 percent, refuses training and search 11.4 percent, refuses search only 0.2 percent.Top 10,000 (5,938 sites)77.3%11.1%11.4%All crawled (8,624 sites)82.7%

Refuses neitherRefuses training onlyRefuses training and searchRefuses search only

Share of sites with a parsed robots.txt, crawl of 2026-11-04. Segments narrower than a few percent carry no label; the table has every number.
The chart as a table: sites and share of sites.
CombinationTop 10,000ShareAll crawledShare
Refuses neither4,59077.3%7,12882.7%
Refuses training only66011.1%7929.2%
Refuses training and search67611.4%6928.0%
Refuses search only120.2%120.1%

Download the table as CSV.

Refusing every training token

Share of top-10,000 sites by how many of the six training tokens they refuseBars from 0 to 6 tokens refused: 0 of 6 refused 77.5 percent, 1 of 6 refused 2.7 percent, 2 of 6 refused 2.0 percent, 3 of 6 refused 1.2 percent, 4 of 6 refused 2.3 percent, 5 of 6 refused 3.3 percent, 6 of 6 refused 11.0 percent.0%20%40%60%80%100%0 of 6 refused77.5%1 of 6 refused2.7%2 of 6 refused2.0%3 of 6 refused1.2%4 of 6 refused2.3%5 of 6 refused3.3%6 of 6 refused11.0%Share of top-10,000 sites by how many of the six training tokens they refuseBars from 0 to 6 tokens refused: 0 of 6 refused 77.5 percent, 1 of 6 refused 2.7 percent, 2 of 6 refused 2.0 percent, 3 of 6 refused 1.2 percent, 4 of 6 refused 2.3 percent, 5 of 6 refused 3.3 percent, 6 of 6 refused 11.0 percent.0%20%40%60%80%100%0 of 6 refused77.5%1 of 6 refused2.7%2 of 6 refused2.0%3 of 6 refused1.2%4 of 6 refused2.3%5 of 6 refused3.3%6 of 6 refused11.0%
Top-10,000 sites by the number of the six training tokens refused. The two ends hold most sites: 77.5 percent refuse none and 11.0 percent refuse all six.

297 top-10,000 sites, 5.0 percent, refuse all six training tokens and none of the three search tokens: the recipe in how to allow search and refuse training. 292 sites, 4.9 percent, refuse all nine. Of those, 168 name none of the nine, so their refusal is a blanket group and not a decision about AI.

What refusing search costs

688 sites in the top 10,000 refuse at least one of the three search tokens. The operators say what that does. OpenAI: Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results.

Limits

The counts read robots.txt and nothing else. A site that refuses a token here may still admit the agent at the WAF, and one that allows it may refuse it there. The six training tokens and the three search tokens are our grouping from the Index purposes; Amazonbot, Bytespider and other tokens that the Index files under training or Other are measured in the Block Rate Report and left out of this mix. Only the top-10,000 population has an upper rank bound, and the all-crawled population includes the Ora leaderboard's sites.

Sources

Last reviewed 2026-11-05