Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Edition 2026-11-04

The Block Rate Report

In short

Of 5,938 top-10,000 sites with a parsed robots.txt, 18.5 percent refuse GPTBot at the root and 17.6 percent refuse ClaudeBot, against 2.8 percent for Googlebot. 17.2 percent of their home pages answered 401 or 403 to the crawl. Crawl of 13,071 domains, 4 November 2026.

8,624
robots.txt files parsed, of 13,071 domains crawled
18.5%
of 5,938 top-10,000 sites refuse GPTBot
17.6%
refuse ClaudeBot
17.2%
of top-10,000 home pages answered 401 or 403

By token

Share of top-10,000 sites that disallow each AI crawler token at the rootBars for 32 tokens, highest CCBot at 19.4 percent.0%5%10%15%20%25%CCBot19.4%Bytespider19.2%GPTBot18.5%ClaudeBot17.6%Google-Extended16.0%Meta-ExternalAgent16.0%Diffbot15.7%omgili15.7%Amazonbot15.6%anthropic-ai15.5%cohere-ai15.5%Applebot14.8%Applebot-Extended14.8%PetalBot14.1%PerplexityBot11.2%Google-CloudVertexBot10.9%YouBot10.3%FacebookBot10.2%Timpibot9.8%ChatGPT-User9.7%ImagesiftBot9.5%Meta-ExternalFetcher8.5%Perplexity-User8.3%Claude-User8.1%DuckAssistBot8.1%Claude-SearchBot7.7%MistralAI-User7.7%OAI-SearchBot7.7%agent-readiness-crawl*4.2%MicrosoftPreview4.0%Bingbot3.1%Googlebot2.8%Share of top-10,000 sites that disallow each AI crawler token at the rootBars for 32 tokens, highest CCBot at 19.4 percent.0%10%20%CCBot19.4%Bytespider19.2%GPTBot18.5%ClaudeBot17.6%Google-Extended16.0%Meta-ExternalAgent16.0%Diffbot15.7%omgili15.7%Amazonbot15.6%anthropic-ai15.5%cohere-ai15.5%Applebot14.8%Applebot-Extended14.8%PetalBot14.1%PerplexityBot11.2%Google-CloudVertexBot10.9%YouBot10.3%FacebookBot10.2%Timpibot9.8%ChatGPT-User9.7%ImagesiftBot9.5%Meta-ExternalFetcher8.5%Perplexity-User8.3%Claude-User8.1%DuckAssistBot8.1%Claude-SearchBot7.7%MistralAI-User7.7%OAI-SearchBot7.7%agent-readiness-crawl*4.2%MicrosoftPreview4.0%Bingbot3.1%Googlebot2.8%

Site names the tokenRefused only by a * group

Share of top-10,000 sites with a parsed robots.txt that refuse each token at the root. *The crawl's own token, a control: its bar is the wildcard baseline.

CCBot is the most refused token at 19.4 percent, and Googlebot the least refused at 2.8 percent. Across all 13,071 crawled domains every rate is lower: GPTBot 13.9 percent, ClaudeBot 13.1 percent.

The legacy token anthropic-ai, which Anthropic no longer lists on its crawler page, is still refused by 15.5 percent of the top 10,000, close to ClaudeBot at 17.6 percent. Files written for one name keep working for the other only if the site names both.

Refusing a token states a preference. User-triggered fetchers such as ChatGPT-User and Perplexity-User are, by their operators' own statements, allowed to ignore it; see ChatGPT-User.

Every token the crawl checked. Top 10,000: 5,938 sites with a parsed robots.txt. All crawled: 8,624 sites.
TokenRefused, top 10,000NamedVia *SitesRefused, all crawledSites
GPTBot18.5%15.2%3.3%5,93813.9%8,624
OAI-SearchBot7.7%4.4%3.3%5,9385.4%8,624
ChatGPT-User9.7%6.3%3.3%5,9386.9%8,624
ClaudeBot17.6%14.0%3.5%5,93813.1%8,624
Claude-User8.1%4.5%3.6%5,9385.7%8,624
Claude-SearchBot7.7%4.1%3.6%5,9385.5%8,624
anthropic-ai15.5%11.7%3.8%5,93811.6%8,624
PerplexityBot11.2%7.9%3.4%5,9387.9%8,624
Perplexity-User8.3%4.6%3.7%5,9385.9%8,624
Google-Extended16.0%12.6%3.4%5,93812.1%8,624
Googlebot2.8%0.5%2.3%5,9382.1%8,624
Google-CloudVertexBot10.9%7.1%3.8%5,9388.3%8,624
Applebot14.8%11.6%3.1%5,93811.1%8,624
Applebot-Extended14.8%11.6%3.1%5,93811.1%8,624
Bingbot3.1%0.4%2.7%5,9382.2%8,624
MicrosoftPreview4.0%0.0%3.9%5,9382.9%8,624
Amazonbot15.6%12.0%3.6%5,93811.7%8,624
Meta-ExternalAgent16.0%12.3%3.7%5,93812.0%8,624
Meta-ExternalFetcher8.5%4.7%3.8%5,9386.0%8,624
FacebookBot10.2%6.4%3.7%5,9387.3%8,624
Bytespider19.2%15.4%3.8%5,93815.0%8,624
CCBot19.4%15.7%3.7%5,93814.9%8,624
DuckAssistBot8.1%4.3%3.8%5,9385.8%8,624
MistralAI-User7.7%3.8%3.9%5,9385.5%8,624
cohere-ai15.5%11.6%3.8%5,93811.7%8,624
YouBot10.3%6.4%3.8%5,9387.3%8,624
Diffbot15.7%11.9%3.8%5,93811.8%8,624
Timpibot9.8%5.8%4.0%5,9387.0%8,624
omgili15.7%11.8%3.9%5,93811.8%8,624
ImagesiftBot9.5%5.5%3.9%5,9386.9%8,624
PetalBot14.1%10.3%3.8%5,93810.9%8,624
ora-scanner4.2%0.0%4.1%5,9383.0%8,624
agent-readiness-crawl4.2%0.0%4.1%5,9383.0%8,624

Two rows are controls, not AI crawlers. agent-readiness-crawl is this crawl's own token. ora-scanner is a name the crawl measured that is not Ora's user agent: Ora's scanner calls itself OraBot (profile), which the crawl did not measure. The two rows carry the same figures, which show the share of sites whose * group refuses everything.

The 401 and 403 share

The crawl fetched 10,565 home pages with its labelled user agent. 12.9 percent answered 401 or 403; among the top 10,000 the share was 17.2 percent (1,300 of 7,545). Pages were fetched only where robots.txt allowed the crawl, so 259 domains that disallow it are not in this share.

Ora's lab found (2026-10-19) that returning 403 to agent user agents left both a plain-fetch and a JavaScript-running client at 0 of 5 retrieved answers, in a test of 19 configurations on one docs site (190 probes). That is a vendor finding, and a crawl cannot say which of these 403s also block agents.

Check it on your own site: Ora's Scan includes a bot detection check and a robots.txt AI crawler check; the CLI is npx ax audit yourdomain.com.

The edition page breaks the 401 and 403 share down by category and by bot-management header.

Added on 2026-11-05

Editions and downloads

Cite this. Agent Ready Crawlers, "Block Rate Report", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/. More on the data page.

Sources

Last reviewed 2026-11-05