Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Block Rate Report: edition 2026-11-04

In short

Edition 4 November 2026 of the Block Rate Report: 13,071 domains, 8,624 parsed robots.txt files. Three charts: refusal by token, GPTBot refusal by category (News & Intelligence 30.3 percent, CRM 0.0 percent), and the 401 and 403 share of home pages.

13,071
crawl rows, read 2026-11-04
10,886
domains reachable
8,624
robots.txt files parsed (5,938 in the top 10,000)
10,565
home pages fetched

2,185 domains did not answer, 767 had no robots.txt (404) and 1,495 answered with something that is not a plain-text robots.txt (an error, a block page or HTML). None of those is in a denominator below. Universe: Tranco list 746WX (top 10,000) plus the Ora public leaderboard and the vendors named on this site. User agent Mozilla/5.0 (compatible; agent-readiness-crawl/1.0).

Chart 1: by token

Share of top-10,000 sites that disallow each AI crawler token at the root32 tokens, from CCBot at 19.4 percent down to Googlebot at 2.8 percent.0%5%10%15%20%25%CCBot19.4%Bytespider19.2%GPTBot18.5%ClaudeBot17.6%Google-Extended16.0%Meta-ExternalAgent16.0%Diffbot15.7%omgili15.7%Amazonbot15.6%anthropic-ai15.5%cohere-ai15.5%Applebot14.8%Applebot-Extended14.8%PetalBot14.1%PerplexityBot11.2%Google-CloudVertexBot10.9%YouBot10.3%FacebookBot10.2%Timpibot9.8%ChatGPT-User9.7%ImagesiftBot9.5%Meta-ExternalFetcher8.5%Perplexity-User8.3%Claude-User8.1%DuckAssistBot8.1%Claude-SearchBot7.7%MistralAI-User7.7%OAI-SearchBot7.7%agent-readiness-crawl*4.2%MicrosoftPreview4.0%Bingbot3.1%Googlebot2.8%Share of top-10,000 sites that disallow each AI crawler token at the root32 tokens, from CCBot at 19.4 percent down to Googlebot at 2.8 percent.0%10%20%CCBot19.4%Bytespider19.2%GPTBot18.5%ClaudeBot17.6%Google-Extended16.0%Meta-ExternalAgent16.0%Diffbot15.7%omgili15.7%Amazonbot15.6%anthropic-ai15.5%cohere-ai15.5%Applebot14.8%Applebot-Extended14.8%PetalBot14.1%PerplexityBot11.2%Google-CloudVertexBot10.9%YouBot10.3%FacebookBot10.2%Timpibot9.8%ChatGPT-User9.7%ImagesiftBot9.5%Meta-ExternalFetcher8.5%Perplexity-User8.3%Claude-User8.1%DuckAssistBot8.1%Claude-SearchBot7.7%MistralAI-User7.7%OAI-SearchBot7.7%agent-readiness-crawl*4.2%MicrosoftPreview4.0%Bingbot3.1%Googlebot2.8%

Site names the tokenRefused only by a * group

Top-10,000 sites with a parsed robots.txt, 5,938 of them. *The crawl's own token, a control: the wildcard baseline.
The same numbers as a table, with the all-crawled column.
TokenRefused, top 10,000NamedVia *SitesRefused, all crawledSites
GPTBot18.5%15.2%3.3%5,93813.9%8,624
OAI-SearchBot7.7%4.4%3.3%5,9385.4%8,624
ChatGPT-User9.7%6.3%3.3%5,9386.9%8,624
ClaudeBot17.6%14.0%3.5%5,93813.1%8,624
Claude-User8.1%4.5%3.6%5,9385.7%8,624
Claude-SearchBot7.7%4.1%3.6%5,9385.5%8,624
anthropic-ai15.5%11.7%3.8%5,93811.6%8,624
PerplexityBot11.2%7.9%3.4%5,9387.9%8,624
Perplexity-User8.3%4.6%3.7%5,9385.9%8,624
Google-Extended16.0%12.6%3.4%5,93812.1%8,624
Googlebot2.8%0.5%2.3%5,9382.1%8,624
Google-CloudVertexBot10.9%7.1%3.8%5,9388.3%8,624
Applebot14.8%11.6%3.1%5,93811.1%8,624
Applebot-Extended14.8%11.6%3.1%5,93811.1%8,624
Bingbot3.1%0.4%2.7%5,9382.2%8,624
MicrosoftPreview4.0%0.0%3.9%5,9382.9%8,624
Amazonbot15.6%12.0%3.6%5,93811.7%8,624
Meta-ExternalAgent16.0%12.3%3.7%5,93812.0%8,624
Meta-ExternalFetcher8.5%4.7%3.8%5,9386.0%8,624
FacebookBot10.2%6.4%3.7%5,9387.3%8,624
Bytespider19.2%15.4%3.8%5,93815.0%8,624
CCBot19.4%15.7%3.7%5,93814.9%8,624
DuckAssistBot8.1%4.3%3.8%5,9385.8%8,624
MistralAI-User7.7%3.8%3.9%5,9385.5%8,624
cohere-ai15.5%11.6%3.8%5,93811.7%8,624
YouBot10.3%6.4%3.8%5,9387.3%8,624
Diffbot15.7%11.9%3.8%5,93811.8%8,624
Timpibot9.8%5.8%4.0%5,9387.0%8,624
omgili15.7%11.8%3.9%5,93811.8%8,624
ImagesiftBot9.5%5.5%3.9%5,9386.9%8,624
PetalBot14.1%10.3%3.8%5,93810.9%8,624
ora-scanner4.2%0.0%4.1%5,9383.0%8,624
agent-readiness-crawl4.2%0.0%4.1%5,9383.0%8,624

Two rows are controls, not AI crawlers. agent-readiness-crawl is this crawl's own token. ora-scanner is a name the crawl measured that is not Ora's user agent: Ora's scanner calls itself OraBot (profile), which the crawl did not measure. The two rows carry the same figures, which show the share of sites whose * group refuses everything.

Top 1,000 against the rest of the top 10,000

TokenRank 1 to 1,000SitesRank 1,001 to 10,000Sites
GPTBot24.2%52117.9%5,417
ClaudeBot24.0%52117.0%5,417
PerplexityBot19.4%52110.4%5,417
Google-Extended20.7%52115.5%5,417
Applebot-Extended18.8%52114.4%5,417
CCBot26.5%52118.7%5,417
Bytespider26.7%52118.5%5,417
Googlebot4.6%5212.6%5,417

Chart 2: by category

Categories are the Ora public leaderboard's own. A category appears when at least 30 of its sites returned a parsed robots.txt. The uncategorised group is the top-10,000 list and the vendors this site names.

Share of sites that disallow GPTBot at the root, by Ora leaderboard category24 categories with at least 30 parsed robots.txt files, from News & Intelligence at 30.3 percent to CRM at 0.0 percent.0%5%10%15%20%25%30%35%News & Intelligence30.3%Gaming & Media16.4%Hardware & Robotics7.5%Identity & Auth7.4%Productivity6.0%Legal & Compliance5.9%Logistics & Supply Chain5.6%Developer Tools5.4%Agent Tools5.0%Education5.0%E-commerce4.5%Payments & Fintech3.8%Real Estate & Construction3.8%Consumer3.5%Healthcare3.3%Communication & Support3.1%Data & Analytics2.9%Climate & Energy2.6%Marketing2.5%AI & ML2.3%Security2.3%Infrastructure2.0%HR & Recruiting1.9%CRM0.0%Share of sites that disallow GPTBot at the root, by Ora leaderboard category24 categories with at least 30 parsed robots.txt files, from News & Intelligence at 30.3 percent to CRM at 0.0 percent.0%10%20%30%News & Intelligence30.3%Gaming & Media16.4%Hardware & Robotics7.5%Identity & Auth7.4%Productivity6.0%Legal & Compliance5.9%Logistics & Supply Chain5.6%Developer Tools5.4%Agent Tools5.0%Education5.0%E-commerce4.5%Payments & Fintech3.8%Real Estate & Construction3.8%Consumer3.5%Healthcare3.3%Communication & Support3.1%Data & Analytics2.9%Climate & Energy2.6%Marketing2.5%AI & ML2.3%Security2.3%Infrastructure2.0%HR & Recruiting1.9%CRM0.0%

Site names GPTBotRefused only by a * group

Share of sites per category that disallow GPTBot at the root. Highest News & Intelligence, 30.3 percent; lowest CRM, 0.0 percent.
CategorySitesGPTBotClaudeBotPerplexityBotGoogle-ExtendedGooglebot
Agent Tools1205.0%4.2%0.0%5.0%0.0%
AI & ML1772.3%2.3%0.6%3.4%1.1%
Climate & Energy392.6%2.6%0.0%2.6%0.0%
Communication & Support1303.1%2.3%0.8%2.3%0.0%
Consumer573.5%3.5%0.0%3.5%0.0%
CRM400.0%0.0%0.0%0.0%0.0%
Data & Analytics1372.9%2.9%0.0%2.2%0.0%
Developer Tools3335.4%4.5%1.8%4.8%0.9%
E-commerce1334.5%4.5%2.3%4.5%1.5%
Education805.0%6.3%0.0%5.0%0.0%
Gaming & Media6116.4%16.4%3.3%11.5%1.6%
Hardware & Robotics807.5%6.3%0.0%7.5%0.0%
Healthcare2153.3%3.3%1.4%3.3%0.0%
HR & Recruiting521.9%1.9%1.9%1.9%1.9%
Identity & Auth687.4%5.9%4.4%7.4%1.5%
Infrastructure2042.0%2.0%0.0%2.0%0.0%
Legal & Compliance855.9%3.5%1.2%4.7%1.2%
Logistics & Supply Chain725.6%4.2%1.4%4.2%1.4%
Marketing2042.5%2.0%0.0%1.5%0.0%
News & Intelligence3330.3%42.4%33.3%42.4%0.0%
Payments & Fintech3643.8%3.6%0.0%2.7%0.0%
Productivity1336.0%3.8%0.8%3.8%0.0%
Real Estate & Construction803.8%3.8%2.5%5.0%1.3%
Security862.3%3.5%0.0%2.3%1.2%

Chart 3: the 401 and 403 share

Share of fetched home pages that answered 401 or 403 to the labelled crawl user agent, Mozilla/5.0 (compatible; agent-readiness-crawl/1.0). Pages were fetched only where robots.txt allowed the crawl; 259 domains disallowed it and are not counted.

Share of home pages that answered 401 or 403 to the crawl user agent, by group27 groups, from News & Intelligence at 25.7 percent to Identity & Auth at 0.0 percent.0%5%10%15%20%25%30%All crawled12.9%Top 10,00017.2%Top 1,00013.0%News & Intelligence25.7%Gaming & Media12.9%CRM7.1%HR & Recruiting5.3%Communication & Support4.7%Education4.3%Marketing4.1%Consumer4.0%Real Estate & Construction3.5%E-commerce3.4%AI & ML2.9%Productivity2.7%Climate & Energy2.4%Legal & Compliance2.2%Security2.1%Data & Analytics2.0%Payments & Fintech2.0%Hardware & Robotics1.9%Infrastructure1.7%Healthcare1.6%Logistics & Supply Chain1.2%Agent Tools0.8%Developer Tools0.8%Identity & Auth0.0%Share of home pages that answered 401 or 403 to the crawl user agent, by group27 groups, from News & Intelligence at 25.7 percent to Identity & Auth at 0.0 percent.0%10%20%30%All crawled12.9%Top 10,00017.2%Top 1,00013.0%News & Intelligence25.7%Gaming & Media12.9%CRM7.1%HR & Recruiting5.3%Communication & Support4.7%Education4.3%Marketing4.1%Consumer4.0%Real Estate & Construction3.5%E-commerce3.4%AI & ML2.9%Productivity2.7%Climate & Energy2.4%Legal & Compliance2.2%Security2.1%Data & Analytics2.0%Payments & Fintech2.0%Hardware & Robotics1.9%Infrastructure1.7%Healthcare1.6%Logistics & Supply Chain1.2%Agent Tools0.8%Developer Tools0.8%Identity & Auth0.0%
401 and 403 share of home pages by group. All crawled: 10,565 pages. Top 10,000: 7,545.
GroupHome pages401403401 or 403
All crawled10,565151,34412.9%
Top 10,0007,545141,28617.2%
Rank 1 to 1,00066318513.0%
Rank 1,001 to 10,0006,882131,20117.6%
Agent Tools131010.8%
AI & ML205062.9%
Climate & Energy42012.4%
Communication & Support149074.7%
Consumer75034.0%
CRM42037.1%
Data & Analytics148032.0%
Developer Tools376030.8%
E-commerce149053.4%
Education92044.3%
Gaming & Media701812.9%
Hardware & Robotics104021.9%
Healthcare246041.6%
HR & Recruiting57035.3%
Identity & Auth70000.0%
Infrastructure232131.7%
Legal & Compliance91022.2%
Logistics & Supply Chain85011.2%
Marketing222094.1%
News & Intelligence350925.7%
Payments & Fintech399082.0%
Productivity148042.7%
Real Estate & Construction85033.5%
Security94022.1%

Ora's lab found (2026-10-19) that a 403 to agent user agents left both a plain-fetch and a JavaScript client at 0 of 5 retrieved answers; that is a vendor finding from one docs site. Check it on your own domain with Ora's bot detection check or npx ax audit yourdomain.com.

Bot-management headers on home pages

The crawl records headers that reveal an edge or bot product. Most only show that a CDN or host is present. Home pages behind Cloudflare headers answered 401 or 403 22.3 percent of the time, against 7.0 percent for pages without them. A header is not proof that bot rules are switched on.

Header signalWhat it tells usHome pagesShare401 or 403 with itWithout it
Cloudflare (cf-ray header)A CDN or host header only; it does not say whether bot rules are on4,06238.4%22.3%7.0%
AWS (CloudFront or WAF header)A CDN or host header only; it does not say whether bot rules are on1,20211.4%4.1%14.0%
Vercel (x-vercel-id header)A CDN or host header only; it does not say whether bot rules are on1,06410.1%0.1%14.3%
Cloudflare challenge (cf-mitigated header)The response says the request was challenged or mitigated6446.1%99.8%7.2%
Fastly (server or x-served-by header)A CDN or host header only; it does not say whether bot rules are on5134.9%6.0%13.2%
Akamai (server header or cookie)A CDN or host header only; it does not say whether bot rules are on4964.7%36.9%11.7%
Netlify (x-nf-request-id header)A CDN or host header only; it does not say whether bot rules are on1561.5%0.0%13.1%
Imperva (header or cookie)A bot product header or cookie is present, which is not proof of enforcement660.6%10.6%12.9%
DataDome (header or cookie)A bot product header or cookie is present, which is not proof of enforcement640.6%70.3%12.5%
Vercel mitigation (x-vercel-mitigated header)The response says the request was challenged or mitigated320.3%3.1%12.9%
HUMAN (cookie)A bot product header or cookie is present, which is not proof of enforcement60.1%50.0%12.8%
Kasada (header)A bot product header or cookie is present, which is not proof of enforcement30.0%0.0%12.9%

Cite and download

Cite this. Agent Ready Crawlers, "Block Rate Report, edition 2026-11-04", https://agentreadycrawlers.com/block-rates/2026-11-04/. Crawl row count 13,071, read 2026-11-04.

Sources

Last reviewed 2026-11-04