--- title: "Block Rate Report: who refuses which crawler | Agent Ready Crawlers" description: "The Block Rate Report: the share of top-10,000 sites that disallow each AI crawler token and the share of home pages answering 401 or 403. 13,071 rows." canonical: https://agentreadycrawlers.com/block-rates site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Block rates" --- # Block Rate Report: who refuses which crawler | Agent Ready Crawlers > The Block Rate Report: the share of top-10,000 sites that disallow each AI crawler token and the share of home pages answering 401 or 403. 13,071 rows. Source: https://agentreadycrawlers.com/block-rates Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Edition 2026-11-04 # The Block Rate Report In short Of 5,938 top-10,000 sites with a parsed robots.txt, 18.5 percent refuse GPTBot at the root and 17.6 percent refuse ClaudeBot, against 2.8 percent for Googlebot. 17.2 percent of their home pages answered 401 or 403 to the crawl. Crawl of 13,071 domains, 4 November 2026. 8,624 robots.txt files parsed, of 13,071 domains crawled 18.5% of 5,938 top-10,000 sites refuse GPTBot 17.6% refuse ClaudeBot 17.2% of top-10,000 home pages answered 401 or 403 ## By token Site names the tokenRefused only by a \* group Share of top-10,000 sites with a parsed robots.txt that refuse each token at the root. \*The crawl's own token, a control: its bar is the wildcard baseline. CCBot is the most refused token at 19.4 percent, and Googlebot the least refused at 2.8 percent. Across all 13,071 crawled domains every rate is lower: GPTBot 13.9 percent, ClaudeBot 13.1 percent. The legacy token anthropic-ai, which Anthropic no longer lists on its crawler page, is still refused by 15.5 percent of the top 10,000, close to ClaudeBot at 17.6 percent. Files written for one name keep working for the other only if the site names both. Refusing a token states a preference. User-triggered fetchers such as ChatGPT-User and Perplexity-User are, by their operators' own statements, allowed to ignore it; see [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/). | Token | Refused, top 10,000 | Named | Via * | Sites | Refused, all crawled | Sites | | --- | --- | --- | --- | --- | --- | --- | | GPTBot | 18.5% | 15.2% | 3.3% | 5,938 | 13.9% | 8,624 | | OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5,938 | 5.4% | 8,624 | | ChatGPT-User | 9.7% | 6.3% | 3.3% | 5,938 | 6.9% | 8,624 | | ClaudeBot | 17.6% | 14.0% | 3.5% | 5,938 | 13.1% | 8,624 | | Claude-User | 8.1% | 4.5% | 3.6% | 5,938 | 5.7% | 8,624 | | Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5,938 | 5.5% | 8,624 | | anthropic-ai | 15.5% | 11.7% | 3.8% | 5,938 | 11.6% | 8,624 | | PerplexityBot | 11.2% | 7.9% | 3.4% | 5,938 | 7.9% | 8,624 | | Perplexity-User | 8.3% | 4.6% | 3.7% | 5,938 | 5.9% | 8,624 | | Google-Extended | 16.0% | 12.6% | 3.4% | 5,938 | 12.1% | 8,624 | | Googlebot | 2.8% | 0.5% | 2.3% | 5,938 | 2.1% | 8,624 | | Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 5,938 | 8.3% | 8,624 | | Applebot | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 | | Applebot-Extended | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 | | Bingbot | 3.1% | 0.4% | 2.7% | 5,938 | 2.2% | 8,624 | | MicrosoftPreview | 4.0% | 0.0% | 3.9% | 5,938 | 2.9% | 8,624 | | Amazonbot | 15.6% | 12.0% | 3.6% | 5,938 | 11.7% | 8,624 | | Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 5,938 | 12.0% | 8,624 | | Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 5,938 | 6.0% | 8,624 | | FacebookBot | 10.2% | 6.4% | 3.7% | 5,938 | 7.3% | 8,624 | | Bytespider | 19.2% | 15.4% | 3.8% | 5,938 | 15.0% | 8,624 | | CCBot | 19.4% | 15.7% | 3.7% | 5,938 | 14.9% | 8,624 | | DuckAssistBot | 8.1% | 4.3% | 3.8% | 5,938 | 5.8% | 8,624 | | MistralAI-User | 7.7% | 3.8% | 3.9% | 5,938 | 5.5% | 8,624 | | cohere-ai | 15.5% | 11.6% | 3.8% | 5,938 | 11.7% | 8,624 | | YouBot | 10.3% | 6.4% | 3.8% | 5,938 | 7.3% | 8,624 | | Diffbot | 15.7% | 11.9% | 3.8% | 5,938 | 11.8% | 8,624 | | Timpibot | 9.8% | 5.8% | 4.0% | 5,938 | 7.0% | 8,624 | | omgili | 15.7% | 11.8% | 3.9% | 5,938 | 11.8% | 8,624 | | ImagesiftBot | 9.5% | 5.5% | 3.9% | 5,938 | 6.9% | 8,624 | | PetalBot | 14.1% | 10.3% | 3.8% | 5,938 | 10.9% | 8,624 | | ora-scanner | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 | | agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 | Two rows are controls, not AI crawlers. **agent-readiness-crawl** is this crawl's own token. **ora-scanner** is a name the crawl measured that is not Ora's user agent: Ora's scanner calls itself OraBot ([profile](https://agentreadycrawlers.com/crawlers/orabot/)), which the crawl did not measure. The two rows carry the same figures, which show the share of sites whose \* group refuses everything. ## The 401 and 403 share The crawl fetched 10,565 home pages with its labelled user agent. 12.9 percent answered 401 or 403; among the top 10,000 the share was 17.2 percent (1,300 of 7,545). Pages were fetched only where robots.txt allowed the crawl, so 259 domains that disallow it are not in this share. Ora's lab found (2026-10-19) that returning 403 to agent user agents left both a plain-fetch and a JavaScript-running client at 0 of 5 retrieved answers, in a test of 19 configurations on one docs site (190 probes). That is a vendor finding, and a crawl cannot say which of these 403s also block agents. Check it on your own site: Ora's Scan includes [a bot detection check and a robots.txt AI crawler check](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=block-rates); the CLI is `npx ax audit yourdomain.com`. [The edition page](https://agentreadycrawlers.com/block-rates/2026-11-04/) breaks the 401 and 403 share down by category and by bot-management header. ## Added on 2026-11-05 - [**Training and search policy in robots.txt**](https://agentreadycrawlers.com/block-rates/training-and-search/) How many top sites refuse training crawlers but keep AI search open, and how many refuse both. - [**401 and 403 by edge and bot-product header**](https://agentreadycrawlers.com/block-rates/edge-and-bot-headers/) The share of home pages that answered 401 or 403, split by the CDN and bot-product headers they carried. ## Editions and downloads - 2026-11-04[Edition 2026-11-04](https://agentreadycrawlers.com/block-rates/2026-11-04/): 13,071 crawl rows, read 2026-11-04; first edition. - [Block rates by token (CSV)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.csv) - [Block rates (JSON)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.json) - [Home page 401 and 403 share (CSV)](https://agentreadycrawlers.com/data/home-status-2026-11-04.csv) - [Bot-management header prevalence (CSV)](https://agentreadycrawlers.com/data/bot-signals-2026-11-04.csv) **Cite this.** Agent Ready Crawlers, "Block Rate Report", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/. More on [the data page](https://agentreadycrawlers.com/data/). ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=report) read 2026-11-04 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=report) read 2026-11-04 Last reviewed 2026-11-05