---
title: "Block Rate Report, edition 2026-11-04 | Agent Ready Crawlers"
description: "Block Rate Report edition 2026-11-04: three charts and tables for token, category and the 401 and 403 share, from a crawl of 13,071 domains."
canonical: https://agentreadycrawlers.com/block-rates/2026-11-04
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
date: 2026-11-04
section: "Block rates"
---

# Block Rate Report, edition 2026-11-04 | Agent Ready Crawlers

> Block Rate Report edition 2026-11-04: three charts and tables for token, category and the 401 and 403 share, from a crawl of 13,071 domains.

Source: https://agentreadycrawlers.com/block-rates/2026-11-04
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

# Block Rate Report: edition 2026-11-04

In short

Edition 4 November 2026 of the Block Rate Report: 13,071 domains, 8,624 parsed robots.txt files. Three charts: refusal by token, GPTBot refusal by category (News & Intelligence 30.3 percent, CRM 0.0 percent), and the 401 and 403 share of home pages.

13,071

crawl rows, read 2026-11-04

10,886

domains reachable

8,624

robots.txt files parsed (5,938 in the top 10,000)

10,565

home pages fetched

2,185 domains did not answer, 767 had no robots.txt (404) and 1,495 answered with something that is not a plain-text robots.txt (an error, a block page or HTML). None of those is in a denominator below. Universe: Tranco list 746WX (top 10,000) plus the Ora public leaderboard and the vendors named on this site. User agent `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`.

## Chart 1: by token

Site names the tokenRefused only by a \* group

Top-10,000 sites with a parsed robots.txt, 5,938 of them. \*The crawl's own token, a control: the wildcard baseline.

| Token | Refused, top 10,000 | Named | Via * | Sites | Refused, all crawled | Sites |
| --- | --- | --- | --- | --- | --- | --- |
| GPTBot | 18.5% | 15.2% | 3.3% | 5,938 | 13.9% | 8,624 |
| OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5,938 | 5.4% | 8,624 |
| ChatGPT-User | 9.7% | 6.3% | 3.3% | 5,938 | 6.9% | 8,624 |
| ClaudeBot | 17.6% | 14.0% | 3.5% | 5,938 | 13.1% | 8,624 |
| Claude-User | 8.1% | 4.5% | 3.6% | 5,938 | 5.7% | 8,624 |
| Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5,938 | 5.5% | 8,624 |
| anthropic-ai | 15.5% | 11.7% | 3.8% | 5,938 | 11.6% | 8,624 |
| PerplexityBot | 11.2% | 7.9% | 3.4% | 5,938 | 7.9% | 8,624 |
| Perplexity-User | 8.3% | 4.6% | 3.7% | 5,938 | 5.9% | 8,624 |
| Google-Extended | 16.0% | 12.6% | 3.4% | 5,938 | 12.1% | 8,624 |
| Googlebot | 2.8% | 0.5% | 2.3% | 5,938 | 2.1% | 8,624 |
| Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 5,938 | 8.3% | 8,624 |
| Applebot | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 |
| Applebot-Extended | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 |
| Bingbot | 3.1% | 0.4% | 2.7% | 5,938 | 2.2% | 8,624 |
| MicrosoftPreview | 4.0% | 0.0% | 3.9% | 5,938 | 2.9% | 8,624 |
| Amazonbot | 15.6% | 12.0% | 3.6% | 5,938 | 11.7% | 8,624 |
| Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 5,938 | 12.0% | 8,624 |
| Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 5,938 | 6.0% | 8,624 |
| FacebookBot | 10.2% | 6.4% | 3.7% | 5,938 | 7.3% | 8,624 |
| Bytespider | 19.2% | 15.4% | 3.8% | 5,938 | 15.0% | 8,624 |
| CCBot | 19.4% | 15.7% | 3.7% | 5,938 | 14.9% | 8,624 |
| DuckAssistBot | 8.1% | 4.3% | 3.8% | 5,938 | 5.8% | 8,624 |
| MistralAI-User | 7.7% | 3.8% | 3.9% | 5,938 | 5.5% | 8,624 |
| cohere-ai | 15.5% | 11.6% | 3.8% | 5,938 | 11.7% | 8,624 |
| YouBot | 10.3% | 6.4% | 3.8% | 5,938 | 7.3% | 8,624 |
| Diffbot | 15.7% | 11.9% | 3.8% | 5,938 | 11.8% | 8,624 |
| Timpibot | 9.8% | 5.8% | 4.0% | 5,938 | 7.0% | 8,624 |
| omgili | 15.7% | 11.8% | 3.9% | 5,938 | 11.8% | 8,624 |
| ImagesiftBot | 9.5% | 5.5% | 3.9% | 5,938 | 6.9% | 8,624 |
| PetalBot | 14.1% | 10.3% | 3.8% | 5,938 | 10.9% | 8,624 |
| ora-scanner | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 |
| agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 |

Two rows are controls, not AI crawlers. **agent-readiness-crawl** is this crawl's own token. **ora-scanner** is a name the crawl measured that is not Ora's user agent: Ora's scanner calls itself OraBot ([profile](https://agentreadycrawlers.com/crawlers/orabot/)), which the crawl did not measure. The two rows carry the same figures, which show the share of sites whose \* group refuses everything.

### Top 1,000 against the rest of the top 10,000

| Token | Rank 1 to 1,000 | Sites | Rank 1,001 to 10,000 | Sites |
| --- | --- | --- | --- | --- |
| GPTBot | 24.2% | 521 | 17.9% | 5,417 |
| ClaudeBot | 24.0% | 521 | 17.0% | 5,417 |
| PerplexityBot | 19.4% | 521 | 10.4% | 5,417 |
| Google-Extended | 20.7% | 521 | 15.5% | 5,417 |
| Applebot-Extended | 18.8% | 521 | 14.4% | 5,417 |
| CCBot | 26.5% | 521 | 18.7% | 5,417 |
| Bytespider | 26.7% | 521 | 18.5% | 5,417 |
| Googlebot | 4.6% | 521 | 2.6% | 5,417 |

## Chart 2: by category

Categories are the Ora public leaderboard's own. A category appears when at least 30 of its sites returned a parsed robots.txt. The uncategorised group is the top-10,000 list and the vendors this site names.

Site names GPTBotRefused only by a \* group

Share of sites per category that disallow GPTBot at the root. Highest News & Intelligence, 30.3 percent; lowest CRM, 0.0 percent.

| Category | Sites | GPTBot | ClaudeBot | PerplexityBot | Google-Extended | Googlebot |
| --- | --- | --- | --- | --- | --- | --- |
| Agent Tools | 120 | 5.0% | 4.2% | 0.0% | 5.0% | 0.0% |
| AI & ML | 177 | 2.3% | 2.3% | 0.6% | 3.4% | 1.1% |
| Climate & Energy | 39 | 2.6% | 2.6% | 0.0% | 2.6% | 0.0% |
| Communication & Support | 130 | 3.1% | 2.3% | 0.8% | 2.3% | 0.0% |
| Consumer | 57 | 3.5% | 3.5% | 0.0% | 3.5% | 0.0% |
| CRM | 40 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Data & Analytics | 137 | 2.9% | 2.9% | 0.0% | 2.2% | 0.0% |
| Developer Tools | 333 | 5.4% | 4.5% | 1.8% | 4.8% | 0.9% |
| E-commerce | 133 | 4.5% | 4.5% | 2.3% | 4.5% | 1.5% |
| Education | 80 | 5.0% | 6.3% | 0.0% | 5.0% | 0.0% |
| Gaming & Media | 61 | 16.4% | 16.4% | 3.3% | 11.5% | 1.6% |
| Hardware & Robotics | 80 | 7.5% | 6.3% | 0.0% | 7.5% | 0.0% |
| Healthcare | 215 | 3.3% | 3.3% | 1.4% | 3.3% | 0.0% |
| HR & Recruiting | 52 | 1.9% | 1.9% | 1.9% | 1.9% | 1.9% |
| Identity & Auth | 68 | 7.4% | 5.9% | 4.4% | 7.4% | 1.5% |
| Infrastructure | 204 | 2.0% | 2.0% | 0.0% | 2.0% | 0.0% |
| Legal & Compliance | 85 | 5.9% | 3.5% | 1.2% | 4.7% | 1.2% |
| Logistics & Supply Chain | 72 | 5.6% | 4.2% | 1.4% | 4.2% | 1.4% |
| Marketing | 204 | 2.5% | 2.0% | 0.0% | 1.5% | 0.0% |
| News & Intelligence | 33 | 30.3% | 42.4% | 33.3% | 42.4% | 0.0% |
| Payments & Fintech | 364 | 3.8% | 3.6% | 0.0% | 2.7% | 0.0% |
| Productivity | 133 | 6.0% | 3.8% | 0.8% | 3.8% | 0.0% |
| Real Estate & Construction | 80 | 3.8% | 3.8% | 2.5% | 5.0% | 1.3% |
| Security | 86 | 2.3% | 3.5% | 0.0% | 2.3% | 1.2% |

## Chart 3: the 401 and 403 share

Share of fetched home pages that answered 401 or 403 to the labelled crawl user agent, `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`. Pages were fetched only where robots.txt allowed the crawl; 259 domains disallowed it and are not counted.

401 and 403 share of home pages by group. All crawled: 10,565 pages. Top 10,000: 7,545.

| Group | Home pages | 401 | 403 | 401 or 403 |
| --- | --- | --- | --- | --- |
| All crawled | 10,565 | 15 | 1,344 | 12.9% |
| Top 10,000 | 7,545 | 14 | 1,286 | 17.2% |
| Rank 1 to 1,000 | 663 | 1 | 85 | 13.0% |
| Rank 1,001 to 10,000 | 6,882 | 13 | 1,201 | 17.6% |
| Agent Tools | 131 | 0 | 1 | 0.8% |
| AI & ML | 205 | 0 | 6 | 2.9% |
| Climate & Energy | 42 | 0 | 1 | 2.4% |
| Communication & Support | 149 | 0 | 7 | 4.7% |
| Consumer | 75 | 0 | 3 | 4.0% |
| CRM | 42 | 0 | 3 | 7.1% |
| Data & Analytics | 148 | 0 | 3 | 2.0% |
| Developer Tools | 376 | 0 | 3 | 0.8% |
| E-commerce | 149 | 0 | 5 | 3.4% |
| Education | 92 | 0 | 4 | 4.3% |
| Gaming & Media | 70 | 1 | 8 | 12.9% |
| Hardware & Robotics | 104 | 0 | 2 | 1.9% |
| Healthcare | 246 | 0 | 4 | 1.6% |
| HR & Recruiting | 57 | 0 | 3 | 5.3% |
| Identity & Auth | 70 | 0 | 0 | 0.0% |
| Infrastructure | 232 | 1 | 3 | 1.7% |
| Legal & Compliance | 91 | 0 | 2 | 2.2% |
| Logistics & Supply Chain | 85 | 0 | 1 | 1.2% |
| Marketing | 222 | 0 | 9 | 4.1% |
| News & Intelligence | 35 | 0 | 9 | 25.7% |
| Payments & Fintech | 399 | 0 | 8 | 2.0% |
| Productivity | 148 | 0 | 4 | 2.7% |
| Real Estate & Construction | 85 | 0 | 3 | 3.5% |
| Security | 94 | 0 | 2 | 2.1% |

Ora's lab found (2026-10-19) that a 403 to agent user agents left both a plain-fetch and a JavaScript client at 0 of 5 retrieved answers; that is a vendor finding from one docs site. Check it on your own domain with [Ora's bot detection check](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=block-rates-edition) or `npx ax audit yourdomain.com`.

### Bot-management headers on home pages

The crawl records headers that reveal an edge or bot product. Most only show that a CDN or host is present. Home pages behind Cloudflare headers answered 401 or 403 22.3 percent of the time, against 7.0 percent for pages without them. A header is not proof that bot rules are switched on.

| Header signal | What it tells us | Home pages | Share | 401 or 403 with it | Without it |
| --- | --- | --- | --- | --- | --- |
| Cloudflare (cf-ray header) | A CDN or host header only; it does not say whether bot rules are on | 4,062 | 38.4% | 22.3% | 7.0% |
| AWS (CloudFront or WAF header) | A CDN or host header only; it does not say whether bot rules are on | 1,202 | 11.4% | 4.1% | 14.0% |
| Vercel (x-vercel-id header) | A CDN or host header only; it does not say whether bot rules are on | 1,064 | 10.1% | 0.1% | 14.3% |
| Cloudflare challenge (cf-mitigated header) | The response says the request was challenged or mitigated | 644 | 6.1% | 99.8% | 7.2% |
| Fastly (server or x-served-by header) | A CDN or host header only; it does not say whether bot rules are on | 513 | 4.9% | 6.0% | 13.2% |
| Akamai (server header or cookie) | A CDN or host header only; it does not say whether bot rules are on | 496 | 4.7% | 36.9% | 11.7% |
| Netlify (x-nf-request-id header) | A CDN or host header only; it does not say whether bot rules are on | 156 | 1.5% | 0.0% | 13.1% |
| Imperva (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 66 | 0.6% | 10.6% | 12.9% |
| DataDome (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 64 | 0.6% | 70.3% | 12.5% |
| Vercel mitigation (x-vercel-mitigated header) | The response says the request was challenged or mitigated | 32 | 0.3% | 3.1% | 12.9% |
| HUMAN (cookie) | A bot product header or cookie is present, which is not proof of enforcement | 6 | 0.1% | 50.0% | 12.8% |
| Kasada (header) | A bot product header or cookie is present, which is not proof of enforcement | 3 | 0.0% | 0.0% | 12.9% |

## Cite and download

-   [Block rates by token (CSV)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.csv)
-   [Block rates (JSON)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.json)
-   [Home page 401 and 403 share (CSV)](https://agentreadycrawlers.com/data/home-status-2026-11-04.csv)
-   [Bot-management header prevalence (CSV)](https://agentreadycrawlers.com/data/bot-signals-2026-11-04.csv)

**Cite this.** Agent Ready Crawlers, "Block Rate Report, edition 2026-11-04", https://agentreadycrawlers.com/block-rates/2026-11-04/. Crawl row count 13,071, read 2026-11-04.

## Sources

-   [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04
-   [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04
-   [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=edition) read 2026-11-04
-   [Tranco list 746WX, top 10,000](https://tranco-list.eu/list/746WX/10000) read 2026-11-04

Last reviewed 2026-11-04