--- title: "401 and 403 by edge and bot-product header | Agent Ready Crawlers" description: "The 401 and 403 share of home pages split by the CDN, host and bot-product headers they carried, from a crawl of 13,071 domains. Crawl of 2026-11-04." canonical: https://agentreadycrawlers.com/block-rates/edge-and-bot-headers site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Block rates" --- # 401 and 403 by edge and bot-product header | Agent Ready Crawlers > The 401 and 403 share of home pages split by the CDN, host and bot-product headers they carried, from a crawl of 13,071 domains. Crawl of 2026-11-04. Source: https://agentreadycrawlers.com/block-rates/edge-and-bot-headers Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # 401 and 403 by edge and bot-product header In short 10,565 home pages were fetched; 12.9 percent answered 401 or 403. Pages with a Cloudflare challenge header answered 401 or 403 99.8 percent of the time, pages with a DataDome header 70.3 percent and pages with a Vercel header 0.1 percent. The challenge header sits on the refusal itself. The crawl records response headers on every home page it fetched. Some name a CDN or host, a few name a bot product, and one says the request was challenged. Setting each beside the status the page answered shows where the refusals sit, with a caution: a header says what the response carried, and Cloudflare's challenge header is set on the refusal itself. ## The share that refused, by header Home pages that answered the crawl user agent with 401 or 403, among those carrying each header; headers seen on at least 60 pages. Crawl of 2026-11-04. | Header | What it tells us | Pages | Share of pages | 401 or 403, with it | Without it | | --- | --- | --- | --- | --- | --- | | Cloudflare (cf-ray header) | A CDN or host header only; it does not say whether bot rules are on | 4,062 | 38.4% | 22.3% | 7.0% | | AWS (CloudFront or WAF header) | A CDN or host header only; it does not say whether bot rules are on | 1,202 | 11.4% | 4.1% | 14.0% | | Vercel (x-vercel-id header) | A CDN or host header only; it does not say whether bot rules are on | 1,064 | 10.1% | 0.1% | 14.3% | | Cloudflare challenge (cf-mitigated header) | The response says the request was challenged or mitigated | 644 | 6.1% | 99.8% | 7.2% | | Fastly (server or x-served-by header) | A CDN or host header only; it does not say whether bot rules are on | 513 | 4.9% | 6.0% | 13.2% | | Akamai (server header or cookie) | A CDN or host header only; it does not say whether bot rules are on | 496 | 4.7% | 36.9% | 11.7% | | Netlify (x-nf-request-id header) | A CDN or host header only; it does not say whether bot rules are on | 156 | 1.5% | 0.0% | 13.1% | | Imperva (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 66 | 0.6% | 10.6% | 12.9% | | DataDome (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 64 | 0.6% | 70.3% | 12.5% | | Vercel mitigation (x-vercel-mitigated header) | The response says the request was challenged or mitigated | 32 | 0.3% | 3.1% | 12.9% | | HUMAN (cookie) | A bot product header or cookie is present, which is not proof of enforcement | 6 | 0.1% | 50.0% | 12.8% | | Kasada (header) | A bot product header or cookie is present, which is not proof of enforcement | 3 | 0.0% | 0.0% | 12.9% | ## How to read it Challenge header 643 of the 644 pages with Cloudflare's challenge header answered 401 or 403. The header is set on the challenge itself, so this is the crawl being challenged and not a rate across a population. Edge or host header 4,062 pages, 38.4 percent, carried a Cloudflare header, and 22.3 percent of them refused against 7.0 percent of the rest. The header shows the site is behind Cloudflare, not that bot rules are on; the challenge pages above are part of the difference. Bot-product header 64 pages carried a DataDome header or cookie, and 70.3 percent refused. A bot-product header is not proof of enforcement, and the share describes this crawl's requests only. Hosts that refused almost nothing 1,064 pages carried a Vercel header and 1 refused; the Netlify header was on 156 pages and none refused. The crawl does not show why. ## What the crawl cannot say A status is what one labelled user agent got on one day from one network, and a header is a presence test. Neither says which agents a site admits. To find out what your own site answers to ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, run [the test in the guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/), and read [what a 403 does to an agent](https://agentreadycrawlers.com/guides/403-to-ai-agents/) first. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Block Rate Report, edition 2026-11-04 (this site)](https://agentreadycrawlers.com/block-rates/2026-11-04/) read 2026-11-04 Last reviewed 2026-11-05