401 and 403 by edge and bot-product header
In short
10,565 home pages were fetched; 12.9 percent answered 401 or 403. Pages with a Cloudflare challenge header answered 401 or 403 99.8 percent of the time, pages with a DataDome header 70.3 percent and pages with a Vercel header 0.1 percent. The challenge header sits on the refusal itself.
The crawl records response headers on every home page it fetched. Some name a CDN or host, a few name a bot product, and one says the request was challenged. Setting each beside the status the page answered shows where the refusals sit, with a caution: a header says what the response carried, and Cloudflare's challenge header is set on the refusal itself.
The share that refused, by header
| Header | What it tells us | Pages | Share of pages | 401 or 403, with it | Without it |
|---|---|---|---|---|---|
| Cloudflare (cf-ray header) | A CDN or host header only; it does not say whether bot rules are on | 4,062 | 38.4% | 22.3% | 7.0% |
| AWS (CloudFront or WAF header) | A CDN or host header only; it does not say whether bot rules are on | 1,202 | 11.4% | 4.1% | 14.0% |
| Vercel (x-vercel-id header) | A CDN or host header only; it does not say whether bot rules are on | 1,064 | 10.1% | 0.1% | 14.3% |
| Cloudflare challenge (cf-mitigated header) | The response says the request was challenged or mitigated | 644 | 6.1% | 99.8% | 7.2% |
| Fastly (server or x-served-by header) | A CDN or host header only; it does not say whether bot rules are on | 513 | 4.9% | 6.0% | 13.2% |
| Akamai (server header or cookie) | A CDN or host header only; it does not say whether bot rules are on | 496 | 4.7% | 36.9% | 11.7% |
| Netlify (x-nf-request-id header) | A CDN or host header only; it does not say whether bot rules are on | 156 | 1.5% | 0.0% | 13.1% |
| Imperva (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 66 | 0.6% | 10.6% | 12.9% |
| DataDome (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 64 | 0.6% | 70.3% | 12.5% |
| Vercel mitigation (x-vercel-mitigated header) | The response says the request was challenged or mitigated | 32 | 0.3% | 3.1% | 12.9% |
| HUMAN (cookie) | A bot product header or cookie is present, which is not proof of enforcement | 6 | 0.1% | 50.0% | 12.8% |
| Kasada (header) | A bot product header or cookie is present, which is not proof of enforcement | 3 | 0.0% | 0.0% | 12.9% |
How to read it
- Challenge header
- 643 of the 644 pages with Cloudflare's challenge header answered 401 or 403. The header is set on the challenge itself, so this is the crawl being challenged and not a rate across a population.
- Edge or host header
- 4,062 pages, 38.4 percent, carried a Cloudflare header, and 22.3 percent of them refused against 7.0 percent of the rest. The header shows the site is behind Cloudflare, not that bot rules are on; the challenge pages above are part of the difference.
- Bot-product header
- 64 pages carried a DataDome header or cookie, and 70.3 percent refused. A bot-product header is not proof of enforcement, and the share describes this crawl's requests only.
- Hosts that refused almost nothing
- 1,064 pages carried a Vercel header and 1 refused; the Netlify header was on 156 pages and none refused. The crawl does not show why.
What the crawl cannot say
A status is what one labelled user agent got on one day from one network, and a header is a presence test. Neither says which agents a site admits. To find out what your own site answers to ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, run the test in the guide, and read what a 403 does to an agent first.
Sources
Last reviewed 2026-11-05