Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

crawler-price header

In short

The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. It is how a crawler learns the price before it retries; AWS WAF uses an x402 response instead. Example: HTTP 402 Payment Required with crawler-price: USD XX.XX. Related: What pay per crawl pays and Cloudflare profile.

Definition

The response header Cloudflare pay per crawl sends with a 402 to state the per-request price.

Sourced example

HTTP 402 Payment Required with crawler-price: USD XX.XX. Source, read 2026-11-04.

How it shows up here

It is how a crawler learns the price before it retries; AWS WAF uses an x402 response instead.

Previous: Crawl-delay. Next: Disallow AI Training. All 41 terms.

Sources

Last reviewed 2026-11-04