Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

What pay per crawl pays and who is paying

In short

Three charging paths exist today: Cloudflare pay per crawl (closed beta), AWS WAF x402 monetization (CloudFront), and partner paywalls such as TollBit; none publishes payout totals. Cloudflare pay per crawl returns HTTP 402 with a crawler-price header; crawlers retry with crawler-exact-price or send crawler-max-price up front.

Three ways to charge a crawler exist today. None publishes payout totals, so what they pay publishers is not yet public.

In practice

  1. Cloudflare pay per crawl answers with HTTP 402 and a crawler-price header; a crawler retries with crawler-exact-price or sends crawler-max-price up front. Cloudflare acts as Merchant of Record, and the feature is in closed beta per documents updated 2026-07-28.
  2. AWS WAF AI traffic monetization returns an x402 HTTP 402 and settles through Coinbase in stablecoins. Stripe payouts were listed as coming soon.
  3. TollBit sells a Bot Paywall that publishers price by usage. TollBit reports more than 2.6 billion AI bots directed to its paywall in the first half of 2026, which is its own count.
  4. Skyfire pay tokens commit the buyer's funds, so a seller that charges a valid token is guaranteed payment up to the committed amount.
  5. Read the vendor pages before choosing. What each does not do is on its profile.

What the sources say

Sources

Last reviewed 2026-11-04