Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Cloudflare vs AWS WAF Bot Control

In short

AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. For a security engineer: Both verify Web Bot Auth (AWS needs rule set Version_4.0 or later selected).

The sourced difference

AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. Source, read 2026-11-04.

Side by side

QuestionCloudflareAWS WAF Bot Control
Tells a verified agent from a spoof Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)
Yes
Web Bot Auth (rule set Version_4.0+), verified self-identifying bots
Per-purpose policy (train, search, answer, act) Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15
Yes
rules on bot category and name labels; prices can differ by bot identity and verification status
Charging for access Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta
Yes
HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins
Lets signed commerce agents through Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)
Not published
Visa TAP or Mastercard Agent Pay support (checked 2026-11-04)
Cost free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance
Reporting Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance
Yes
Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs
Where it runs edge: enforced on Cloudflare's reverse proxy, no app code needed edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway
Points, rank8 of 10, rank 1 (tied)8 of 10, rank 1 (tied)
Price read 2026-11-04AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge.

Verdicts by reader

Platform engineer
On AWS, Bot Control is USD 10 a month per web ACL plus per-request fees; Cloudflare AI Crawl Control is on the free plan.
Security engineer
Both verify Web Bot Auth (AWS needs rule set Version_4.0 or later selected).
Publisher operations
AWS charging is generally available with Test mode; Cloudflare pay per crawl is in private beta.

What each does not do

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

AWS WAF Bot Control

  • Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs).
  • Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15).
  • Web Bot Auth is off unless the rule group version is pinned to Version_4.0 or later (Bot Control docs).

More: alternatives to Cloudflare, alternatives to AWS WAF Bot Control.

Sources

Last reviewed 2026-11-04