Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Akamai vs Cloudflare

In short

Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). On per-purpose policy the rubric grades Akamai partial and Cloudflare yes, and on commerce agents the rubric grades Akamai yes and Cloudflare partial. For a security engineer: Akamai announced Visa TAP integration.

The sourced difference

Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). Source, read 2026-11-04.

Side by side

QuestionAkamaiCloudflare
Tells a verified agent from a spoof Yes
Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework
Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)
Per-purpose policy (train, search, answer, act) Partial
allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04)
Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15
Charging for access Partial
through TollBit and Skyfire partners, no native billing
Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta
Lets signed commerce agents through Yes
Visa Trusted Agent Protocol integration, Skyfire and Experian KYA
Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)
Cost no public price; sales-led free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led
Reporting Yes
TrafficPeak and web security analytics
Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance
Where it runs edge: verification and enforcement on Akamai's edge network edge: enforced on Cloudflare's reverse proxy, no app code needed
Points, rank8 of 10, rank 1 (tied)8 of 10, rank 1 (tied)
Price read 2026-11-04no public price; sales-led (checked 2026-11-04)AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).

Verdicts by reader

Platform engineer
Both verify Web Bot Auth at the edge. Akamai prices are sales-led; Cloudflare has a free tier.
Security engineer
Akamai announced Visa TAP integration (2025-12-17); Cloudflare announced managed rulesets for TAP and Agent Pay in 2025-10 but the current rule name is not in its docs (checked 2026-11-04).
Publisher operations
Cloudflare lets you charge without a third party contract; Akamai needs a TollBit or Skyfire relationship.

What each does not do

Akamai

  • No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release).
  • The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04).
  • No published control that separates search crawling from model training as purposes (pages read 2026-11-04).

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

More: alternatives to Akamai, alternatives to Cloudflare.

Sources

Last reviewed 2026-11-04