Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Fastly vs Cloudflare

In short

Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. On verifies agents the rubric grades Fastly partial and Cloudflare yes, and on per-purpose policy the rubric grades Fastly partial and Cloudflare yes.

The sourced difference

Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. Source, read 2026-11-04.

Side by side

QuestionFastlyCloudflare
Tells a verified agent from a spoof Partial
verified-bot signals for AI crawlers and fetchers; Skyfire credentials at the edge (announced); Web Bot Auth not found in docs (checked 2026-11-04)		 Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)
Per-purpose policy (train, search, answer, act) Partial
crawler versus fetcher signals, suspected versus verified		 Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15
Charging for access Partial
through TollBit paywall redirect		 Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta
Lets signed commerce agents through Partial
Skyfire KYA/KYAPay partnership (Jun 2026); no Visa TAP or Mastercard Agent Pay found		 Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)
Cost no public price; sales-led free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led
Reporting Not published
AI bot reporting not described on pages read (checked 2026-11-04)		 Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance
Where it runs edge: classification and actions on Fastly's edge network edge: enforced on Cloudflare's reverse proxy, no app code needed
Points, rank4 of 10, rank 108 of 10, rank 1 (tied)
Price read 2026-11-04no public price for Bot Management or AI Bot Management; contact sales (checked 2026-11-04). Free tier covers up to 100,000 requests/month for delivery.AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).

Verdicts by reader

Platform engineer
Both are edge platforms; Fastly bot features are contact-sales.
Security engineer
Fastly docs read did not mention Web Bot Auth (checked 2026-11-04); Cloudflare verifies it.
Publisher operations
Cloudflare offers a free training opt-out; Fastly needs a sales contract.

What each does not do

Fastly

  • Bot Management is not available on the Essential platform for Next-Gen WAF (Bot Management docs).
  • No native per-crawl billing; charging relies on TollBit (Fastly-TollBit blog).
  • No public price for bot products (pricing page).

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

More: alternatives to Fastly, alternatives to Cloudflare.

Sources

Last reviewed 2026-11-04