Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Cloudflare vs DataDome

In short

Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. On per-purpose policy the rubric grades Cloudflare yes and DataDome partial, and on charging the rubric grades Cloudflare partial and DataDome yes.

The sourced difference

Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. Source, read 2026-11-04.

Side by side

QuestionCloudflareDataDome
Tells a verified agent from a spoof Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)
Yes
Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback
Per-purpose policy (train, search, answer, act) Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15
Partial
policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04)
Charging for access Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta
Yes
AI crawler and agent monetization through partners, from the Advanced plan
Lets signed commerce agents through Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)
Partial
TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01
Cost free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led published: from USD 3,830/month
Reporting Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance
Yes
agent traffic by intent, identity, geo, URL and endpoint type
Where it runs edge: enforced on Cloudflare's reverse proxy, no app code needed both: server-side module on web servers, CDNs or API gateways plus client-side tag
Points, rank8 of 10, rank 1 (tied)8 of 10, rank 1 (tied)
Price read 2026-11-04AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced.

Verdicts by reader

Platform engineer
If you already proxy through Cloudflare, its zone settings cover AI crawlers with no code. DataDome needs a server module plus a client tag.
Security engineer
DataDome scores each agent on identity strength and fraud history and blocks bad intent even when the agent is authenticated; Cloudflare focuses on identity and category.
Publisher operations
Cloudflare is the cheaper start for training opt-outs; DataDome is a paid bot and fraud stack with monetization partners.

What each does not do

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

DataDome

  • Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs).
  • Monetization is not in the Essentials plan; it starts at Advanced (pricing page).
  • No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04).

More: alternatives to Cloudflare, alternatives to DataDome.

Sources

Last reviewed 2026-11-04