Vendor, ranked 1 (tied) of 15

Cloudflare

In short

Cloudflare lets a publisher treat search, training and agent crawlers differently on any plan, free included, and scores 8 of 10 points. Charging and commerce agents grade partial: per-request charging for crawlers is a private beta and the Visa and Mastercard agent rules are announced but unconfirmed, with Pro from USD 20 a month.

CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta. Vendor description of itself, checked against its documents on 2026-11-04.

Alternatives to Cloudflare · cloudflare.com

The rubric, with evidence

Cost

AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). Source, read 2026-11-04.

What Cloudflare does not do

• BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
• Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
• Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
• Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.
• Cloudflare's pay per crawl cannot price crawlers differently: one Charge price covers every crawler set to Charge (FAQ updated 2026-04-23).
• Cloudflare's managed robots.txt only states a preference; Cloudflare's own docs say crawlers may disregard it, so blocking needs AI Crawl Control rules.
• No managed ruleset for Trusted Agent Protocol or Agent Pay agents appears in Cloudflare's bot docs index read on 2026-11-05, a year after the October 2025 announcement.

Documented facts

Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim.

• Primary source AI Crawl Control (formerly AI Audit) is available on all plans and lets site owners set allow or block policies per AI crawler, review crawler activity and check which crawlers follow robots.txt. Source, read 2026-11-04, dated 2026-08-14.
• Primary source Under pay per crawl a crawler either sends payment intent in request headers and gets HTTP 200, or receives HTTP 402 with pricing, and Cloudflare acts as Merchant of Record. Source, read 2026-11-04, dated 2026-07-28.
• Primary source Disallow AI Training, launched 15 Sep 2026, publishes a no-training preference in robots.txt, keeps crawlers labelled Accountable allowed for search, and blocks training-only crawlers. Source, read 2026-11-04, dated 2026-09-15.
• Primary source To be labelled Accountable, a crawler operator must offer a training opt-out, a way to opt out of AI summaries, URL-level visibility into pages used for training, and an assurance that opting out of training does not affect search results. Source, read 2026-11-04, dated 2026-09-15.
• Primary source Cloudflare labels Googlebot, Applebot and Bingbot as Accountable mixed-use crawlers, and also labels the separate search and training crawlers of Amazon, Anthropic, Meta and OpenAI. Source, read 2026-11-04, dated 2026-09-15.
• Primary source Disallow AI Training does not yet pass a no-training preference to Bing through robots.txt; Cloudflare says Microsoft targets early 2027 for domain-level support. Source, read 2026-11-04, dated 2026-09-15.
• Primary source The Block AI Bots toggle is being replaced by separate Search, Training and Agent controls, and Managed Robots.txt is being replaced by Bot Preference Sync; existing Training settings of Block move to Disallow AI Training. Source, read 2026-11-04, dated 2026-09-15.
• Primary source Verified bots must validate with Web Bot Auth or IP validation; since 1 Jul 2026 signed agents sit under the verified bot umbrella with a Direct versus Intermediary metadata field. Source, read 2026-11-04, dated 2026-07-01.
• Primary source Cloudflare's behavior taxonomy has 11 entries: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social / Link Preview, Feed Fetching, and Monitoring & Operations. Source, read 2026-11-04, dated 2026-07-01.
• Primary source BotBase, the directory of known bots and agents, is available only to Enterprise Bot Management customers and is a visibility tool; blocking needs Security rules or the AI traffic options. Source, read 2026-11-04, dated 2026-10-08.
• Primary source Bots register for Web Bot Auth by hosting a signed key directory at /.well-known/http-message-signatures-directory; Cloudflare accepts valid Ed25519 keys and currently does no nonce validation. Source, read 2026-11-04, dated 2026-10-08.
• Primary source Visa's Trusted Agent Protocol and Mastercard's Agent Pay both use Web Bot Auth, tagging requests agent-browser-auth or agent-payer-auth, and Cloudflare said it would ship managed rulesets to admit registered agents. Source, read 2026-11-04, dated 2025-10-24.

Dated news

• 2026-09-15Cloudflare launches Disallow AI Training and the Accountable crawler label. Cloudflare added a Disallow AI Training option to its Training control. It writes a no-training preference into robots.txt, keeps Accountable mixed-use crawlers such as Googlebot, Applebot and Bingbot allowed for search, and blocks training-only crawlers. Block AI Bots and Managed Robots.txt are being deprecated. Source.
• 2026-09-28Coverage: Cloudflare asks Google to skip AI training without blocking search. Trade coverage of the Disallow AI Training rollout, describing how the setting relies on robots.txt tokens such as Google-Extended so pages stay in search while opting out of model training, and noting the Bing gap until Microsoft adds support. Source.

Compared with

• Cloudflare vs DataDome: Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan.
• Akamai vs Cloudflare: Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta).
• TollBit vs Cloudflare: TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network.
• Cloudflare vs AWS WAF Bot Control: AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record.
• Cloudflare vs Vercel: Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt.
• Fastly vs Cloudflare: Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls.
• Cloudflare vs Netlify: Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers.
• Cloudflare vs HUMAN Security: Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action.
• Arcjet vs Cloudflare: Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan.
• Imperva vs Cloudflare: Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone.
• Skyfire vs Cloudflare: Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory.

Last reviewed 2026-11-05