Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Cloudflare vs Netlify

In short

Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. On verifies agents the rubric grades Cloudflare yes and Netlify not published, and on per-purpose policy the rubric grades Cloudflare yes and Netlify partial.

The sourced difference

Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. Source, read 2026-11-04.

Side by side

QuestionCloudflareNetlify
Tells a verified agent from a spoof Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)		 Not published
support (checked 2026-11-04): blocking matches user-agent strings only
Per-purpose policy (train, search, answer, act) Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15		 Partial
choose AI crawlers versus SEO or search crawlers from a preset list
Charging for access Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta		 Not published
support (checked 2026-11-04)
Lets signed commerce agents through Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)		 Not published
support (checked 2026-11-04)
Cost free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led published: platform plans from USD 0; extension price not stated
Reporting Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance		 Not published
no AI bot reporting found (checked 2026-11-04)
Where it runs edge: enforced on Cloudflare's reverse proxy, no app code needed edge: Netlify Edge Function and edge firewall rules
Points, rank8 of 10, rank 1 (tied)1 of 10, rank 15
Price read 2026-11-04AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).Free USD 0 (300 credits), Personal USD 9/month (1,000 credits), Pro USD 20/month (3,000 credits), Enterprise custom; Firewall Traffic Rules and basic rate limiting listed from Free; User Agent Blocker price not stated.

Verdicts by reader

Platform engineer
Netlify users can front sites with Cloudflare for per-purpose control.
Security engineer
User-agent matching blocks spoofers and real bots alike; verification needs Cloudflare or similar.
Publisher operations
Netlify gives no AI bot reporting (checked 2026-11-04); Cloudflare does.

What each does not do

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

Netlify

  • No verified-bot list, reverse DNS or Web Bot Auth checks; user agents can be spoofed (block AI crawlers docs).
  • Preset list is incomplete by Netlify's own statement (block AI crawlers docs, 2026-09-18).
  • No AI crawler monetization (docs read 2026-11-04).

More: alternatives to Cloudflare, alternatives to Netlify.

Sources

Last reviewed 2026-11-04