Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Arcjet vs Cloudflare

In short

Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. On verifies agents the rubric grades Arcjet partial and Cloudflare yes, and on per-purpose policy the rubric grades Arcjet partial and Cloudflare yes. For publisher operations: Cloudflare can write your robots.txt preferences; Arcjet cannot.

The sourced difference

Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. Source, read 2026-11-04.

Side by side

QuestionArcjetCloudflare
Tells a verified agent from a spoof Partial
IP data and reverse DNS for known bots; Web Bot Auth not mentioned in docs (checked 2026-11-04)		 Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)
Per-purpose policy (train, search, answer, act) Partial
allow or deny by bot name or category (e.g. CATEGORY:AI) per route		 Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15
Charging for access Not published
support (checked 2026-11-04)		 Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta
Lets signed commerce agents through Not published
Visa TAP or Mastercard Agent Pay support (checked 2026-11-04)		 Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)
Cost published: from USD 25/month plus USD 5 per 1M web requests free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led
Reporting Partial
request decision logs with 1 to 24 hours retention on self-serve plans		 Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance
Where it runs app: SDK calls in application code edge: enforced on Cloudflare's reverse proxy, no app code needed
Points, rank3 of 10, rank 11 (tied)8 of 10, rank 1 (tied)
Price read 2026-11-04Individual USD 25/month, Startup USD 299/month, Enterprise custom; usage USD 5 per 1M web requests and USD 50 per 1M agent requests; free plan 10,000 requests/month (hard stop) after a 15-day trial.AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).

Verdicts by reader

Platform engineer
Arcjet suits apps not behind a CDN; Cloudflare needs DNS on Cloudflare.
Security engineer
Cloudflare verifies Web Bot Auth; Arcjet uses IP and reverse DNS.
Publisher operations
Cloudflare can write your robots.txt preferences; Arcjet cannot.

What each does not do

Arcjet

  • No Web Bot Auth or cryptographic agent verification in bot protection docs (checked 2026-11-04).
  • Does not run at the CDN edge; requests reach the app before Arcjet decides (concepts page).
  • No AI crawler monetization or pay-per-crawl (docs read 2026-11-04).

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

More: alternatives to Arcjet, alternatives to Cloudflare.

Sources

Last reviewed 2026-11-04