Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Head-to-head

Cloudflare vs HUMAN Security

In short

Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. On commerce agents the rubric grades Cloudflare partial and HUMAN Security not published. For a security engineer: HUMAN grades trust per agent; Cloudflare separates verified bots and signed agents.

The sourced difference

Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. Source, read 2026-11-04.

Side by side

QuestionCloudflareHUMAN Security
Tells a verified agent from a spoof Yes
Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase)
Yes
per-request cryptographic keys (e.g. ChatGPT's published key), plus behavior and private signals; Web Bot Auth not named on pages read
Per-purpose policy (train, search, answer, act) Yes
separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15
Yes
per-action categories (content, engage, account creation, checkout, rate limit) by trust level
Charging for access Partial
Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta
Partial
through scraping monetization partners (TollBit logo on product page); no native charging found
Lets signed commerce agents through Partial
Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04)
Not published
Visa TAP or Mastercard Agent Pay support (checked 2026-11-04)
Cost free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led no public price; sales-led
Reporting Yes
AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance
Yes
user and session insights on agent actions, monthly State of Agentic Traffic benchmark
Where it runs edge: enforced on Cloudflare's reverse proxy, no app code needed platform: Sightline integrates with the customer's stack; exact deployment points not confirmed on pages read
Points, rank8 of 10, rank 1 (tied)7 of 10, rank 5
Price read 2026-11-04AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta).no public price; sales-led (checked 2026-11-04)

Verdicts by reader

Platform engineer
Cloudflare is a WAF rule on detection ID; HUMAN is a console policy.
Security engineer
HUMAN grades trust per agent; Cloudflare separates verified bots and signed agents.
Publisher operations
Cloudflare offers charging and a free tier; HUMAN does neither publicly.

What each does not do

Cloudflare

  • BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08).
  • Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15).
  • Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs).
  • Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta.

HUMAN Security

  • Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02).
  • No native pay-per-crawl; monetization is through partner tools (AgenticTrust page).
  • No published price list (package plans page).

More: alternatives to Cloudflare, alternatives to HUMAN Security.

Sources

Last reviewed 2026-11-04