--- title: "Cloudflare: bot management profile | Agent Ready Crawlers" description: "Cloudflare: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15." canonical: https://agentreadycrawlers.com/vendors/cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Cloudflare: bot management profile | Agent Ready Crawlers > Cloudflare: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. Source: https://agentreadycrawlers.com/vendors/cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 1 (tied) of 15 # Cloudflare In short Cloudflare lets a publisher treat search, training and agent crawlers differently on any plan, free included, and scores 8 of 10 points. Charging and commerce agents grade partial: per-request charging for crawlers is a private beta and the Visa and Mastercard agent rules are announced but unconfirmed, with Pro from USD 20 a month. CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/) · [cloudflare.com](https://www.cloudflare.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Based in United States Homepage [cloudflare.com](https://www.cloudflare.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Verified bots must validate with Web Bot Auth or IP validation; signed agents tracked with a Direct versus Intermediary field. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Yes | separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Training control has Allow, Block and Disallow AI Training; Search and Agent are set separately per zone. Source, read 2026-11-04. | | Charging for access | Partial | Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Joining requires contacting Cloudflare; not self-serve. Cloudflare's overview page calls it a private beta. Source, read 2026-11-04. | | Lets signed commerce agents through | Partial | Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Blog describes planned managed rulesets that admit Visa or Mastercard registered agents. Source, read 2026-11-04. | | Cost | Not graded | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | Plan table lists Free, Pro and Business prices; Enterprise is custom. Source, read 2026-11-04. | | Reporting | Yes | AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Docs list Analyze AI traffic and Track robots.txt features. Source, read 2026-11-04. | | Where it runs | Not graded | edge: enforced on Cloudflare's reverse proxy, no app code needed | Controls are set per zone under Security Settings. Source, read 2026-11-04. | ## Cost AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). [Source](https://www.cloudflare.com/plans/network-cdn.md), read 2026-11-04. ## What Cloudflare does not do - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. - Cloudflare's pay per crawl cannot price crawlers differently: one Charge price covers every crawler set to Charge (FAQ updated 2026-04-23). - Cloudflare's managed robots.txt only states a preference; Cloudflare's own docs say crawlers may disregard it, so blocking needs AI Crawl Control rules. - No managed ruleset for Trusted Agent Protocol or Agent Pay agents appears in Cloudflare's bot docs index read on 2026-11-05, a year after the October 2025 announcement. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source AI Crawl Control (formerly AI Audit) is available on all plans and lets site owners set allow or block policies per AI crawler, review crawler activity and check which crawlers follow robots.txt. [Source](https://developers.cloudflare.com/ai-crawl-control/), read 2026-11-04, dated 2026-08-14. - Primary source Under pay per crawl a crawler either sends payment intent in request headers and gets HTTP 200, or receives HTTP 402 with pricing, and Cloudflare acts as Merchant of Record. [Source](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/), read 2026-11-04, dated 2026-07-28. - Primary source Disallow AI Training, launched 15 Sep 2026, publishes a no-training preference in robots.txt, keeps crawlers labelled Accountable allowed for search, and blocks training-only crawlers. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source To be labelled Accountable, a crawler operator must offer a training opt-out, a way to opt out of AI summaries, URL-level visibility into pages used for training, and an assurance that opting out of training does not affect search results. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source Cloudflare labels Googlebot, Applebot and Bingbot as Accountable mixed-use crawlers, and also labels the separate search and training crawlers of Amazon, Anthropic, Meta and OpenAI. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source Disallow AI Training does not yet pass a no-training preference to Bing through robots.txt; Cloudflare says Microsoft targets early 2027 for domain-level support. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source The Block AI Bots toggle is being replaced by separate Search, Training and Agent controls, and Managed Robots.txt is being replaced by Bot Preference Sync; existing Training settings of Block move to Disallow AI Training. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source Verified bots must validate with Web Bot Auth or IP validation; since 1 Jul 2026 signed agents sit under the verified bot umbrella with a Direct versus Intermediary metadata field. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04, dated 2026-07-01. - Primary source Cloudflare's behavior taxonomy has 11 entries: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social / Link Preview, Feed Fetching, and Monitoring & Operations. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04, dated 2026-07-01. - Primary source BotBase, the directory of known bots and agents, is available only to Enterprise Bot Management customers and is a visibility tool; blocking needs Security rules or the AI traffic options. [Source](https://developers.cloudflare.com/bots/botbase/), read 2026-11-04, dated 2026-10-08. - Primary source Bots register for Web Bot Auth by hosting a signed key directory at /.well-known/http-message-signatures-directory; Cloudflare accepts valid Ed25519 keys and currently does no nonce validation. [Source](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/), read 2026-11-04, dated 2026-10-08. - Primary source Visa's Trusted Agent Protocol and Mastercard's Agent Pay both use Web Bot Auth, tagging requests agent-browser-auth or agent-payer-auth, and Cloudflare said it would ship managed rulesets to admit registered agents. [Source](https://blog.cloudflare.com/secure-agentic-commerce/), read 2026-11-04, dated 2025-10-24. ## Dated news - 2026-09-15**Cloudflare launches Disallow AI Training and the Accountable crawler label**. Cloudflare added a Disallow AI Training option to its Training control. It writes a no-training preference into robots.txt, keeps Accountable mixed-use crawlers such as Googlebot, Applebot and Bingbot allowed for search, and blocks training-only crawlers. Block AI Bots and Managed Robots.txt are being deprecated. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/). - 2026-09-28**Coverage: Cloudflare asks Google to skip AI training without blocking search**. Trade coverage of the Disallow AI Training rollout, describing how the setting relies on robots.txt tokens such as Google-Extended so pages stay in search while opting out of model training, and noting the Bing gap until Microsoft adds support. [Source](https://winbuzzer.com/2026/09/28/cloudflare-ask-google-skip-ai-training-without-blocking-search-a002-xcxwbn/). ## Compared with - [Cloudflare vs DataDome](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome/): Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. - [Akamai vs Cloudflare](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare/): Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). - [TollBit vs Cloudflare](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/): TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. - [Cloudflare vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control/): AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. - [Cloudflare vs Vercel](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid/): Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. - [Fastly vs Cloudflare](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare/): Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. - [Cloudflare vs Netlify](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify/): Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. - [Cloudflare vs HUMAN Security](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security/): Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. - [Arcjet vs Cloudflare](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare/): Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. - [Imperva vs Cloudflare](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare/): Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. - [Skyfire vs Cloudflare](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare/): Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. ## What Cloudflare does for this layer ### From the vendor's own pages Cloudflare's pay per crawl FAQ lists /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json as always free to crawl, and says error responses are never billed. ([developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/index.md), dated 2026-04-23) A crawler that wants to pay through Cloudflare must sign with Web Bot Auth, follow the verified bots policy and be submitted through the Bot Submission Form; until then it receives HTTP 402 with a crawler-price header such as "USD 0.01". ([developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler/index.md), dated 2026-07-01) Cloudflare's Bot Preference Sync prepends managed robots.txt directives and the Content Signals Policy only when at least one Search, Agent or Training policy is set to Block, Block on pages with ads, or, for Training, Disallow. ([developers.cloudflare.com](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), dated 2026-11-02) Cloudflare's robots.txt documentation states that robots.txt compliance is voluntary and sends site owners to AI Crawl Control to enforce crawler blocks. ([developers.cloudflare.com](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), dated 2026-11-02) Cloudflare published the Content Signals Policy on 24 Sep 2025 under a CC0 licence, defining three signals for robots.txt: search, ai-input and ai-train. ([blog.cloudflare.com](https://blog.cloudflare.com/content-signals-policy/), dated 2025-09-24) Cloudflare Browser Run, formerly Browser Rendering, signs its requests with Web Bot Auth and adds headers that cannot be removed (cf-brapi-request-id, cf-brapi-devtools, cf-biso-devtools); its crawl endpoint uses the fixed user agent CloudflareBrowserRenderingCrawler/1.0. ([developers.cloudflare.com](https://developers.cloudflare.com/browser-run/reference/automatic-request-headers/), dated 2026-10-30) Cloudflare's 24 Oct 2025 post says Visa and Mastercard will each host a key directory for their registered agents, and that commerce agents must include a nonce in the signature to guard against replay. ([blog.cloudflare.com](https://blog.cloudflare.com/secure-agentic-commerce/), dated 2025-10-24) Cloudflare's plan page grades bot mitigation by tier: simple bots on Free, easy-to-detect bots on Pro, sophisticated bots with basic bot analytics on Business, and anomaly detection with advanced bot analytics on Contract. ([cloudflare.com](https://www.cloudflare.com/plans/network-cdn.md)) Cloudflare announced on 15 Jan 2026 that it had acquired Human Native, which its release calls "an AI data marketplace that connects creators and AI developers", to help publishers prepare and price content for AI companies willing to pay. ([cloudflare.com](https://www.cloudflare.com/press/press-releases/2026/cloudflare-strengthens-content-offering-to-ai-companies-with-acquisition-of-human-native/), dated 2026-01-15) ### Claims the vendor makes about itself At the Content Signals launch Cloudflare said its customers had turned on managed robots.txt for more than 3.8 million domains, and that it would not serve an ai-input signal on their behalf. ([blog.cloudflare.com](https://blog.cloudflare.com/content-signals-policy/), dated 2025-09-24) ### What our crawl saw A GET with the crawl user agent on 2026-11-05 to the Browser Run key directory at web-bot-auth.cloudflare-browser-rendering-085.workers.dev returned HTTP 200, media type application/http-message-signatures-directory+json, and three Ed25519 keys. ([web-bot-auth.cloudflare-browser-rendering-085.workers.dev](https://web-bot-auth.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | AI Crawl Control | Per-crawler allow or block policies, AI traffic analytics and robots.txt compliance tracking. | included in every plan Overview page says available on all plans. | USD 0 | 2026-11-05 | | Pay per crawl | HTTP 402 charging for verified AI crawlers, Stripe payouts, Cloudflare as Merchant of Record; private beta. | site owner sets one per-request price Example header in the docs: crawler-price USD 0.01. | Set by the site owner | 2026-11-05 | | Bot Management (Contract plan) | Bot scores, anomaly detection, custom CAPTCHAs and advanced bot analytics; BotBase directory for Enterprise. | contract Pro USD 20 a month billed annually (USD 25 monthly); Business USD 200 a month billed annually (USD 250 monthly). | Custom | 2026-11-05 | | Browser Run (formerly Browser Rendering) | Headless browser service whose requests are signed with Web Bot Auth; Cloudflare is the operator of this agent. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Bot Preference Sync writes managed robots.txt directives from the Search, Agent and Training policies. (developers.cloudflare.com) | | Web Bot Auth | Supports | Pay per crawl requires crawlers to sign with Web Bot Auth; Browser Run signs its own requests. (developers.cloudflare.com) | | Content Signals | Supports | Cloudflare wrote the Content Signals Policy (CC0) and serves it in managed robots.txt. (blog.cloudflare.com) | | Published IP list | Supports | The bots docs index lists IP validation: matching request IPs against published bot IP lists. (developers.cloudflare.com) | | Reverse DNS check | No statement | The bots docs index (read 2026-11-05) lists IP validation and Web Bot Auth, not reverse DNS. | | Cloudflare verified bots | Supports | Cloudflare runs the verified bots program; paying crawlers must be added to it. (developers.cloudflare.com) | | Visa Trusted Agent Protocol | Announced | Cloudflare said in Oct 2025 it would create managed rulesets for agents using the Trusted Agent Protocol. (blog.cloudflare.com) | | Mastercard Agent Pay | Announced | The same Oct 2025 post announced managed rulesets for agents using Agent Pay. (blog.cloudflare.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Browser Run Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 3 Ed25519 keys | GET with the crawl user agent | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Cloudflare Browser Rendering](https://agentreadycrawlers.com/crawlers/cloudflare-browser-rendering/) no token - [Alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/) graded on the rubric ## Sources - [cloudflare.com: cloudflare.com](https://www.cloudflare.com) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [developers.cloudflare.com: ai crawl control](https://developers.cloudflare.com/ai-crawl-control/) read 2026-11-04 - [developers.cloudflare.com: botbase](https://developers.cloudflare.com/bots/botbase/) read 2026-11-04 - [developers.cloudflare.com: web bot auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) read 2026-11-04 - [winbuzzer.com: cloudflare ask google skip ai training without blocking search a002 xcxwbn](https://winbuzzer.com/2026/09/28/cloudflare-ask-google-skip-ai-training-without-blocking-search-a002-xcxwbn/) read 2026-11-04 - [Pay Per Crawl FAQ · Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/index.md) read 2026-11-05 - [Verify your AI crawler · Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler/index.md) read 2026-11-05 - [robots.txt setting · Cloudflare bot solutions docs](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) read 2026-11-05 - [Giving users choice with Cloudflare’s new Content Signals Policy | Cloudflare Blog](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-05 - [Automatic request headers · Cloudflare Browser Run docs](https://developers.cloudflare.com/browser-run/reference/automatic-request-headers/) read 2026-11-05 - [Cloudflare Browser Run Web Bot Auth key directory (JSON)](https://web-bot-auth.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory) read 2026-11-05 - [Cloudflare Strengthens Content Offering to AI Companies with Acquisition of Human Native](https://www.cloudflare.com/press/press-releases/2026/cloudflare-strengthens-content-offering-to-ai-companies-with-acquisition-of-human-native/) read 2026-11-05 - [Cloudflare bot solutions docs index (llms.txt)](https://developers.cloudflare.com/bots/llms.txt) read 2026-11-05 Last reviewed 2026-11-05