# Agent Ready Crawlers: every page > The Crawler Index: 70 AI crawlers and agents with exact tokens, verification and block rates from 13,071 crawled sites, plus the vendors at the door. Edition 2026-11-04. 305 pages. Index: https://agentreadycrawlers.com/llms.txt --- title: "The Crawler Index" description: "The Crawler Index: 70 AI crawlers and agents with exact tokens, verification and block rates from 13,071 crawled sites, plus the vendors at the door." canonical: https://agentreadycrawlers.com/ site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Home" --- # The Crawler Index > The Crawler Index: 70 AI crawlers and agents with exact tokens, verification and block rates from 13,071 crawled sites, plus the vendors at the door. Source: https://agentreadycrawlers.com/ Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The Crawler Index In short 70 AI crawlers and agents from 32 operators, each with its exact user-agent string, robots.txt token, purpose and verification method. 35 of them publish nothing a site can check. The block rate column comes from a crawl of 13,071 domains on 4 November 2026. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [All 70 rows](https://agentreadycrawlers.com/crawlers/) The first fifteen of 70 rows. Block rate: share of top-10,000 sites with a parsed robots.txt that disallow the token at the root. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | GPTBot | Training | Yes | IP list | 18.5% | Profile | | OpenAI | OAI-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | OpenAI | ChatGPT-User | Live answer fetch | May not apply | IP list | 9.7% | Profile | | Anthropic | ClaudeBot | Training | Yes | IP list | 17.6% | Profile | | Anthropic | Claude-User | Live answer fetch | Yes | IP list | 8.1% | Profile | | Anthropic | Claude-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Perplexity | PerplexityBot | Search index | Yes | IP list | 11.2% | Profile | | Perplexity | Perplexity-User | Live answer fetch | May not apply | IP list | 8.3% | Profile | | Google | Google-Extended | Training | Yes | None published | 16.0% | Profile | | Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile | | Apple | Applebot-Extended | Training | Yes | None published | 14.8% | Profile | | Microsoft | bingbot | Search index | Yes | Reverse DNS | 3.1% | Profile | | Amazon | Amazonbot | Training | Yes | IP list | 15.6% | Profile | | Meta | meta-externalagent | Training | Yes | None published | 16.0% | Profile | | Common Crawl | CCBot | Training | Yes | Reverse DNS | 19.4% | Profile | [Open the full index](https://agentreadycrawlers.com/crawlers/), or read how the columns are defined on [the method page](https://agentreadycrawlers.com/method/). ## The Block Rate Report Site names the tokenRefused only by a \* group Share of top-10,000 sites (5,938 with a parsed robots.txt) that refuse each token at the root. Crawl of 2026-11-04, 13,071 rows. \*The crawl's own token, a control: its bar is the wildcard baseline. **CCBot** is refused by 19.4 percent of the sites, the most of any token in the crawl. **GPTBot** is refused by 18.5 percent and **Googlebot** by 2.8 percent. Between 2.3 and 4.1 percent of sites refuse each token only because their wildcard group refuses everything (the light part of every bar). The rest of each bar is a site that named the token on purpose. 12.9 percent of the 10,565 home pages fetched answered 401 or 403 to the crawl's labelled user agent; among the top 10,000 it was 17.2 percent. Ora's lab found (2026-10-19) that a 403 to agent user agents left a plain-fetch and a JavaScript client at 0 of 5 retrieved answers; that is a vendor finding. To see what your own site answers, run the check we recommend, described [below](#check-we-recommend). [The full report](https://agentreadycrawlers.com/block-rates/) and [the 2026-11-04 edition with three charts](https://agentreadycrawlers.com/block-rates/2026-11-04/). [Download the data](https://agentreadycrawlers.com/data/). Table behind the chart | Token | Named | Via * | Refused | | --- | --- | --- | --- | | CCBot | 15.7% | 3.7% | 19.4% | | Bytespider | 15.4% | 3.8% | 19.2% | | GPTBot | 15.2% | 3.3% | 18.5% | | ClaudeBot | 14.0% | 3.6% | 17.6% | | Google-Extended | 12.6% | 3.4% | 16.0% | | Meta-ExternalAgent | 12.3% | 3.7% | 16.0% | | Diffbot | 11.9% | 3.8% | 15.7% | | omgili | 11.8% | 3.9% | 15.7% | | Amazonbot | 12.0% | 3.6% | 15.6% | | anthropic-ai | 11.7% | 3.8% | 15.5% | | cohere-ai | 11.6% | 3.9% | 15.5% | | Applebot | 11.6% | 3.2% | 14.8% | | Applebot-Extended | 11.6% | 3.2% | 14.8% | | PetalBot | 10.3% | 3.8% | 14.1% | | PerplexityBot | 7.9% | 3.3% | 11.2% | | Google-CloudVertexBot | 7.1% | 3.8% | 10.9% | | YouBot | 6.4% | 3.9% | 10.3% | | FacebookBot | 6.4% | 3.8% | 10.2% | | Timpibot | 5.8% | 4.0% | 9.8% | | ChatGPT-User | 6.3% | 3.4% | 9.7% | | ImagesiftBot | 5.5% | 4.0% | 9.5% | | Meta-ExternalFetcher | 4.7% | 3.8% | 8.5% | | Perplexity-User | 4.6% | 3.7% | 8.3% | | Claude-User | 4.5% | 3.6% | 8.1% | | DuckAssistBot | 4.3% | 3.8% | 8.1% | | Claude-SearchBot | 4.1% | 3.6% | 7.7% | | MistralAI-User | 3.8% | 3.9% | 7.7% | | OAI-SearchBot | 4.4% | 3.3% | 7.7% | | agent-readiness-crawl* | 0.0% | 4.2% | 4.2% | | MicrosoftPreview | 0.0% | 4.0% | 4.0% | | Bingbot | 0.4% | 2.7% | 3.1% | | Googlebot | 0.5% | 2.3% | 2.8% | ## The check we recommend: Ora The Block Rate Report says what sites write in robots.txt and how their home pages answered one crawler. It cannot say what yours answers to the agents that matter to you. For that we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home), whose scan reads a public URL and scores it from 0 to 100 on four layers. Run it before and after any change to robots.txt or the WAF. Discovery layer 16 checks in Ora's catalog (read 2026-11-05, contract 1.38.0). One is **robots.txt AI crawler policy**, which asks whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Ora says blocking training crawlers such as GPTBot and ClaudeBot never lowers the score. Access layer 42 checks. One is **Not blocked by bot detection**: Ora fetches the home page as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot with each vendor's published user agent, and again as a normal browser. An agent refused with a 401, 402, 403, 451 or a bot challenge, while a normal browser visit gets in, counts as blocked (Ora's methodology, read 2026-11-05). Journey A real agent attempts a curated task against your domain and the run is recorded ([Ora's llms.txt](https://ora.ai/llms.txt?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home), read 2026-11-05). A status code shows whether a request gets in; a run shows whether an agent can then finish something. Scale Ora's stats endpoint reports 145,915 sites scanned and an average score of 44 out of 100 on 2026-11-05. Both are vendor figures. In your logs The scan shows up as [OraBot](https://agentreadycrawlers.com/crawlers/orabot/), which signs its requests with Web Bot Auth. It also sends a few unsigned home page requests that carry other crawlers' user agents, such as GPTBot and ClaudeBot, to see how you answer them. Limits Ora reads public URLs only, and a refusal it records comes from its own network, so an agent on another network may get through. Ora says the same scan run twice converges within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home)); the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home), read 2026-11-05). Quick alternative [Cloudflare's scanner](https://isitagentready.com/) takes a URL and shows no sign-in step or price on its page (read 2026-11-05). Its Bot Access Control category covers AI bot rules in robots.txt, Content Signals and Web Bot Auth, and its check list names no request made with an agent's user agent. [How to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/), step by step, with the curl commands. ## What changed this month - 2026-11-05Ora's scanner joins the Index as [OraBot](https://agentreadycrawlers.com/crawlers/orabot/), which signs its requests with Web Bot Auth; the Verification Ledger gains its row. - 2026-11-05[How to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/), with the Ora check counts read from its catalog the same day. - 2026-11-02[Signed-Agent-Metadata header proposed for Web Bot Auth](https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-auth/) - 2026-10-29[Vercel splits AI crawlers from AI assistants](https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistants/) - 2026-10-29[Web Bot Auth working-group draft reaches -01](https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01/) - 2026-10-27[OpenAI updates the ChatGPT-User IP list](https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list/) - 2026-10-22[IETF drafts propose a common IP-range format and crawler best practices](https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-bes/) - 2026-10-21[Anthropic's Claude Duet starts browsing with a Claude-Agent token](https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent/) - 2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) - 2026-10-15[DataDome Q3 report: 25 billion AI agent requests](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) ## The vendors at the door Fifteen bot management, agent identity and monetisation vendors on one rubric. Grades are for features the vendor documents, read 2026-11-04. "Not published" means we found nothing in its documents, not that the feature is absent. [Ranking and method](https://agentreadycrawlers.com/vendors/ranking/). | Vendor | Verifies agents | Per-purpose policy | Charging | Commerce agents | Cost, read 2026-11-04 | | --- | --- | --- | --- | --- | --- | | Akamai | Yes | Partial | Partial | Yes | No public price; sales-led | | AWS WAF Bot Control | Yes | Yes | Yes | Not published | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | Yes | Yes | Partial | Partial | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | Yes | Partial | Yes | Partial | From USD 3,830/month | | HUMAN Security | Yes | Yes | Partial | Not published | No public price; sales-led | | Skyfire | Yes | Not its job | Yes | Yes | Not published (checked 2026-11-04) | | TollBit | Partial | Partial | Yes | Not published | Free for publishers to start | | Vercel | Yes | Yes | Not published | Not published | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | Yes | Partial | Not published | Not published | No public price; sales-led | | Fastly | Partial | Partial | Partial | Partial | No public price; sales-led | | Arcjet | Partial | Partial | Not published | Not published | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | Not its job | Partial | Not published | Partial | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | Partial | Not published | Not published | Not published | No public price; sales-led | | Stytch IsAgent | Partial | Not published | Not published | Not published | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | Not published | Partial | Not published | Not published | Platform plans from USD 0; extension price not stated | ## The robots.txt builder Choose which purposes to allow and refuse, then copy the file. It is built from the tokens in the index, so it names only agents that publish a token. The default refuses training and allows everything else; the file below is that default. [Notes on what it cannot do](https://agentreadycrawlers.com/robots-builder/). Your robots.txt \# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: \* Allow: / ## Query the index as data The rows on this page also answer over an MCP server at `/mcp`, a REST API under `/api/v1/` and an NLWeb `/ask` endpoint. All three are read-only and take no key. The [API and MCP server documentation](https://agentreadycrawlers.com/developers) lists the five tools and the limits. The vendor list is a separate download: [73 vendors](https://agentreadycrawlers.com/vendors/) as CSV and JSON. ```bash curl -s 'https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot&stratum=tranco' ``` ## Earlier this autumn - 2026-10-01[DataDome joins Experian Agent Trust](https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trust/) - 2026-09-17[Arcjet launches Agent Runtime Security](https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-security/) - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) - 2026-09-08[Meta launches Muse, a personal agent that does not identify itself](https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i/) - 2026-09-02[HUMAN publishes Agent Trust Levels](https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levels/) - 2026-08-28[Cloudflare opens BotBase for Operators](https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators/) - 2026-08-19[Google retires the Google-NotebookLM user agent](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/) [All news](https://agentreadycrawlers.com/news/). Machine-readable: [llms.txt](https://agentreadycrawlers.com/llms.txt), [llms-full.txt](https://agentreadycrawlers.com/llms-full.txt), [sitemap.xml](https://agentreadycrawlers.com/sitemap.xml), [feed.xml](https://agentreadycrawlers.com/feed.xml). The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home) read 2026-11-04 - [Method: index, block rates, vendor rubric (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Ora's check catalog, GET /api/checks (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home) read 2026-11-05 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=home) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "About Agent Ready Crawlers | Agent Ready Crawlers" description: "About Agent Ready Crawlers: what the publication covers, who it is written for, how the Crawler Index, the Block Rate Report and the vendor ranking are made." canonical: https://agentreadycrawlers.com/about site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Publication" --- # About Agent Ready Crawlers | Agent Ready Crawlers > About Agent Ready Crawlers: what the publication covers, who it is written for, how the Crawler Index, the Block Rate Report and the vendor ranking are made. Source: https://agentreadycrawlers.com/about Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # About In short Agent Ready Crawlers is a reference publication on the AI crawlers and agents that visit websites, and the products that decide whether they get in. It publishes 70 crawler and agent profiles, 15 vendor profiles, a crawl of 13,071 sites and a ledger of verification methods. Every fact has a source and a date. Agent Ready Crawlers is written for the people who own robots.txt and the WAF: web platform engineers, security engineers, technical SEOs, publishers' operations leads and the ecommerce engineer who has to let a card network's agents through while keeping scrapers out. It uses exact tokens, exact IP ranges and exact dates, and it names no alarm. ## What it publishes - [The Crawler Index](https://agentreadycrawlers.com/crawlers/): 70 rows, one profile page each, with the operator's own statement on robots.txt and the way to verify the agent. - [The Block Rate Report](https://agentreadycrawlers.com/block-rates/): the share of sites that disallow each token and the share of home pages that answer 401 or 403, from a crawl. - [The Verification Ledger](https://agentreadycrawlers.com/verification/): which operators publish IP lists, reverse DNS patterns and key directories, and which publish nothing. - [15 vendors](https://agentreadycrawlers.com/vendors/ranking/) on a published rubric, with head-to-heads and alternatives. - [A robots.txt builder](https://agentreadycrawlers.com/robots-builder/), [14 guides](https://agentreadycrawlers.com/guides/), [a glossary](https://agentreadycrawlers.com/glossary/), [dated news](https://agentreadycrawlers.com/news/) and [a FAQ](https://agentreadycrawlers.com/faq/). ## How it is made Rows are read from operators' own pages and dated. The block rates come from one crawl of 13,071 domains on 2026-11-04, parsed the way RFC 9309 describes. Vendors are graded on what they document, with the evidence beside each grade. [The method page](https://agentreadycrawlers.com/method/) has the detail, including what our own crawl does and where Ora's scan fits on this layer. The site is static HTML. Every page has a markdown twin, [llms.txt](https://agentreadycrawlers.com/llms.txt) and [llms-full.txt](https://agentreadycrawlers.com/llms-full.txt) list the whole site, and its own robots.txt allows the AI crawlers it writes about. ## Sources - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Editorial policy (this site)](https://agentreadycrawlers.com/editorial-policy/) read 2026-11-04 - [Corrections (this site)](https://agentreadycrawlers.com/corrections/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Block Rate Report: who refuses which crawler | Agent Ready Crawlers" description: "The Block Rate Report: the share of top-10,000 sites that disallow each AI crawler token and the share of home pages answering 401 or 403. 13,071 rows." canonical: https://agentreadycrawlers.com/block-rates site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Block rates" --- # Block Rate Report: who refuses which crawler | Agent Ready Crawlers > The Block Rate Report: the share of top-10,000 sites that disallow each AI crawler token and the share of home pages answering 401 or 403. 13,071 rows. Source: https://agentreadycrawlers.com/block-rates Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Edition 2026-11-04 # The Block Rate Report In short Of 5,938 top-10,000 sites with a parsed robots.txt, 18.5 percent refuse GPTBot at the root and 17.6 percent refuse ClaudeBot, against 2.8 percent for Googlebot. 17.2 percent of their home pages answered 401 or 403 to the crawl. Crawl of 13,071 domains, 4 November 2026. 8,624 robots.txt files parsed, of 13,071 domains crawled 18.5% of 5,938 top-10,000 sites refuse GPTBot 17.6% refuse ClaudeBot 17.2% of top-10,000 home pages answered 401 or 403 ## By token Site names the tokenRefused only by a \* group Share of top-10,000 sites with a parsed robots.txt that refuse each token at the root. \*The crawl's own token, a control: its bar is the wildcard baseline. CCBot is the most refused token at 19.4 percent, and Googlebot the least refused at 2.8 percent. Across all 13,071 crawled domains every rate is lower: GPTBot 13.9 percent, ClaudeBot 13.1 percent. The legacy token anthropic-ai, which Anthropic no longer lists on its crawler page, is still refused by 15.5 percent of the top 10,000, close to ClaudeBot at 17.6 percent. Files written for one name keep working for the other only if the site names both. Refusing a token states a preference. User-triggered fetchers such as ChatGPT-User and Perplexity-User are, by their operators' own statements, allowed to ignore it; see [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/). | Token | Refused, top 10,000 | Named | Via * | Sites | Refused, all crawled | Sites | | --- | --- | --- | --- | --- | --- | --- | | GPTBot | 18.5% | 15.2% | 3.3% | 5,938 | 13.9% | 8,624 | | OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5,938 | 5.4% | 8,624 | | ChatGPT-User | 9.7% | 6.3% | 3.3% | 5,938 | 6.9% | 8,624 | | ClaudeBot | 17.6% | 14.0% | 3.5% | 5,938 | 13.1% | 8,624 | | Claude-User | 8.1% | 4.5% | 3.6% | 5,938 | 5.7% | 8,624 | | Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5,938 | 5.5% | 8,624 | | anthropic-ai | 15.5% | 11.7% | 3.8% | 5,938 | 11.6% | 8,624 | | PerplexityBot | 11.2% | 7.9% | 3.4% | 5,938 | 7.9% | 8,624 | | Perplexity-User | 8.3% | 4.6% | 3.7% | 5,938 | 5.9% | 8,624 | | Google-Extended | 16.0% | 12.6% | 3.4% | 5,938 | 12.1% | 8,624 | | Googlebot | 2.8% | 0.5% | 2.3% | 5,938 | 2.1% | 8,624 | | Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 5,938 | 8.3% | 8,624 | | Applebot | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 | | Applebot-Extended | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 | | Bingbot | 3.1% | 0.4% | 2.7% | 5,938 | 2.2% | 8,624 | | MicrosoftPreview | 4.0% | 0.0% | 3.9% | 5,938 | 2.9% | 8,624 | | Amazonbot | 15.6% | 12.0% | 3.6% | 5,938 | 11.7% | 8,624 | | Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 5,938 | 12.0% | 8,624 | | Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 5,938 | 6.0% | 8,624 | | FacebookBot | 10.2% | 6.4% | 3.7% | 5,938 | 7.3% | 8,624 | | Bytespider | 19.2% | 15.4% | 3.8% | 5,938 | 15.0% | 8,624 | | CCBot | 19.4% | 15.7% | 3.7% | 5,938 | 14.9% | 8,624 | | DuckAssistBot | 8.1% | 4.3% | 3.8% | 5,938 | 5.8% | 8,624 | | MistralAI-User | 7.7% | 3.8% | 3.9% | 5,938 | 5.5% | 8,624 | | cohere-ai | 15.5% | 11.6% | 3.8% | 5,938 | 11.7% | 8,624 | | YouBot | 10.3% | 6.4% | 3.8% | 5,938 | 7.3% | 8,624 | | Diffbot | 15.7% | 11.9% | 3.8% | 5,938 | 11.8% | 8,624 | | Timpibot | 9.8% | 5.8% | 4.0% | 5,938 | 7.0% | 8,624 | | omgili | 15.7% | 11.8% | 3.9% | 5,938 | 11.8% | 8,624 | | ImagesiftBot | 9.5% | 5.5% | 3.9% | 5,938 | 6.9% | 8,624 | | PetalBot | 14.1% | 10.3% | 3.8% | 5,938 | 10.9% | 8,624 | | ora-scanner | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 | | agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 | Two rows are controls, not AI crawlers. **agent-readiness-crawl** is this crawl's own token. **ora-scanner** is a name the crawl measured that is not Ora's user agent: Ora's scanner calls itself OraBot ([profile](https://agentreadycrawlers.com/crawlers/orabot/)), which the crawl did not measure. The two rows carry the same figures, which show the share of sites whose \* group refuses everything. ## The 401 and 403 share The crawl fetched 10,565 home pages with its labelled user agent. 12.9 percent answered 401 or 403; among the top 10,000 the share was 17.2 percent (1,300 of 7,545). Pages were fetched only where robots.txt allowed the crawl, so 259 domains that disallow it are not in this share. Ora's lab found (2026-10-19) that returning 403 to agent user agents left both a plain-fetch and a JavaScript-running client at 0 of 5 retrieved answers, in a test of 19 configurations on one docs site (190 probes). That is a vendor finding, and a crawl cannot say which of these 403s also block agents. Check it on your own site: Ora's Scan includes [a bot detection check and a robots.txt AI crawler check](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=block-rates); the CLI is `npx ax audit yourdomain.com`. [The edition page](https://agentreadycrawlers.com/block-rates/2026-11-04/) breaks the 401 and 403 share down by category and by bot-management header. ## Added on 2026-11-05 - [**Training and search policy in robots.txt**](https://agentreadycrawlers.com/block-rates/training-and-search/) How many top sites refuse training crawlers but keep AI search open, and how many refuse both. - [**401 and 403 by edge and bot-product header**](https://agentreadycrawlers.com/block-rates/edge-and-bot-headers/) The share of home pages that answered 401 or 403, split by the CDN and bot-product headers they carried. ## Editions and downloads - 2026-11-04[Edition 2026-11-04](https://agentreadycrawlers.com/block-rates/2026-11-04/): 13,071 crawl rows, read 2026-11-04; first edition. - [Block rates by token (CSV)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.csv) - [Block rates (JSON)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.json) - [Home page 401 and 403 share (CSV)](https://agentreadycrawlers.com/data/home-status-2026-11-04.csv) - [Bot-management header prevalence (CSV)](https://agentreadycrawlers.com/data/bot-signals-2026-11-04.csv) **Cite this.** Agent Ready Crawlers, "Block Rate Report", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/. More on [the data page](https://agentreadycrawlers.com/data/). ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=report) read 2026-11-04 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=report) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Block Rate Report, edition 2026-11-04 | Agent Ready Crawlers" description: "Block Rate Report edition 2026-11-04: three charts and tables for token, category and the 401 and 403 share, from a crawl of 13,071 domains." canonical: https://agentreadycrawlers.com/block-rates/2026-11-04 site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Block rates" --- # Block Rate Report, edition 2026-11-04 | Agent Ready Crawlers > Block Rate Report edition 2026-11-04: three charts and tables for token, category and the 401 and 403 share, from a crawl of 13,071 domains. Source: https://agentreadycrawlers.com/block-rates/2026-11-04 Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Block Rate Report: edition 2026-11-04 In short Edition 4 November 2026 of the Block Rate Report: 13,071 domains, 8,624 parsed robots.txt files. Three charts: refusal by token, GPTBot refusal by category (News & Intelligence 30.3 percent, CRM 0.0 percent), and the 401 and 403 share of home pages. 13,071 crawl rows, read 2026-11-04 10,886 domains reachable 8,624 robots.txt files parsed (5,938 in the top 10,000) 10,565 home pages fetched 2,185 domains did not answer, 767 had no robots.txt (404) and 1,495 answered with something that is not a plain-text robots.txt (an error, a block page or HTML). None of those is in a denominator below. Universe: Tranco list 746WX (top 10,000) plus the Ora public leaderboard and the vendors named on this site. User agent `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`. ## Chart 1: by token Site names the tokenRefused only by a \* group Top-10,000 sites with a parsed robots.txt, 5,938 of them. \*The crawl's own token, a control: the wildcard baseline. | Token | Refused, top 10,000 | Named | Via * | Sites | Refused, all crawled | Sites | | --- | --- | --- | --- | --- | --- | --- | | GPTBot | 18.5% | 15.2% | 3.3% | 5,938 | 13.9% | 8,624 | | OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5,938 | 5.4% | 8,624 | | ChatGPT-User | 9.7% | 6.3% | 3.3% | 5,938 | 6.9% | 8,624 | | ClaudeBot | 17.6% | 14.0% | 3.5% | 5,938 | 13.1% | 8,624 | | Claude-User | 8.1% | 4.5% | 3.6% | 5,938 | 5.7% | 8,624 | | Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5,938 | 5.5% | 8,624 | | anthropic-ai | 15.5% | 11.7% | 3.8% | 5,938 | 11.6% | 8,624 | | PerplexityBot | 11.2% | 7.9% | 3.4% | 5,938 | 7.9% | 8,624 | | Perplexity-User | 8.3% | 4.6% | 3.7% | 5,938 | 5.9% | 8,624 | | Google-Extended | 16.0% | 12.6% | 3.4% | 5,938 | 12.1% | 8,624 | | Googlebot | 2.8% | 0.5% | 2.3% | 5,938 | 2.1% | 8,624 | | Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 5,938 | 8.3% | 8,624 | | Applebot | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 | | Applebot-Extended | 14.8% | 11.6% | 3.1% | 5,938 | 11.1% | 8,624 | | Bingbot | 3.1% | 0.4% | 2.7% | 5,938 | 2.2% | 8,624 | | MicrosoftPreview | 4.0% | 0.0% | 3.9% | 5,938 | 2.9% | 8,624 | | Amazonbot | 15.6% | 12.0% | 3.6% | 5,938 | 11.7% | 8,624 | | Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 5,938 | 12.0% | 8,624 | | Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 5,938 | 6.0% | 8,624 | | FacebookBot | 10.2% | 6.4% | 3.7% | 5,938 | 7.3% | 8,624 | | Bytespider | 19.2% | 15.4% | 3.8% | 5,938 | 15.0% | 8,624 | | CCBot | 19.4% | 15.7% | 3.7% | 5,938 | 14.9% | 8,624 | | DuckAssistBot | 8.1% | 4.3% | 3.8% | 5,938 | 5.8% | 8,624 | | MistralAI-User | 7.7% | 3.8% | 3.9% | 5,938 | 5.5% | 8,624 | | cohere-ai | 15.5% | 11.6% | 3.8% | 5,938 | 11.7% | 8,624 | | YouBot | 10.3% | 6.4% | 3.8% | 5,938 | 7.3% | 8,624 | | Diffbot | 15.7% | 11.9% | 3.8% | 5,938 | 11.8% | 8,624 | | Timpibot | 9.8% | 5.8% | 4.0% | 5,938 | 7.0% | 8,624 | | omgili | 15.7% | 11.8% | 3.9% | 5,938 | 11.8% | 8,624 | | ImagesiftBot | 9.5% | 5.5% | 3.9% | 5,938 | 6.9% | 8,624 | | PetalBot | 14.1% | 10.3% | 3.8% | 5,938 | 10.9% | 8,624 | | ora-scanner | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 | | agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 5,938 | 3.0% | 8,624 | Two rows are controls, not AI crawlers. **agent-readiness-crawl** is this crawl's own token. **ora-scanner** is a name the crawl measured that is not Ora's user agent: Ora's scanner calls itself OraBot ([profile](https://agentreadycrawlers.com/crawlers/orabot/)), which the crawl did not measure. The two rows carry the same figures, which show the share of sites whose \* group refuses everything. ### Top 1,000 against the rest of the top 10,000 | Token | Rank 1 to 1,000 | Sites | Rank 1,001 to 10,000 | Sites | | --- | --- | --- | --- | --- | | GPTBot | 24.2% | 521 | 17.9% | 5,417 | | ClaudeBot | 24.0% | 521 | 17.0% | 5,417 | | PerplexityBot | 19.4% | 521 | 10.4% | 5,417 | | Google-Extended | 20.7% | 521 | 15.5% | 5,417 | | Applebot-Extended | 18.8% | 521 | 14.4% | 5,417 | | CCBot | 26.5% | 521 | 18.7% | 5,417 | | Bytespider | 26.7% | 521 | 18.5% | 5,417 | | Googlebot | 4.6% | 521 | 2.6% | 5,417 | ## Chart 2: by category Categories are the Ora public leaderboard's own. A category appears when at least 30 of its sites returned a parsed robots.txt. The uncategorised group is the top-10,000 list and the vendors this site names. Site names GPTBotRefused only by a \* group Share of sites per category that disallow GPTBot at the root. Highest News & Intelligence, 30.3 percent; lowest CRM, 0.0 percent. | Category | Sites | GPTBot | ClaudeBot | PerplexityBot | Google-Extended | Googlebot | | --- | --- | --- | --- | --- | --- | --- | | Agent Tools | 120 | 5.0% | 4.2% | 0.0% | 5.0% | 0.0% | | AI & ML | 177 | 2.3% | 2.3% | 0.6% | 3.4% | 1.1% | | Climate & Energy | 39 | 2.6% | 2.6% | 0.0% | 2.6% | 0.0% | | Communication & Support | 130 | 3.1% | 2.3% | 0.8% | 2.3% | 0.0% | | Consumer | 57 | 3.5% | 3.5% | 0.0% | 3.5% | 0.0% | | CRM | 40 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | | Data & Analytics | 137 | 2.9% | 2.9% | 0.0% | 2.2% | 0.0% | | Developer Tools | 333 | 5.4% | 4.5% | 1.8% | 4.8% | 0.9% | | E-commerce | 133 | 4.5% | 4.5% | 2.3% | 4.5% | 1.5% | | Education | 80 | 5.0% | 6.3% | 0.0% | 5.0% | 0.0% | | Gaming & Media | 61 | 16.4% | 16.4% | 3.3% | 11.5% | 1.6% | | Hardware & Robotics | 80 | 7.5% | 6.3% | 0.0% | 7.5% | 0.0% | | Healthcare | 215 | 3.3% | 3.3% | 1.4% | 3.3% | 0.0% | | HR & Recruiting | 52 | 1.9% | 1.9% | 1.9% | 1.9% | 1.9% | | Identity & Auth | 68 | 7.4% | 5.9% | 4.4% | 7.4% | 1.5% | | Infrastructure | 204 | 2.0% | 2.0% | 0.0% | 2.0% | 0.0% | | Legal & Compliance | 85 | 5.9% | 3.5% | 1.2% | 4.7% | 1.2% | | Logistics & Supply Chain | 72 | 5.6% | 4.2% | 1.4% | 4.2% | 1.4% | | Marketing | 204 | 2.5% | 2.0% | 0.0% | 1.5% | 0.0% | | News & Intelligence | 33 | 30.3% | 42.4% | 33.3% | 42.4% | 0.0% | | Payments & Fintech | 364 | 3.8% | 3.6% | 0.0% | 2.7% | 0.0% | | Productivity | 133 | 6.0% | 3.8% | 0.8% | 3.8% | 0.0% | | Real Estate & Construction | 80 | 3.8% | 3.8% | 2.5% | 5.0% | 1.3% | | Security | 86 | 2.3% | 3.5% | 0.0% | 2.3% | 1.2% | ## Chart 3: the 401 and 403 share Share of fetched home pages that answered 401 or 403 to the labelled crawl user agent, `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`. Pages were fetched only where robots.txt allowed the crawl; 259 domains disallowed it and are not counted. 401 and 403 share of home pages by group. All crawled: 10,565 pages. Top 10,000: 7,545. | Group | Home pages | 401 | 403 | 401 or 403 | | --- | --- | --- | --- | --- | | All crawled | 10,565 | 15 | 1,344 | 12.9% | | Top 10,000 | 7,545 | 14 | 1,286 | 17.2% | | Rank 1 to 1,000 | 663 | 1 | 85 | 13.0% | | Rank 1,001 to 10,000 | 6,882 | 13 | 1,201 | 17.6% | | Agent Tools | 131 | 0 | 1 | 0.8% | | AI & ML | 205 | 0 | 6 | 2.9% | | Climate & Energy | 42 | 0 | 1 | 2.4% | | Communication & Support | 149 | 0 | 7 | 4.7% | | Consumer | 75 | 0 | 3 | 4.0% | | CRM | 42 | 0 | 3 | 7.1% | | Data & Analytics | 148 | 0 | 3 | 2.0% | | Developer Tools | 376 | 0 | 3 | 0.8% | | E-commerce | 149 | 0 | 5 | 3.4% | | Education | 92 | 0 | 4 | 4.3% | | Gaming & Media | 70 | 1 | 8 | 12.9% | | Hardware & Robotics | 104 | 0 | 2 | 1.9% | | Healthcare | 246 | 0 | 4 | 1.6% | | HR & Recruiting | 57 | 0 | 3 | 5.3% | | Identity & Auth | 70 | 0 | 0 | 0.0% | | Infrastructure | 232 | 1 | 3 | 1.7% | | Legal & Compliance | 91 | 0 | 2 | 2.2% | | Logistics & Supply Chain | 85 | 0 | 1 | 1.2% | | Marketing | 222 | 0 | 9 | 4.1% | | News & Intelligence | 35 | 0 | 9 | 25.7% | | Payments & Fintech | 399 | 0 | 8 | 2.0% | | Productivity | 148 | 0 | 4 | 2.7% | | Real Estate & Construction | 85 | 0 | 3 | 3.5% | | Security | 94 | 0 | 2 | 2.1% | Ora's lab found (2026-10-19) that a 403 to agent user agents left both a plain-fetch and a JavaScript client at 0 of 5 retrieved answers; that is a vendor finding from one docs site. Check it on your own domain with [Ora's bot detection check](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=block-rates-edition) or `npx ax audit yourdomain.com`. ### Bot-management headers on home pages The crawl records headers that reveal an edge or bot product. Most only show that a CDN or host is present. Home pages behind Cloudflare headers answered 401 or 403 22.3 percent of the time, against 7.0 percent for pages without them. A header is not proof that bot rules are switched on. | Header signal | What it tells us | Home pages | Share | 401 or 403 with it | Without it | | --- | --- | --- | --- | --- | --- | | Cloudflare (cf-ray header) | A CDN or host header only; it does not say whether bot rules are on | 4,062 | 38.4% | 22.3% | 7.0% | | AWS (CloudFront or WAF header) | A CDN or host header only; it does not say whether bot rules are on | 1,202 | 11.4% | 4.1% | 14.0% | | Vercel (x-vercel-id header) | A CDN or host header only; it does not say whether bot rules are on | 1,064 | 10.1% | 0.1% | 14.3% | | Cloudflare challenge (cf-mitigated header) | The response says the request was challenged or mitigated | 644 | 6.1% | 99.8% | 7.2% | | Fastly (server or x-served-by header) | A CDN or host header only; it does not say whether bot rules are on | 513 | 4.9% | 6.0% | 13.2% | | Akamai (server header or cookie) | A CDN or host header only; it does not say whether bot rules are on | 496 | 4.7% | 36.9% | 11.7% | | Netlify (x-nf-request-id header) | A CDN or host header only; it does not say whether bot rules are on | 156 | 1.5% | 0.0% | 13.1% | | Imperva (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 66 | 0.6% | 10.6% | 12.9% | | DataDome (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 64 | 0.6% | 70.3% | 12.5% | | Vercel mitigation (x-vercel-mitigated header) | The response says the request was challenged or mitigated | 32 | 0.3% | 3.1% | 12.9% | | HUMAN (cookie) | A bot product header or cookie is present, which is not proof of enforcement | 6 | 0.1% | 50.0% | 12.8% | | Kasada (header) | A bot product header or cookie is present, which is not proof of enforcement | 3 | 0.0% | 0.0% | 12.9% | ## Cite and download - [Block rates by token (CSV)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.csv) - [Block rates (JSON)](https://agentreadycrawlers.com/data/block-rates-2026-11-04.json) - [Home page 401 and 403 share (CSV)](https://agentreadycrawlers.com/data/home-status-2026-11-04.csv) - [Bot-management header prevalence (CSV)](https://agentreadycrawlers.com/data/bot-signals-2026-11-04.csv) **Cite this.** Agent Ready Crawlers, "Block Rate Report, edition 2026-11-04", https://agentreadycrawlers.com/block-rates/2026-11-04/. Crawl row count 13,071, read 2026-11-04. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=edition) read 2026-11-04 - [Tranco list 746WX, top 10,000](https://tranco-list.eu/list/746WX/10000) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "401 and 403 by edge and bot-product header | Agent Ready Crawlers" description: "The 401 and 403 share of home pages split by the CDN, host and bot-product headers they carried, from a crawl of 13,071 domains. Crawl of 2026-11-04." canonical: https://agentreadycrawlers.com/block-rates/edge-and-bot-headers site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Block rates" --- # 401 and 403 by edge and bot-product header | Agent Ready Crawlers > The 401 and 403 share of home pages split by the CDN, host and bot-product headers they carried, from a crawl of 13,071 domains. Crawl of 2026-11-04. Source: https://agentreadycrawlers.com/block-rates/edge-and-bot-headers Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # 401 and 403 by edge and bot-product header In short 10,565 home pages were fetched; 12.9 percent answered 401 or 403. Pages with a Cloudflare challenge header answered 401 or 403 99.8 percent of the time, pages with a DataDome header 70.3 percent and pages with a Vercel header 0.1 percent. The challenge header sits on the refusal itself. The crawl records response headers on every home page it fetched. Some name a CDN or host, a few name a bot product, and one says the request was challenged. Setting each beside the status the page answered shows where the refusals sit, with a caution: a header says what the response carried, and Cloudflare's challenge header is set on the refusal itself. ## The share that refused, by header Home pages that answered the crawl user agent with 401 or 403, among those carrying each header; headers seen on at least 60 pages. Crawl of 2026-11-04. | Header | What it tells us | Pages | Share of pages | 401 or 403, with it | Without it | | --- | --- | --- | --- | --- | --- | | Cloudflare (cf-ray header) | A CDN or host header only; it does not say whether bot rules are on | 4,062 | 38.4% | 22.3% | 7.0% | | AWS (CloudFront or WAF header) | A CDN or host header only; it does not say whether bot rules are on | 1,202 | 11.4% | 4.1% | 14.0% | | Vercel (x-vercel-id header) | A CDN or host header only; it does not say whether bot rules are on | 1,064 | 10.1% | 0.1% | 14.3% | | Cloudflare challenge (cf-mitigated header) | The response says the request was challenged or mitigated | 644 | 6.1% | 99.8% | 7.2% | | Fastly (server or x-served-by header) | A CDN or host header only; it does not say whether bot rules are on | 513 | 4.9% | 6.0% | 13.2% | | Akamai (server header or cookie) | A CDN or host header only; it does not say whether bot rules are on | 496 | 4.7% | 36.9% | 11.7% | | Netlify (x-nf-request-id header) | A CDN or host header only; it does not say whether bot rules are on | 156 | 1.5% | 0.0% | 13.1% | | Imperva (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 66 | 0.6% | 10.6% | 12.9% | | DataDome (header or cookie) | A bot product header or cookie is present, which is not proof of enforcement | 64 | 0.6% | 70.3% | 12.5% | | Vercel mitigation (x-vercel-mitigated header) | The response says the request was challenged or mitigated | 32 | 0.3% | 3.1% | 12.9% | | HUMAN (cookie) | A bot product header or cookie is present, which is not proof of enforcement | 6 | 0.1% | 50.0% | 12.8% | | Kasada (header) | A bot product header or cookie is present, which is not proof of enforcement | 3 | 0.0% | 0.0% | 12.9% | ## How to read it Challenge header 643 of the 644 pages with Cloudflare's challenge header answered 401 or 403. The header is set on the challenge itself, so this is the crawl being challenged and not a rate across a population. Edge or host header 4,062 pages, 38.4 percent, carried a Cloudflare header, and 22.3 percent of them refused against 7.0 percent of the rest. The header shows the site is behind Cloudflare, not that bot rules are on; the challenge pages above are part of the difference. Bot-product header 64 pages carried a DataDome header or cookie, and 70.3 percent refused. A bot-product header is not proof of enforcement, and the share describes this crawl's requests only. Hosts that refused almost nothing 1,064 pages carried a Vercel header and 1 refused; the Netlify header was on 156 pages and none refused. The crawl does not show why. ## What the crawl cannot say A status is what one labelled user agent got on one day from one network, and a header is a presence test. Neither says which agents a site admits. To find out what your own site answers to ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, run [the test in the guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/), and read [what a 403 does to an agent](https://agentreadycrawlers.com/guides/403-to-ai-agents/) first. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Block Rate Report, edition 2026-11-04 (this site)](https://agentreadycrawlers.com/block-rates/2026-11-04/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Training and search policy in robots.txt | Agent Ready Crawlers" description: "How many top sites refuse training crawlers but keep AI search open, how many refuse both, and how many refuse neither. Crawl of 2026-11-04." canonical: https://agentreadycrawlers.com/block-rates/training-and-search site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Block rates" --- # Training and search policy in robots.txt | Agent Ready Crawlers > How many top sites refuse training crawlers but keep AI search open, how many refuse both, and how many refuse neither. Crawl of 2026-11-04. Source: https://agentreadycrawlers.com/block-rates/training-and-search Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Training and search policy in robots.txt In short 11.1 percent of 5,938 top-10,000 sites refuse a training crawler and no AI search crawler, 11.4 percent refuse both, and 77.3 percent refuse neither. Only 12 sites refuse search alone. The groups are six training tokens and three AI search tokens taken from the Crawler Index. Crawl of 13,071 domains, 2026-11-04. Operators now let a site refuse training and still be found by AI search. The Block Rate Report counts refusals one token at a time; this page reads each site as a whole and asks which combination it chose. Training means GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. AI search means OAI-SearchBot, Claude-SearchBot and PerplexityBot. A site counts as refusing a token when RFC 9309 matching disallows the root for it, whether it named the token or a \* group refuses everything. ## The four combinations Refuses neitherRefuses training onlyRefuses training and searchRefuses search only Share of sites with a parsed robots.txt, crawl of 2026-11-04. Segments narrower than a few percent carry no label; the table has every number. | Combination | Top 10,000 | Share | All crawled | Share | | --- | --- | --- | --- | --- | | Refuses neither | 4,590 | 77.3% | 7,128 | 82.7% | | Refuses training only | 660 | 11.1% | 792 | 9.2% | | Refuses training and search | 676 | 11.4% | 692 | 8.0% | | Refuses search only | 12 | 0.2% | 12 | 0.1% | [Download the table as CSV](https://agentreadycrawlers.com/data/policy-mix-2026-11-04.csv). ## Refusing every training token Top-10,000 sites by the number of the six training tokens refused. The two ends hold most sites: 77.5 percent refuse none and 11.0 percent refuse all six. 297 top-10,000 sites, 5.0 percent, refuse all six training tokens and none of the three search tokens: the recipe in [how to allow search and refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/). 292 sites, 4.9 percent, refuse all nine. Of those, 168 name none of the nine, so their refusal is a blanket group and not a decision about AI. ## What refusing search costs 688 sites in the top 10,000 refuse at least one of the three search tokens. The operators say what that does. OpenAI: Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. ## Limits The counts read robots.txt and nothing else. A site that refuses a token here may still admit the agent at the WAF, and one that allows it may refuse it there. The six training tokens and the three search tokens are our grouping from the Index purposes; Amazonbot, Bytespider and other tokens that the Index files under training or Other are measured in the [Block Rate Report](https://agentreadycrawlers.com/block-rates/) and left out of this mix. Only the top-10,000 population has an upper rank bound, and the all-crawled population includes the Ora leaderboard's sites. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Allow search, refuse training (this site)](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Akamai vs Cloudflare | Agent Ready Crawlers" description: "Akamai vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/akamai-vs-cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Akamai vs Cloudflare | Agent Ready Crawlers > Akamai vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/akamai-vs-cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Akamai vs Cloudflare In short Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). On per-purpose policy the rubric grades Akamai partial and Cloudflare yes, and on commerce agents the rubric grades Akamai yes and Cloudflare partial. For a security engineer: Akamai announced Visa TAP integration. ## The sourced difference Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). [Source](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue), read 2026-11-04. ## Side by side | Question | Akamai | Cloudflare | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | | Per-purpose policy (train, search, answer, act) | Partial; allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | | Charging for access | Partial; through TollBit and Skyfire partners, no native billing | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | | Lets signed commerce agents through | Yes; Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | | Cost | no public price; sales-led | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | Reporting | Yes; TrafficPeak and web security analytics | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | | Where it runs | edge: verification and enforcement on Akamai's edge network | edge: enforced on Cloudflare's reverse proxy, no app code needed | | Points, rank | 8 of 10, rank 1 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04) | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | ## Verdicts by reader Platform engineer Both verify Web Bot Auth at the edge. Akamai prices are sales-led; Cloudflare has a free tier. Security engineer Akamai announced Visa TAP integration (2025-12-17); Cloudflare announced managed rulesets for TAP and Agent Pay in 2025-10 but the current rule name is not in its docs (checked 2026-11-04). Publisher operations Cloudflare lets you charge without a third party contract; Akamai needs a TollBit or Skyfire relationship. ## What each does not do ### Akamai - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. More: [alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/), [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/). ## Sources - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Akamai vs HUMAN Security | Agent Ready Crawlers" description: "Akamai vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/akamai-vs-human-security site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Akamai vs HUMAN Security | Agent Ready Crawlers > Akamai vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/akamai-vs-human-security Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Akamai vs HUMAN Security In short Akamai is listed by OpenAI as recognising ChatGPT's Cloud browser under the bot name "ChatGPT Agent"; HUMAN lists the same agent in Known Bots & Crawlers and AgenticTrust permissions. On per-purpose policy the rubric grades Akamai partial and HUMAN Security yes, and on commerce agents the rubric grades Akamai yes and HUMAN Security not published. ## The sourced difference Akamai is listed by OpenAI as recognising ChatGPT's Cloud browser under the bot name "ChatGPT Agent"; HUMAN lists the same agent in Known Bots & Crawlers and AgenticTrust permissions. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ## Side by side | Question | Akamai | HUMAN Security | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | Yes; per-request cryptographic keys (e.g. ChatGPT's published key), plus behavior and private signals; Web Bot Auth not named on pages read | | Per-purpose policy (train, search, answer, act) | Partial; allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | Yes; per-action categories (content, engage, account creation, checkout, rate limit) by trust level | | Charging for access | Partial; through TollBit and Skyfire partners, no native billing | Partial; through scraping monetization partners (TollBit logo on product page); no native charging found | | Lets signed commerce agents through | Yes; Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | no public price; sales-led | no public price; sales-led | | Reporting | Yes; TrafficPeak and web security analytics | Yes; user and session insights on agent actions, monthly State of Agentic Traffic benchmark | | Where it runs | edge: verification and enforcement on Akamai's edge network | platform: Sightline integrates with the customer's stack; exact deployment points not confirmed on pages read | | Points, rank | 8 of 10, rank 1 (tied) | 7 of 10, rank 5 | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04) | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer Both verify the signature automatically, per OpenAI. Security engineer HUMAN adds per-action permissions; Akamai adds allow, challenge, delay or block actions. Publisher operations Akamai can route bots to TollBit and Skyfire; HUMAN shows TollBit as a partner. ## What each does not do ### Akamai - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). ### HUMAN Security - Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02). - No native pay-per-crawl; monetization is through partner tools (AgenticTrust page). - No published price list (package plans page). More: [alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/), [alternatives to HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/alternatives/). ## Sources - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [humansecurity.com: agentic ai](https://www.humansecurity.com/applications/agentic-ai/) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Arcjet vs Cloudflare | Agent Ready Crawlers" description: "Arcjet vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Arcjet vs Cloudflare | Agent Ready Crawlers > Arcjet vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Arcjet vs Cloudflare In short Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. On verifies agents the rubric grades Arcjet partial and Cloudflare yes, and on per-purpose policy the rubric grades Arcjet partial and Cloudflare yes. For publisher operations: Cloudflare can write your robots.txt preferences; Arcjet cannot. ## The sourced difference Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. [Source](https://arcjet.com/pricing), read 2026-11-04. ## Side by side | Question | Arcjet | Cloudflare | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; IP data and reverse DNS for known bots; Web Bot Auth not mentioned in docs (checked 2026-11-04) | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | | Per-purpose policy (train, search, answer, act) | Partial; allow or deny by bot name or category (e.g. CATEGORY:AI) per route | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | | Charging for access | Not published; support (checked 2026-11-04) | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | | Cost | published: from USD 25/month plus USD 5 per 1M web requests | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | Reporting | Partial; request decision logs with 1 to 24 hours retention on self-serve plans | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | | Where it runs | app: SDK calls in application code | edge: enforced on Cloudflare's reverse proxy, no app code needed | | Points, rank | 3 of 10, rank 11 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | Individual USD 25/month, Startup USD 299/month, Enterprise custom; usage USD 5 per 1M web requests and USD 50 per 1M agent requests; free plan 10,000 requests/month (hard stop) after a 15-day trial. | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | ## Verdicts by reader Platform engineer Arcjet suits apps not behind a CDN; Cloudflare needs DNS on Cloudflare. Security engineer Cloudflare verifies Web Bot Auth; Arcjet uses IP and reverse DNS. Publisher operations Cloudflare can write your robots.txt preferences; Arcjet cannot. ## What each does not do ### Arcjet - No Web Bot Auth or cryptographic agent verification in bot protection docs (checked 2026-11-04). - Does not run at the CDN edge; requests reach the app before Arcjet decides (concepts page). - No AI crawler monetization or pay-per-crawl (docs read 2026-11-04). ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. More: [alternatives to Arcjet](https://agentreadycrawlers.com/vendors/arcjet/alternatives/), [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/). ## Sources - [arcjet.com: pricing](https://arcjet.com/pricing) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [docs.arcjet.com: concepts](https://docs.arcjet.com/bot-protection/concepts) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [docs.arcjet.com: reference](https://docs.arcjet.com/bot-protection/reference) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Arcjet vs Vercel | Agent Ready Crawlers" description: "Arcjet vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Arcjet vs Vercel | Agent Ready Crawlers > Arcjet vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Arcjet vs Vercel In short Arcjet verifies known bots by IP and reverse DNS inside application code through its SDK; Vercel verifies by IP, reverse DNS and Web Bot Auth at its edge and splits AI crawlers from AI assistants in a managed ruleset since 2026-10-29. For a security engineer: Vercel accepts Web Bot Auth signatures. ## The sourced difference Arcjet verifies known bots by IP and reverse DNS inside application code through its SDK; Vercel verifies by IP, reverse DNS and Web Bot Auth at its edge and splits AI crawlers from AI assistants in a managed ruleset since 2026-10-29. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. ## Side by side | Question | Arcjet | Vercel | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; IP data and reverse DNS for known bots; Web Bot Auth not mentioned in docs (checked 2026-11-04) | Yes; IP, reverse DNS and Web Bot Auth; directory at bots.fyi | | Per-purpose policy (train, search, answer, act) | Partial; allow or deny by bot name or category (e.g. CATEGORY:AI) per route | Yes; AI crawlers and AI assistants handled separately since 2026-10-29 | | Charging for access | Not published; support (checked 2026-11-04) | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | published: from USD 25/month plus USD 5 per 1M web requests | published: Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Reporting | Partial; request decision logs with 1 to 24 hours retention on self-serve plans | Yes; Firewall tab BotID filter and Observability Plus metrics | | Where it runs | app: SDK calls in application code | both: managed rulesets at Vercel's edge firewall; BotID verdict read in server code via checkBotId() | | Points, rank | 3 of 10, rank 11 (tied) | 6 of 10, rank 6 (tied) | | Price read 2026-11-04 | Individual USD 25/month, Startup USD 299/month, Enterprise custom; usage USD 5 per 1M web requests and USD 50 per 1M agent requests; free plan 10,000 requests/month (hard stop) after a 15-day trial. | BotID Basic free on all plans; Deep Analysis USD 1 per 1,000 checkBotId() calls on Pro, custom on Enterprise; AI Bots managed ruleset on all plans. | ## Verdicts by reader Platform engineer On Vercel hosting, the managed ruleset is a toggle; Arcjet works on any host but every rule is code. Security engineer Vercel accepts Web Bot Auth signatures; Arcjet docs do not mention Web Bot Auth (checked 2026-11-04). Publisher operations Vercel can block training crawlers while keeping answer fetchers; Arcjet can do it per route with CATEGORY rules. ## What each does not do ### Arcjet - No Web Bot Auth or cryptographic agent verification in bot protection docs (checked 2026-11-04). - Does not run at the CDN edge; requests reach the app before Arcjet decides (concepts page). - No AI crawler monetization or pay-per-crawl (docs read 2026-11-04). ### Vercel - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). More: [alternatives to Arcjet](https://agentreadycrawlers.com/vendors/arcjet/alternatives/), [alternatives to Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/). ## Sources - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [arcjet.com: pricing](https://arcjet.com/pricing) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [docs.arcjet.com: concepts](https://docs.arcjet.com/bot-protection/concepts) read 2026-11-04 - [vercel.com: vercels bot verification now supports web bot auth](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth) read 2026-11-04 - [docs.arcjet.com: reference](https://docs.arcjet.com/bot-protection/reference) read 2026-11-04 - [vercel.com: verified bots](https://vercel.com/docs/botid/verified-bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Auth0 for AI Agents vs Skyfire | Agent Ready Crawlers" description: "Auth0 for AI Agents vs Skyfire: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Auth0 for AI Agents vs Skyfire | Agent Ready Crawlers > Auth0 for AI Agents vs Skyfire: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Auth0 for AI Agents vs Skyfire In short Auth0 issues identities and short-lived tokens to agents a company builds for its own users; Skyfire issues KYA tokens that any third-party site can verify. On verifies agents the rubric grades Auth0 for AI Agents not its job and Skyfire yes, and on per-purpose policy the rubric grades Auth0 for AI Agents partial and Skyfire not its job. ## The sourced difference Auth0 issues identities and short-lived tokens to agents a company builds for its own users; Skyfire issues KYA tokens that any third-party site can verify. [Source](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md), read 2026-11-04. ## Side by side | Question | Auth0 for AI Agents | Skyfire | | --- | --- | --- | | Tells a verified agent from a spoof | Not its job; issues identities and scoped tokens to the customer's own agents; does not verify inbound third-party bots | Yes; KYA JWTs that sites verify via Skyfire's JWKS; not a bot manager itself | | Per-purpose policy (train, search, answer, act) | Partial; fine-grained permissions and authorization for RAG scoped to each agent task | Not its job; sellers decide access from token claims such as agent and principal identity | | Charging for access | Not published; support (checked 2026-11-04) | Yes; pay and kya-pay tokens charged by sellers, settled between Skyfire wallets | | Lets signed commerce agents through | Partial; product page mentions agentic protocols for commerce and procurement; no named TAP or Agent Pay support found | Yes; KYAPay carries identity plus payment intent; card payment flow documented | | Cost | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | not published (checked 2026-11-04) | | Reporting | Partial; audit trail for agent actions claimed | Not published; on pages read (checked 2026-11-04) | | Where it runs | app: SDKs and tenant APIs inside the agent and its backend | either: sellers verify tokens in their backend, in a Cloudflare Worker, or through partner bot managers (F5, Akamai, Fastly) | | Points, rank | 3 of 10, rank 11 (tied) | 6 of 10, rank 6 (tied) | | Price read 2026-11-04 | AI Agents add-on at 50 percent of the base plan price (USD 18 a month on B2C Essentials with 500 users, USD 75 on B2B Essentials); free plan available, paid tiers by monthly active users (checked 2026-11-05) | no public fee schedule found on homepage or docs index (checked 2026-11-04) | ## Verdicts by reader Platform engineer Auth0 inside your stack; Skyfire across sites. Security engineer Auth0 gates sensitive actions behind user consent; Skyfire binds agent, principal and payment. Publisher operations Only Skyfire moves money. ## What each does not do ### Auth0 for AI Agents - Does not detect, verify or block third-party crawlers or agents hitting a website; it secures agents the customer builds (product page). - Agent Gateway for MCP and Agent as Principal are listed as shipping next, not available (product page read 2026-11-04). ### Skyfire - Does not detect or block unsigned bots; it only issues and settles tokens that sites or bot managers must check (docs). - KYA tokens alone cannot be charged; only pay and kya-pay tokens move money (introduction docs). - No published fee schedule (homepage and docs read 2026-11-04). More: [alternatives to Auth0 for AI Agents](https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives/), [alternatives to Skyfire](https://agentreadycrawlers.com/vendors/skyfire/alternatives/). ## Sources - [docs.skyfire.xyz: introduction to skyfire.md](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md) read 2026-11-04 - [auth0.com: pricing](https://auth0.com/pricing) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [auth0.com: ai](https://auth0.com/ai) read 2026-11-04 - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [auth0.com: overview](https://auth0.com/ai/docs/intro/overview) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [kyapay.org: kyapay.org](https://kyapay.org/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "AWS WAF Bot Control vs Akamai | Agent Ready Crawlers" description: "AWS WAF Bot Control vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # AWS WAF Bot Control vs Akamai | Agent Ready Crawlers > AWS WAF Bot Control vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # AWS WAF Bot Control vs Akamai In short AWS WAF publishes Bot Control prices (USD 10 a month per web ACL, USD 1 per million common requests after 10 million) and charges bots natively over x402; Akamai is sales-led and charges through partners. For a security engineer: Both verify Web Bot Auth; Akamai integrates Visa TAP. For a platform engineer: AWS works on CloudFront and regional resources. ## The sourced difference AWS WAF publishes Bot Control prices (USD 10 a month per web ACL, USD 1 per million common requests after 10 million) and charges bots natively over x402; Akamai is sales-led and charges through partners. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. ## Side by side | Question | AWS WAF Bot Control | Akamai | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth (rule set Version_4.0+), verified self-identifying bots | Yes; Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | | Per-purpose policy (train, search, answer, act) | Yes; rules on bot category and name labels; prices can differ by bot identity and verification status | Partial; allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | | Charging for access | Yes; HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins | Partial; through TollBit and Skyfire partners, no native billing | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Yes; Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | | Cost | published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | no public price; sales-led | | Reporting | Yes; Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs | Yes; TrafficPeak and web security analytics | | Where it runs | edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway | edge: verification and enforcement on Akamai's edge network | | Points, rank | 8 of 10, rank 1 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge. | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer AWS works on CloudFront and regional resources; Akamai at its edge. Security engineer Both verify Web Bot Auth; Akamai integrates Visa TAP. Publisher operations AWS monetization pays in stablecoins today; Stripe payouts were listed as coming soon on 2026-06-15. ## What each does not do ### AWS WAF Bot Control - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). ### Akamai - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). More: [alternatives to AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/), [alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/). ## Sources - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "AWS WAF Bot Control vs DataDome | Agent Ready Crawlers" description: "AWS WAF Bot Control vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # AWS WAF Bot Control vs DataDome | Agent Ready Crawlers > AWS WAF Bot Control vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # AWS WAF Bot Control vs DataDome In short AWS WAF charges AI bots natively over x402 at no extra fee beyond WAF charges; DataDome charges through partners from its USD 8,670 Advanced plan. On per-purpose policy the rubric grades AWS WAF Bot Control yes and DataDome partial, and on commerce agents the rubric grades AWS WAF Bot Control not published and DataDome partial. ## The sourced difference AWS WAF charges AI bots natively over x402 at no extra fee beyond WAF charges; DataDome charges through partners from its USD 8,670 Advanced plan. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. ## Side by side | Question | AWS WAF Bot Control | DataDome | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth (rule set Version_4.0+), verified self-identifying bots | Yes; Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | | Per-purpose policy (train, search, answer, act) | Yes; rules on bot category and name labels; prices can differ by bot identity and verification status | Partial; policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | | Charging for access | Yes; HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins | Yes; AI crawler and agent monetization through partners, from the Advanced plan | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Partial; TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | | Cost | published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | published: from USD 3,830/month | | Reporting | Yes; Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs | Yes; agent traffic by intent, identity, geo, URL and endpoint type | | Where it runs | edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway | both: server-side module on web servers, CDNs or API gateways plus client-side tag | | Points, rank | 8 of 10, rank 1 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge. | Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. | ## Verdicts by reader Platform engineer AWS for CloudFront shops; DataDome for multi-CDN. Security engineer DataDome adds intent detection on authenticated agents; AWS adds targeted ML for bots that do not self-identify. Publisher operations AWS pays out in stablecoins; check that suits your finance team. ## What each does not do ### AWS WAF Bot Control - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). ### DataDome - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). More: [alternatives to AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/), [alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/). ## Sources - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare vs AWS WAF Bot Control | Agent Ready Crawlers" description: "Cloudflare vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts by reader type. Read 2026-11-04." canonical: https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Cloudflare vs AWS WAF Bot Control | Agent Ready Crawlers > Cloudflare vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts by reader type. Read 2026-11-04. Source: https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Cloudflare vs AWS WAF Bot Control In short AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. For a security engineer: Both verify Web Bot Auth (AWS needs rule set Version\_4.0 or later selected). ## The sourced difference AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. ## Side by side | Question | Cloudflare | AWS WAF Bot Control | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Yes; Web Bot Auth (rule set Version_4.0+), verified self-identifying bots | | Per-purpose policy (train, search, answer, act) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Yes; rules on bot category and name labels; prices can differ by bot identity and verification status | | Charging for access | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Yes; HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins | | Lets signed commerce agents through | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Reporting | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Yes; Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs | | Where it runs | edge: enforced on Cloudflare's reverse proxy, no app code needed | edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway | | Points, rank | 8 of 10, rank 1 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge. | ## Verdicts by reader Platform engineer On AWS, Bot Control is USD 10 a month per web ACL plus per-request fees; Cloudflare AI Crawl Control is on the free plan. Security engineer Both verify Web Bot Auth (AWS needs rule set Version\_4.0 or later selected). Publisher operations AWS charging is generally available with Test mode; Cloudflare pay per crawl is in private beta. ## What each does not do ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. ### AWS WAF Bot Control - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). More: [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/), [alternatives to AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/). ## Sources - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare vs DataDome | Agent Ready Crawlers" description: "Cloudflare vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/cloudflare-vs-datadome site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Cloudflare vs DataDome | Agent Ready Crawlers > Cloudflare vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/cloudflare-vs-datadome Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Cloudflare vs DataDome In short Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. On per-purpose policy the rubric grades Cloudflare yes and DataDome partial, and on charging the rubric grades Cloudflare partial and DataDome yes. ## The sourced difference Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. [Source](https://datadome.co/pricing/), read 2026-11-04. ## Side by side | Question | Cloudflare | DataDome | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Yes; Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | | Per-purpose policy (train, search, answer, act) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Partial; policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | | Charging for access | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Yes; AI crawler and agent monetization through partners, from the Advanced plan | | Lets signed commerce agents through | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Partial; TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | | Cost | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | published: from USD 3,830/month | | Reporting | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Yes; agent traffic by intent, identity, geo, URL and endpoint type | | Where it runs | edge: enforced on Cloudflare's reverse proxy, no app code needed | both: server-side module on web servers, CDNs or API gateways plus client-side tag | | Points, rank | 8 of 10, rank 1 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. | ## Verdicts by reader Platform engineer If you already proxy through Cloudflare, its zone settings cover AI crawlers with no code. DataDome needs a server module plus a client tag. Security engineer DataDome scores each agent on identity strength and fraud history and blocks bad intent even when the agent is authenticated; Cloudflare focuses on identity and category. Publisher operations Cloudflare is the cheaper start for training opt-outs; DataDome is a paid bot and fraud stack with monetization partners. ## What each does not do ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. ### DataDome - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). More: [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/), [alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/). ## Sources - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare vs HUMAN Security | Agent Ready Crawlers" description: "Cloudflare vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/cloudflare-vs-human-security site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Cloudflare vs HUMAN Security | Agent Ready Crawlers > Cloudflare vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/cloudflare-vs-human-security Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Cloudflare vs HUMAN Security In short Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. On commerce agents the rubric grades Cloudflare partial and HUMAN Security not published. For a security engineer: HUMAN grades trust per agent; Cloudflare separates verified bots and signed agents. ## The sourced difference Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ## Side by side | Question | Cloudflare | HUMAN Security | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Yes; per-request cryptographic keys (e.g. ChatGPT's published key), plus behavior and private signals; Web Bot Auth not named on pages read | | Per-purpose policy (train, search, answer, act) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Yes; per-action categories (content, engage, account creation, checkout, rate limit) by trust level | | Charging for access | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Partial; through scraping monetization partners (TollBit logo on product page); no native charging found | | Lets signed commerce agents through | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | no public price; sales-led | | Reporting | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Yes; user and session insights on agent actions, monthly State of Agentic Traffic benchmark | | Where it runs | edge: enforced on Cloudflare's reverse proxy, no app code needed | platform: Sightline integrates with the customer's stack; exact deployment points not confirmed on pages read | | Points, rank | 8 of 10, rank 1 (tied) | 7 of 10, rank 5 | | Price read 2026-11-04 | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer Cloudflare is a WAF rule on detection ID; HUMAN is a console policy. Security engineer HUMAN grades trust per agent; Cloudflare separates verified bots and signed agents. Publisher operations Cloudflare offers charging and a free tier; HUMAN does neither publicly. ## What each does not do ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. ### HUMAN Security - Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02). - No native pay-per-crawl; monetization is through partner tools (AgenticTrust page). - No published price list (package plans page). More: [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/), [alternatives to HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/alternatives/). ## Sources - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [humansecurity.com: agentic ai](https://www.humansecurity.com/applications/agentic-ai/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare vs Netlify | Agent Ready Crawlers" description: "Cloudflare vs Netlify: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/cloudflare-vs-netlify site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Cloudflare vs Netlify | Agent Ready Crawlers > Cloudflare vs Netlify: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/cloudflare-vs-netlify Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Cloudflare vs Netlify In short Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. On verifies agents the rubric grades Cloudflare yes and Netlify not published, and on per-purpose policy the rubric grades Cloudflare yes and Netlify partial. ## The sourced difference Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04. ## Side by side | Question | Cloudflare | Netlify | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Not published; support (checked 2026-11-04): blocking matches user-agent strings only | | Per-purpose policy (train, search, answer, act) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Partial; choose AI crawlers versus SEO or search crawlers from a preset list | | Charging for access | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Not published; support (checked 2026-11-04) | | Cost | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | published: platform plans from USD 0; extension price not stated | | Reporting | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Not published; no AI bot reporting found (checked 2026-11-04) | | Where it runs | edge: enforced on Cloudflare's reverse proxy, no app code needed | edge: Netlify Edge Function and edge firewall rules | | Points, rank | 8 of 10, rank 1 (tied) | 1 of 10, rank 15 | | Price read 2026-11-04 | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | Free USD 0 (300 credits), Personal USD 9/month (1,000 credits), Pro USD 20/month (3,000 credits), Enterprise custom; Firewall Traffic Rules and basic rate limiting listed from Free; User Agent Blocker price not stated. | ## Verdicts by reader Platform engineer Netlify users can front sites with Cloudflare for per-purpose control. Security engineer User-agent matching blocks spoofers and real bots alike; verification needs Cloudflare or similar. Publisher operations Netlify gives no AI bot reporting (checked 2026-11-04); Cloudflare does. ## What each does not do ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. ### Netlify - No verified-bot list, reverse DNS or Web Bot Auth checks; user agents can be spoofed (block AI crawlers docs). - Preset list is incomplete by Netlify's own statement (block AI crawlers docs, 2026-09-18). - No AI crawler monetization (docs read 2026-11-04). More: [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/), [alternatives to Netlify](https://agentreadycrawlers.com/vendors/netlify/alternatives/). ## Sources - [docs.netlify.com: block ai crawlers](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [netlify.com: pricing](https://www.netlify.com/pricing/) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare vs Vercel | Agent Ready Crawlers" description: "Cloudflare vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Cloudflare vs Vercel | Agent Ready Crawlers > Cloudflare vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Cloudflare vs Vercel In short Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. On charging the rubric grades Cloudflare partial and Vercel not published, and on commerce agents the rubric grades Cloudflare partial and Vercel not published. ## The sourced difference Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. ## Side by side | Question | Cloudflare | Vercel | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Yes; IP, reverse DNS and Web Bot Auth; directory at bots.fyi | | Per-purpose policy (train, search, answer, act) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Yes; AI crawlers and AI assistants handled separately since 2026-10-29 | | Charging for access | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | published: Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Reporting | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Yes; Firewall tab BotID filter and Observability Plus metrics | | Where it runs | edge: enforced on Cloudflare's reverse proxy, no app code needed | both: managed rulesets at Vercel's edge firewall; BotID verdict read in server code via checkBotId() | | Points, rank | 8 of 10, rank 1 (tied) | 6 of 10, rank 6 (tied) | | Price read 2026-11-04 | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | BotID Basic free on all plans; Deep Analysis USD 1 per 1,000 checkBotId() calls on Pro, custom on Enterprise; AI Bots managed ruleset on all plans. | ## Verdicts by reader Platform engineer Pick by host: both are zone or project toggles. Security engineer Both verify Web Bot Auth; Vercel exposes isVerifiedBot in checkBotId(). Publisher operations Cloudflare keeps Googlebot, Applebot and Bingbot in search while refusing training; Vercel blocks by category. ## What each does not do ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. ### Vercel - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). More: [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/), [alternatives to Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/). ## Sources - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [vercel.com: vercels bot verification now supports web bot auth](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [vercel.com: verified bots](https://vercel.com/docs/botid/verified-bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DataDome vs Akamai | Agent Ready Crawlers" description: "DataDome vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/datadome-vs-akamai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # DataDome vs Akamai | Agent Ready Crawlers > DataDome vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/datadome-vs-akamai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # DataDome vs Akamai In short DataDome publishes plan prices and lists AI monetization from USD 8,670 a month; Akamai publishes no prices and names Visa TAP, Skyfire and Experian KYA, Auth0 and TollBit in its June 2026 agentic framework. On charging the rubric grades DataDome yes and Akamai partial, and on commerce agents the rubric grades DataDome partial and Akamai yes. ## The sourced difference DataDome publishes plan prices and lists AI monetization from USD 8,670 a month; Akamai publishes no prices and names Visa TAP, Skyfire and Experian KYA, Auth0 and TollBit in its June 2026 agentic framework. [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04. ## Side by side | Question | DataDome | Akamai | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | Yes; Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | | Per-purpose policy (train, search, answer, act) | Partial; policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | Partial; allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | | Charging for access | Yes; AI crawler and agent monetization through partners, from the Advanced plan | Partial; through TollBit and Skyfire partners, no native billing | | Lets signed commerce agents through | Partial; TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | Yes; Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | | Cost | published: from USD 3,830/month | no public price; sales-led | | Reporting | Yes; agent traffic by intent, identity, geo, URL and endpoint type | Yes; TrafficPeak and web security analytics | | Where it runs | both: server-side module on web servers, CDNs or API gateways plus client-side tag | edge: verification and enforcement on Akamai's edge network | | Points, rank | 8 of 10, rank 1 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer Akamai runs at its edge; DataDome sits in your server, CDN or API gateway plus a client tag. Security engineer Both accept Web Bot Auth; Akamai has a named Visa TAP collaboration. Publisher operations Akamai suits sites already on Akamai; DataDome is CDN-agnostic. ## What each does not do ### DataDome - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). ### Akamai - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). More: [alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/), [alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/). ## Sources - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DataDome vs HUMAN Security | Agent Ready Crawlers" description: "DataDome vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/datadome-vs-human-security site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # DataDome vs HUMAN Security | Agent Ready Crawlers > DataDome vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/datadome-vs-human-security Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # DataDome vs HUMAN Security In short DataDome publishes plan prices (from USD 3,830 a month); HUMAN publishes none and rates agents High, Medium or Low trust, with High reserved for agents that carry a cryptographic key on each request. On per-purpose policy the rubric grades DataDome partial and HUMAN Security yes, and on charging the rubric grades DataDome yes and HUMAN Security partial. ## The sourced difference DataDome publishes plan prices (from USD 3,830 a month); HUMAN publishes none and rates agents High, Medium or Low trust, with High reserved for agents that carry a cryptographic key on each request. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04. ## Side by side | Question | DataDome | HUMAN Security | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | Yes; per-request cryptographic keys (e.g. ChatGPT's published key), plus behavior and private signals; Web Bot Auth not named on pages read | | Per-purpose policy (train, search, answer, act) | Partial; policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | Yes; per-action categories (content, engage, account creation, checkout, rate limit) by trust level | | Charging for access | Yes; AI crawler and agent monetization through partners, from the Advanced plan | Partial; through scraping monetization partners (TollBit logo on product page); no native charging found | | Lets signed commerce agents through | Partial; TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | published: from USD 3,830/month | no public price; sales-led | | Reporting | Yes; agent traffic by intent, identity, geo, URL and endpoint type | Yes; user and session insights on agent actions, monthly State of Agentic Traffic benchmark | | Where it runs | both: server-side module on web servers, CDNs or API gateways plus client-side tag | platform: Sightline integrates with the customer's stack; exact deployment points not confirmed on pages read | | Points, rank | 8 of 10, rank 1 (tied) | 7 of 10, rank 5 | | Price read 2026-11-04 | Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer Both need integration work; DataDome documents a server module plus client tag, HUMAN integrates through Sightline. Security engineer HUMAN lets you set permissions per action (content, account creation, checkout) by trust level; DataDome adds intent models for scraping, scalping and payment fraud. Publisher operations Both are fraud-led; neither replaces robots.txt for training opt-outs. ## What each does not do ### DataDome - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). ### HUMAN Security - Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02). - No native pay-per-crawl; monetization is through partner tools (AgenticTrust page). - No published price list (package plans page). More: [alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/), [alternatives to HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/alternatives/). ## Sources - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [humansecurity.com: agentic ai](https://www.humansecurity.com/applications/agentic-ai/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DataDome vs Kasada | Agent Ready Crawlers" description: "DataDome vs Kasada: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/datadome-vs-kasada site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # DataDome vs Kasada | Agent Ready Crawlers > DataDome vs Kasada: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/datadome-vs-kasada Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # DataDome vs Kasada In short DataDome publishes prices and monetization tiers; Kasada publishes none but says it can deploy behind an incumbent bot manager. On charging the rubric grades DataDome yes and Kasada not published, and on commerce agents the rubric grades DataDome partial and Kasada not published. For a security engineer: Both verify Web Bot Auth; DataDome adds KYA tokens and reverse DNS. ## The sourced difference DataDome publishes prices and monetization tiers; Kasada publishes none but says it can deploy behind an incumbent bot manager. [Source](https://www.kasada.io/), read 2026-11-04. ## Side by side | Question | DataDome | Kasada | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | Yes; Web Bot Auth plus Kasada's verified agent directory | | Per-purpose policy (train, search, answer, act) | Partial; policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | Partial; endpoint-level access policies per verified agent; no published search versus training split (checked 2026-11-04) | | Charging for access | Yes; AI crawler and agent monetization through partners, from the Advanced plan | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Partial; TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | published: from USD 3,830/month | no public price; sales-led | | Reporting | Yes; agent traffic by intent, identity, geo, URL and endpoint type | Yes; Kasada Portal and Custom Reporting for verified agent activity | | Where it runs | both: server-side module on web servers, CDNs or API gateways plus client-side tag | edge: enforcement upstream of origin | | Points, rank | 8 of 10, rank 1 (tied) | 5 of 10, rank 9 | | Price read 2026-11-04 | Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. | no public price; sales-led (checked 2026-11-04). Indirectly, Vercel BotID Deep Analysis (powered by Kasada) costs USD 1 per 1,000 checkBotId() calls on Vercel Pro. | ## Verdicts by reader Platform engineer Kasada can be layered; DataDome usually replaces. Security engineer Both verify Web Bot Auth; DataDome adds KYA tokens and reverse DNS. Publisher operations DataDome can monetize through partners; Kasada lists no charging. ## What each does not do ### DataDome - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). ### Kasada - No monetization or pay-per-crawl feature in AI Agent Trust (launch post, 2026-01-21). - No public price list (homepage read 2026-11-04). - No published support for Visa TAP or Mastercard Agent Pay (pages read 2026-11-04). More: [alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/), [alternatives to Kasada](https://agentreadycrawlers.com/vendors/kasada/alternatives/). ## Sources - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Fastly vs Akamai | Agent Ready Crawlers" description: "Fastly vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/fastly-vs-akamai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Fastly vs Akamai | Agent Ready Crawlers > Fastly vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/fastly-vs-akamai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Fastly vs Akamai In short Both route AI bot charging to TollBit, and both partner with Skyfire (Fastly 2026-06-10, Akamai 2026-06-15); Akamai documents Web Bot Auth verification and Visa TAP, Fastly docs read did not. On verifies agents the rubric grades Fastly partial and Akamai yes, and on commerce agents the rubric grades Fastly partial and Akamai yes. For a platform engineer: Both are edge. ## The sourced difference Both route AI bot charging to TollBit, and both partner with Skyfire (Fastly 2026-06-10, Akamai 2026-06-15); Akamai documents Web Bot Auth verification and Visa TAP, Fastly docs read did not. [Source](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce), read 2026-11-04. ## Side by side | Question | Fastly | Akamai | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; verified-bot signals for AI crawlers and fetchers; Skyfire credentials at the edge (announced); Web Bot Auth not found in docs (checked 2026-11-04) | Yes; Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | | Per-purpose policy (train, search, answer, act) | Partial; crawler versus fetcher signals, suspected versus verified | Partial; allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | | Charging for access | Partial; through TollBit paywall redirect | Partial; through TollBit and Skyfire partners, no native billing | | Lets signed commerce agents through | Partial; Skyfire KYA/KYAPay partnership (Jun 2026); no Visa TAP or Mastercard Agent Pay found | Yes; Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | | Cost | no public price; sales-led | no public price; sales-led | | Reporting | Not published; AI bot reporting not described on pages read (checked 2026-11-04) | Yes; TrafficPeak and web security analytics | | Where it runs | edge: classification and actions on Fastly's edge network | edge: verification and enforcement on Akamai's edge network | | Points, rank | 4 of 10, rank 10 | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public price for Bot Management or AI Bot Management; contact sales (checked 2026-11-04). Free tier covers up to 100,000 requests/month for delivery. | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer Both are edge; both sales-led. Security engineer Akamai has the stronger published verification story on the pages read. Publisher operations Equivalent charging path through TollBit. ## What each does not do ### Fastly - Bot Management is not available on the Essential platform for Next-Gen WAF (Bot Management docs). - No native per-crawl billing; charging relies on TollBit (Fastly-TollBit blog). - No public price for bot products (pricing page). ### Akamai - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). More: [alternatives to Fastly](https://agentreadycrawlers.com/vendors/fastly/alternatives/), [alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/). ## Sources - [investors.fastly.com: fastly and skyfire partner enable trusted agentic commerce](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce) read 2026-11-04 - [fastly.com: pricing](https://www.fastly.com/pricing) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Fastly vs Cloudflare | Agent Ready Crawlers" description: "Fastly vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/fastly-vs-cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Fastly vs Cloudflare | Agent Ready Crawlers > Fastly vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/fastly-vs-cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Fastly vs Cloudflare In short Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. On verifies agents the rubric grades Fastly partial and Cloudflare yes, and on per-purpose policy the rubric grades Fastly partial and Cloudflare yes. ## The sourced difference Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. ## Side by side | Question | Fastly | Cloudflare | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; verified-bot signals for AI crawlers and fetchers; Skyfire credentials at the edge (announced); Web Bot Auth not found in docs (checked 2026-11-04) | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | | Per-purpose policy (train, search, answer, act) | Partial; crawler versus fetcher signals, suspected versus verified | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | | Charging for access | Partial; through TollBit paywall redirect | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | | Lets signed commerce agents through | Partial; Skyfire KYA/KYAPay partnership (Jun 2026); no Visa TAP or Mastercard Agent Pay found | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | | Cost | no public price; sales-led | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | Reporting | Not published; AI bot reporting not described on pages read (checked 2026-11-04) | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | | Where it runs | edge: classification and actions on Fastly's edge network | edge: enforced on Cloudflare's reverse proxy, no app code needed | | Points, rank | 4 of 10, rank 10 | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public price for Bot Management or AI Bot Management; contact sales (checked 2026-11-04). Free tier covers up to 100,000 requests/month for delivery. | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | ## Verdicts by reader Platform engineer Both are edge platforms; Fastly bot features are contact-sales. Security engineer Fastly docs read did not mention Web Bot Auth (checked 2026-11-04); Cloudflare verifies it. Publisher operations Cloudflare offers a free training opt-out; Fastly needs a sales contract. ## What each does not do ### Fastly - Bot Management is not available on the Essential platform for Next-Gen WAF (Bot Management docs). - No native per-crawl billing; charging relies on TollBit (Fastly-TollBit blog). - No public price for bot products (pricing page). ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. More: [alternatives to Fastly](https://agentreadycrawlers.com/vendors/fastly/alternatives/), [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/). ## Sources - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [fastly.com: pricing](https://www.fastly.com/pricing) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [investors.fastly.com: fastly and skyfire partner enable trusted agentic commerce](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "HUMAN Security vs Kasada | Agent Ready Crawlers" description: "HUMAN Security vs Kasada: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/human-security-vs-kasada site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # HUMAN Security vs Kasada | Agent Ready Crawlers > HUMAN Security vs Kasada: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/human-security-vs-kasada Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # HUMAN Security vs Kasada In short Kasada launched AI Agent Trust on 2026-01-21 with a verified agent directory that supports Web Bot Auth; HUMAN's AgenticTrust grades agents by trust level and per-action permissions, without naming Web Bot Auth on the pages read. For a security engineer: Kasada enforces upstream of origin; HUMAN reports spoofed AI crawler user agents at 5.7 percent of such traffic. ## The sourced difference Kasada launched AI Agent Trust on 2026-01-21 with a verified agent directory that supports Web Bot Auth; HUMAN's AgenticTrust grades agents by trust level and per-action permissions, without naming Web Bot Auth on the pages read. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04. ## Side by side | Question | HUMAN Security | Kasada | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; per-request cryptographic keys (e.g. ChatGPT's published key), plus behavior and private signals; Web Bot Auth not named on pages read | Yes; Web Bot Auth plus Kasada's verified agent directory | | Per-purpose policy (train, search, answer, act) | Yes; per-action categories (content, engage, account creation, checkout, rate limit) by trust level | Partial; endpoint-level access policies per verified agent; no published search versus training split (checked 2026-11-04) | | Charging for access | Partial; through scraping monetization partners (TollBit logo on product page); no native charging found | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | no public price; sales-led | no public price; sales-led | | Reporting | Yes; user and session insights on agent actions, monthly State of Agentic Traffic benchmark | Yes; Kasada Portal and Custom Reporting for verified agent activity | | Where it runs | platform: Sightline integrates with the customer's stack; exact deployment points not confirmed on pages read | edge: enforcement upstream of origin | | Points, rank | 7 of 10, rank 5 | 5 of 10, rank 9 | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04) | no public price; sales-led (checked 2026-11-04). Indirectly, Vercel BotID Deep Analysis (powered by Kasada) costs USD 1 per 1,000 checkBotId() calls on Vercel Pro. | ## Verdicts by reader Platform engineer Kasada says it can sit behind an incumbent bot manager; HUMAN integrates through Sightline. Security engineer Kasada enforces upstream of origin; HUMAN reports spoofed AI crawler user agents at 5.7 percent of such traffic. Publisher operations Neither publishes prices or AI charging. ## What each does not do ### HUMAN Security - Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02). - No native pay-per-crawl; monetization is through partner tools (AgenticTrust page). - No published price list (package plans page). ### Kasada - No monetization or pay-per-crawl feature in AI Agent Trust (launch post, 2026-01-21). - No public price list (homepage read 2026-11-04). - No published support for Visa TAP or Mastercard Agent Pay (pages read 2026-11-04). More: [alternatives to HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/alternatives/), [alternatives to Kasada](https://agentreadycrawlers.com/vendors/kasada/alternatives/). ## Sources - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [humansecurity.com: agentic ai](https://www.humansecurity.com/applications/agentic-ai/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Imperva vs Cloudflare | Agent Ready Crawlers" description: "Imperva vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/imperva-vs-cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Imperva vs Cloudflare | Agent Ready Crawlers > Imperva vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/imperva-vs-cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Imperva vs Cloudflare In short Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. On verifies agents the rubric grades Imperva partial and Cloudflare yes, and on per-purpose policy the rubric grades Imperva not published and Cloudflare yes. For publisher operations: Imperva lists no AI charging. ## The sourced difference Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. [Source](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), read 2026-11-04. ## Side by side | Question | Imperva | Cloudflare | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; verified trust for verified AI agents in Imperva for Google Cloud (controlled availability); no Web Bot Auth support found (checked 2026-11-04) | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | | Per-purpose policy (train, search, answer, act) | Not published; per-purpose AI crawler policy (checked 2026-11-04) | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | | Charging for access | Not published; support (checked 2026-11-04) | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | | Cost | no public price; sales-led | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | Reporting | Yes; customizable dashboards and reports | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | | Where it runs | edge or cloud: Imperva cloud WAF stack, Imperva for AWS and Imperva for Google Cloud | edge: enforced on Cloudflare's reverse proxy, no app code needed | | Points, rank | 3 of 10, rank 11 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04) | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | ## Verdicts by reader Platform engineer Imperva ABP is a sales-led bundle; Cloudflare starts free. Security engineer No Web Bot Auth support found for Imperva (checked 2026-11-04). Publisher operations Imperva lists no AI charging. ## What each does not do ### Imperva - No Web Bot Auth or signed-agent verification found on Imperva or Thales pages read (checked 2026-11-04). - No AI crawler monetization or pay-per-crawl feature found (ABP page). - Agent verified-trust controls are limited to Imperva for Google Cloud and are in Controlled Availability (Thales blog, 2026-05-04). ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. More: [alternatives to Imperva](https://agentreadycrawlers.com/vendors/imperva/alternatives/), [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/). ## Sources - [cpl.thalesgroup.com: thales google protect ai agent ecosystem](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem) read 2026-11-04 - [imperva.com: advanced bot protection management](https://www.imperva.com/products/advanced-bot-protection-management/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Imperva vs DataDome | Agent Ready Crawlers" description: "Imperva vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/imperva-vs-datadome site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Imperva vs DataDome | Agent Ready Crawlers > Imperva vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/imperva-vs-datadome Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Imperva vs DataDome In short Imperva's product page names agentic AI as a traffic class but lists no AI crawler or agent controls; DataDome documents Web Bot Auth, KYA, IP lists and reverse DNS for agent identification. On verifies agents the rubric grades Imperva partial and DataDome yes, and on per-purpose policy the rubric grades Imperva not published and DataDome partial. ## The sourced difference Imperva's product page names agentic AI as a traffic class but lists no AI crawler or agent controls; DataDome documents Web Bot Auth, KYA, IP lists and reverse DNS for agent identification. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04. ## Side by side | Question | Imperva | DataDome | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; verified trust for verified AI agents in Imperva for Google Cloud (controlled availability); no Web Bot Auth support found (checked 2026-11-04) | Yes; Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | | Per-purpose policy (train, search, answer, act) | Not published; per-purpose AI crawler policy (checked 2026-11-04) | Partial; policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | | Charging for access | Not published; support (checked 2026-11-04) | Yes; AI crawler and agent monetization through partners, from the Advanced plan | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Partial; TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | | Cost | no public price; sales-led | published: from USD 3,830/month | | Reporting | Yes; customizable dashboards and reports | Yes; agent traffic by intent, identity, geo, URL and endpoint type | | Where it runs | edge or cloud: Imperva cloud WAF stack, Imperva for AWS and Imperva for Google Cloud | both: server-side module on web servers, CDNs or API gateways plus client-side tag | | Points, rank | 3 of 10, rank 11 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04) | Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. | ## Verdicts by reader Platform engineer Both need integration; DataDome publishes how. Security engineer DataDome has the documented agent identity stack. Publisher operations DataDome lists monetization from Advanced; Imperva none. ## What each does not do ### Imperva - No Web Bot Auth or signed-agent verification found on Imperva or Thales pages read (checked 2026-11-04). - No AI crawler monetization or pay-per-crawl feature found (ABP page). - Agent verified-trust controls are limited to Imperva for Google Cloud and are in Controlled Availability (Thales blog, 2026-05-04). ### DataDome - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). More: [alternatives to Imperva](https://agentreadycrawlers.com/vendors/imperva/alternatives/), [alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/). ## Sources - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [imperva.com: advanced bot protection management](https://www.imperva.com/products/advanced-bot-protection-management/) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [cpl.thalesgroup.com: thales google protect ai agent ecosystem](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Kasada vs Akamai | Agent Ready Crawlers" description: "Kasada vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/kasada-vs-akamai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Kasada vs Akamai | Agent Ready Crawlers > Kasada vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/kasada-vs-akamai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Kasada vs Akamai In short Kasada says it can be deployed behind an incumbent bot manager; Akamai's Bot & Agent Control runs on Akamai's own edge with partner integrations for identity and charging. On charging the rubric grades Kasada not published and Akamai partial, and on commerce agents the rubric grades Kasada not published and Akamai yes. For publisher operations: Only Akamai lists charging partners. ## The sourced difference Kasada says it can be deployed behind an incumbent bot manager; Akamai's Bot & Agent Control runs on Akamai's own edge with partner integrations for identity and charging. [Source](https://www.kasada.io/), read 2026-11-04. ## Side by side | Question | Kasada | Akamai | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth plus Kasada's verified agent directory | Yes; Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | | Per-purpose policy (train, search, answer, act) | Partial; endpoint-level access policies per verified agent; no published search versus training split (checked 2026-11-04) | Partial; allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | | Charging for access | Not published; support (checked 2026-11-04) | Partial; through TollBit and Skyfire partners, no native billing | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Yes; Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | | Cost | no public price; sales-led | no public price; sales-led | | Reporting | Yes; Kasada Portal and Custom Reporting for verified agent activity | Yes; TrafficPeak and web security analytics | | Where it runs | edge: enforcement upstream of origin | edge: verification and enforcement on Akamai's edge network | | Points, rank | 5 of 10, rank 9 | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04). Indirectly, Vercel BotID Deep Analysis (powered by Kasada) costs USD 1 per 1,000 checkBotId() calls on Vercel Pro. | no public price; sales-led (checked 2026-11-04) | ## Verdicts by reader Platform engineer Kasada can sit behind Akamai. Security engineer Both support Web Bot Auth. Publisher operations Only Akamai lists charging partners. ## What each does not do ### Kasada - No monetization or pay-per-crawl feature in AI Agent Trust (launch post, 2026-01-21). - No public price list (homepage read 2026-11-04). - No published support for Visa TAP or Mastercard Agent Pay (pages read 2026-11-04). ### Akamai - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). More: [alternatives to Kasada](https://agentreadycrawlers.com/vendors/kasada/alternatives/), [alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/). ## Sources - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Kasada vs Vercel | Agent Ready Crawlers" description: "Kasada vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Kasada vs Vercel | Agent Ready Crawlers > Kasada vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Kasada vs Vercel In short Kasada powers Vercel BotID Deep Analysis, sold by Vercel at USD 1 per 1,000 checkBotId() calls on Pro; Kasada itself is sales-led and adds an AI agent directory and edge policies. On per-purpose policy the rubric grades Kasada partial and Vercel yes. For a security engineer: Kasada direct adds Web Bot Auth directory and endpoint policies; BotID returns verified-bot fields. ## The sourced difference Kasada powers Vercel BotID Deep Analysis, sold by Vercel at USD 1 per 1,000 checkBotId() calls on Pro; Kasada itself is sales-led and adds an AI agent directory and edge policies. [Source](https://vercel.com/docs/botid), read 2026-11-04. ## Side by side | Question | Kasada | Vercel | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; Web Bot Auth plus Kasada's verified agent directory | Yes; IP, reverse DNS and Web Bot Auth; directory at bots.fyi | | Per-purpose policy (train, search, answer, act) | Partial; endpoint-level access policies per verified agent; no published search versus training split (checked 2026-11-04) | Yes; AI crawlers and AI assistants handled separately since 2026-10-29 | | Charging for access | Not published; support (checked 2026-11-04) | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | no public price; sales-led | published: Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Reporting | Yes; Kasada Portal and Custom Reporting for verified agent activity | Yes; Firewall tab BotID filter and Observability Plus metrics | | Where it runs | edge: enforcement upstream of origin | both: managed rulesets at Vercel's edge firewall; BotID verdict read in server code via checkBotId() | | Points, rank | 5 of 10, rank 9 | 6 of 10, rank 6 (tied) | | Price read 2026-11-04 | no public price; sales-led (checked 2026-11-04). Indirectly, Vercel BotID Deep Analysis (powered by Kasada) costs USD 1 per 1,000 checkBotId() calls on Vercel Pro. | BotID Basic free on all plans; Deep Analysis USD 1 per 1,000 checkBotId() calls on Pro, custom on Enterprise; AI Bots managed ruleset on all plans. | ## Verdicts by reader Platform engineer On Vercel you get Kasada's detection without a separate contract. Security engineer Kasada direct adds Web Bot Auth directory and endpoint policies; BotID returns verified-bot fields. Publisher operations Neither charges bots. ## What each does not do ### Kasada - No monetization or pay-per-crawl feature in AI Agent Trust (launch post, 2026-01-21). - No public price list (homepage read 2026-11-04). - No published support for Visa TAP or Mastercard Agent Pay (pages read 2026-11-04). ### Vercel - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). More: [alternatives to Kasada](https://agentreadycrawlers.com/vendors/kasada/alternatives/), [alternatives to Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/). ## Sources - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 - [vercel.com: vercels bot verification now supports web bot auth](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [vercel.com: verified bots](https://vercel.com/docs/botid/verified-bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Skyfire vs Cloudflare | Agent Ready Crawlers" description: "Skyfire vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Skyfire vs Cloudflare | Agent Ready Crawlers > Skyfire vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Skyfire vs Cloudflare In short Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. On per-purpose policy the rubric grades Skyfire not its job and Cloudflare yes, and on charging the rubric grades Skyfire yes and Cloudflare partial. ## The sourced difference Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. Skyfire documents enforcing its tokens on Cloudflare with a Worker. [Source](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md), read 2026-11-04. ## Side by side | Question | Skyfire | Cloudflare | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; KYA JWTs that sites verify via Skyfire's JWKS; not a bot manager itself | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | | Per-purpose policy (train, search, answer, act) | Not its job; sellers decide access from token claims such as agent and principal identity | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | | Charging for access | Yes; pay and kya-pay tokens charged by sellers, settled between Skyfire wallets | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | | Lets signed commerce agents through | Yes; KYAPay carries identity plus payment intent; card payment flow documented | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | | Cost | not published (checked 2026-11-04) | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | Reporting | Not published; on pages read (checked 2026-11-04) | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | | Where it runs | either: sellers verify tokens in their backend, in a Cloudflare Worker, or through partner bot managers (F5, Akamai, Fastly) | edge: enforced on Cloudflare's reverse proxy, no app code needed | | Points, rank | 6 of 10, rank 6 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | no public fee schedule found on homepage or docs index (checked 2026-11-04) | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | ## Verdicts by reader Platform engineer You can run both: Cloudflare for identity at the edge, a Worker for Skyfire tokens. Security engineer Different trust anchors: one issuer (Skyfire) versus each operator's own keys (Web Bot Auth). Publisher operations Skyfire carries committed funds; Cloudflare pay per crawl uses its own billing. ## What each does not do ### Skyfire - Does not detect or block unsigned bots; it only issues and settles tokens that sites or bot managers must check (docs). - KYA tokens alone cannot be charged; only pay and kya-pay tokens move money (introduction docs). - No published fee schedule (homepage and docs read 2026-11-04). ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. More: [alternatives to Skyfire](https://agentreadycrawlers.com/vendors/skyfire/alternatives/), [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/). ## Sources - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [kyapay.org: kyapay.org](https://kyapay.org/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Skyfire vs Stytch IsAgent | Agent Ready Crawlers" description: "Skyfire vs Stytch IsAgent: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Skyfire vs Stytch IsAgent | Agent Ready Crawlers > Skyfire vs Stytch IsAgent: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Skyfire vs Stytch IsAgent In short Skyfire issues signed KYA and pay tokens that sellers verify against its JWKS; Stytch IsAgent is a front-end flag that Stytch says must not be used as a security control. On verifies agents the rubric grades Skyfire yes and Stytch IsAgent partial, and on per-purpose policy the rubric grades Skyfire not its job and Stytch IsAgent not published. ## The sourced difference Skyfire issues signed KYA and pay tokens that sellers verify against its JWKS; Stytch IsAgent is a front-end flag that Stytch says must not be used as a security control. [Source](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0), read 2026-11-04. ## Side by side | Question | Skyfire | Stytch IsAgent | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; KYA JWTs that sites verify via Skyfire's JWKS; not a bot manager itself | Partial; Web Bot Auth signatures plus heuristic detection (user agent, TLS fingerprint, network traits) | | Per-purpose policy (train, search, answer, act) | Not its job; sellers decide access from token claims such as agent and principal identity | Not published; per-purpose policy (checked 2026-11-04) | | Charging for access | Yes; pay and kya-pay tokens charged by sellers, settled between Skyfire wallets | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Yes; KYAPay carries identity plus payment intent; card payment flow documented | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | not published (checked 2026-11-04) | published: IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Reporting | Not published; on pages read (checked 2026-11-04) | Partial; client hint with agent flag and identity, meant for analytics | | Where it runs | either: sellers verify tokens in their backend, in a Cloudflare Worker, or through partner bot managers (F5, Akamai, Fastly) | app: client-side SDK calling Stytch's API, plus backend Lookup API for fingerprinting | | Points, rank | 6 of 10, rank 6 (tied) | 2 of 10, rank 14 | | Price read 2026-11-04 | no public fee schedule found on homepage or docs index (checked 2026-11-04) | IsAgent free up to 10,000 calls/month (contact Stytch above that); Device Fingerprinting USD 0.005 per fingerprint beyond 10,000 included; free tier 10,000 MAUs and AI agents. | ## Verdicts by reader Platform engineer IsAgent is a client hint for analytics and agent-friendly flows; Skyfire tokens are checked server side or in a Worker. Security engineer Only Skyfire gives you a verifiable credential; IsAgent can be altered in the browser, per Stytch. Publisher operations Skyfire carries payment intent; IsAgent carries none. ## What each does not do ### Skyfire - Does not detect or block unsigned bots; it only issues and settles tokens that sites or bot managers must check (docs). - KYA tokens alone cannot be charged; only pay and kya-pay tokens move money (introduction docs). - No published fee schedule (homepage and docs read 2026-11-04). ### Stytch IsAgent - IsAgent is not a security control; its client-side result can be tampered with (Stytch changelog, 2025-08-01). - Detection is heuristic and not 100% accurate (IsAgent FAQ). - No crawler monetization or per-purpose crawler policy (pages read 2026-11-04). More: [alternatives to Skyfire](https://agentreadycrawlers.com/vendors/skyfire/alternatives/), [alternatives to Stytch IsAgent](https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives/). ## Sources - [changelog.stytch.com: ann\_iPdXgiqlNrBk0](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [stytch.com: pricing](https://stytch.com/pricing) read 2026-11-04 - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [changelog.stytch.com: ann\_uSpDBNlbdXL8X](https://changelog.stytch.com/announcements/ann_uSpDBNlbdXL8X) read 2026-11-04 - [isagent-web.vercel.app: faq](https://isagent-web.vercel.app/faq) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [kyapay.org: kyapay.org](https://kyapay.org/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Skyfire vs TollBit | Agent Ready Crawlers" description: "Skyfire vs TollBit: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/skyfire-vs-tollbit site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Skyfire vs TollBit | Agent Ready Crawlers > Skyfire vs TollBit: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/skyfire-vs-tollbit Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Skyfire vs TollBit In short Skyfire is the payer side: agents carry kya-pay tokens with committed funds; TollBit is the publisher side: a paywall with publisher-set rates that AI companies access with TollBit tokens. On verifies agents the rubric grades Skyfire yes and TollBit partial, and on per-purpose policy the rubric grades Skyfire not its job and TollBit partial. ## The sourced difference Skyfire is the payer side: agents carry kya-pay tokens with committed funds; TollBit is the publisher side: a paywall with publisher-set rates that AI companies access with TollBit tokens. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04. ## Side by side | Question | Skyfire | TollBit | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; KYA JWTs that sites verify via Skyfire's JWKS; not a bot manager itself | Partial; issues access tokens to AI companies and depends on CDN or bot-manager partners for detection; Web Bot Auth support not confirmed (checked 2026-11-04) | | Per-purpose policy (train, search, answer, act) | Not its job; sellers decide access from token claims such as agent and principal identity | Partial; rules and rates per bot set by the publisher | | Charging for access | Yes; pay and kya-pay tokens charged by sellers, settled between Skyfire wallets | Yes; bot paywall with publisher-set usage-based pricing | | Lets signed commerce agents through | Yes; KYAPay carries identity plus payment intent; card payment flow documented | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | not published (checked 2026-11-04) | free for publishers to start | | Reporting | Not published; on pages read (checked 2026-11-04) | Yes; TollBit Analytics and portfolio analytics across sites | | Where it runs | either: sellers verify tokens in their backend, in a Cloudflare Worker, or through partner bot managers (F5, Akamai, Fastly) | integrates at the CDN or bot-manager layer (Fastly, Akamai, DataDome, Arc XP) by redirect | | Points, rank | 6 of 10, rank 6 (tied) | 6 of 10, rank 6 (tied) | | Price read 2026-11-04 | no public fee schedule found on homepage or docs index (checked 2026-11-04) | Free for publishers to start ('Get started free'); publishers set their own per-access rates; TollBit's fees to AI companies are not published (checked 2026-11-04). | ## Verdicts by reader Platform engineer Skyfire tokens are verified in your backend or a Worker; TollBit is reached by redirect from your CDN. Security engineer Skyfire tokens are ES256 JWTs checked against a JWKS; TollBit identity relies on partners. Publisher operations TollBit is free to start for publishers; Skyfire publishes no fee schedule (checked 2026-11-04). ## What each does not do ### Skyfire - Does not detect or block unsigned bots; it only issues and settles tokens that sites or bot managers must check (docs). - KYA tokens alone cannot be charged; only pay and kya-pay tokens move money (introduction docs). - No published fee schedule (homepage and docs read 2026-11-04). ### TollBit - Does not detect bots on its own network edge; it relies on CDN and bot-management partners to classify and redirect (Fastly integration blog). - No published fee schedule for AI companies (homepage read 2026-11-04). - Usage statistics are TollBit's own counts, not independently audited (homepage). More: [alternatives to Skyfire](https://agentreadycrawlers.com/vendors/skyfire/alternatives/), [alternatives to TollBit](https://agentreadycrawlers.com/vendors/tollbit/alternatives/). ## Sources - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [kyapay.org: kyapay.org](https://kyapay.org/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Stytch IsAgent vs Auth0 for AI Agents | Agent Ready Crawlers" description: "Stytch IsAgent vs Auth0 for AI Agents: rubric grades side by side, the sourced difference and verdicts by reader type. Read 2026-11-04." canonical: https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Stytch IsAgent vs Auth0 for AI Agents | Agent Ready Crawlers > Stytch IsAgent vs Auth0 for AI Agents: rubric grades side by side, the sourced difference and verdicts by reader type. Read 2026-11-04. Source: https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Stytch IsAgent vs Auth0 for AI Agents In short Stytch IsAgent flags inbound agent traffic on a site; Auth0 for AI Agents gives identities and scoped tokens to agents a company builds, and does not police inbound bots. For a security engineer: Neither is a bot blocker; Stytch says IsAgent is not a security control. For publisher operations: Neither charges crawlers. ## The sourced difference Stytch IsAgent flags inbound agent traffic on a site; Auth0 for AI Agents gives identities and scoped tokens to agents a company builds, and does not police inbound bots. [Source](https://auth0.com/ai), read 2026-11-04. ## Side by side | Question | Stytch IsAgent | Auth0 for AI Agents | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; Web Bot Auth signatures plus heuristic detection (user agent, TLS fingerprint, network traits) | Not its job; issues identities and scoped tokens to the customer's own agents; does not verify inbound third-party bots | | Per-purpose policy (train, search, answer, act) | Not published; per-purpose policy (checked 2026-11-04) | Partial; fine-grained permissions and authorization for RAG scoped to each agent task | | Charging for access | Not published; support (checked 2026-11-04) | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Partial; product page mentions agentic protocols for commerce and procurement; no named TAP or Agent Pay support found | | Cost | published: IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Reporting | Partial; client hint with agent flag and identity, meant for analytics | Partial; audit trail for agent actions claimed | | Where it runs | app: client-side SDK calling Stytch's API, plus backend Lookup API for fingerprinting | app: SDKs and tenant APIs inside the agent and its backend | | Points, rank | 2 of 10, rank 14 | 3 of 10, rank 11 (tied) | | Price read 2026-11-04 | IsAgent free up to 10,000 calls/month (contact Stytch above that); Device Fingerprinting USD 0.005 per fingerprint beyond 10,000 included; free tier 10,000 MAUs and AI agents. | AI Agents add-on at 50 percent of the base plan price (USD 18 a month on B2C Essentials with 500 users, USD 75 on B2B Essentials); free plan available, paid tiers by monthly active users (checked 2026-11-05) | ## Verdicts by reader Platform engineer Use IsAgent on the receiving site; use Auth0 inside your own agent and backend. Security engineer Neither is a bot blocker; Stytch says IsAgent is not a security control. Publisher operations Neither charges crawlers. ## What each does not do ### Stytch IsAgent - IsAgent is not a security control; its client-side result can be tampered with (Stytch changelog, 2025-08-01). - Detection is heuristic and not 100% accurate (IsAgent FAQ). - No crawler monetization or per-purpose crawler policy (pages read 2026-11-04). ### Auth0 for AI Agents - Does not detect, verify or block third-party crawlers or agents hitting a website; it secures agents the customer builds (product page). - Agent Gateway for MCP and Agent as Principal are listed as shipping next, not available (product page read 2026-11-04). More: [alternatives to Stytch IsAgent](https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives/), [alternatives to Auth0 for AI Agents](https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives/). ## Sources - [auth0.com: ai](https://auth0.com/ai) read 2026-11-04 - [stytch.com: pricing](https://stytch.com/pricing) read 2026-11-04 - [auth0.com: pricing](https://auth0.com/pricing) read 2026-11-04 - [changelog.stytch.com: ann\_uSpDBNlbdXL8X](https://changelog.stytch.com/announcements/ann_uSpDBNlbdXL8X) read 2026-11-04 - [isagent-web.vercel.app: faq](https://isagent-web.vercel.app/faq) read 2026-11-04 - [auth0.com: overview](https://auth0.com/ai/docs/intro/overview) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "TollBit vs AWS WAF Bot Control | Agent Ready Crawlers" description: "TollBit vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # TollBit vs AWS WAF Bot Control | Agent Ready Crawlers > TollBit vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # TollBit vs AWS WAF Bot Control In short TollBit sells publisher analytics and a paywall reached by redirect; AWS WAF returns HTTP 402 itself and settles through Coinbase. On verifies agents the rubric grades TollBit partial and AWS WAF Bot Control yes, and on per-purpose policy the rubric grades TollBit partial and AWS WAF Bot Control yes. For a security engineer: AWS classification is probabilistic per AWS docs. ## The sourced difference TollBit sells publisher analytics and a paywall reached by redirect; AWS WAF returns HTTP 402 itself and settles through Coinbase. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. ## Side by side | Question | TollBit | AWS WAF Bot Control | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; issues access tokens to AI companies and depends on CDN or bot-manager partners for detection; Web Bot Auth support not confirmed (checked 2026-11-04) | Yes; Web Bot Auth (rule set Version_4.0+), verified self-identifying bots | | Per-purpose policy (train, search, answer, act) | Partial; rules and rates per bot set by the publisher | Yes; rules on bot category and name labels; prices can differ by bot identity and verification status | | Charging for access | Yes; bot paywall with publisher-set usage-based pricing | Yes; HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | free for publishers to start | published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Reporting | Yes; TollBit Analytics and portfolio analytics across sites | Yes; Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs | | Where it runs | integrates at the CDN or bot-manager layer (Fastly, Akamai, DataDome, Arc XP) by redirect | edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway | | Points, rank | 6 of 10, rank 6 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | Free for publishers to start ('Get started free'); publishers set their own per-access rates; TollBit's fees to AI companies are not published (checked 2026-11-04). | Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge. | ## Verdicts by reader Platform engineer AWS needs CloudFront; TollBit needs a partner CDN. Security engineer AWS classification is probabilistic per AWS docs; TollBit relies on partner detection. Publisher operations TollBit has cross-site reporting of bot scrapes; AWS has CloudWatch metrics. ## What each does not do ### TollBit - Does not detect bots on its own network edge; it relies on CDN and bot-management partners to classify and redirect (Fastly integration blog). - No published fee schedule for AI companies (homepage read 2026-11-04). - Usage statistics are TollBit's own counts, not independently audited (homepage). ### AWS WAF Bot Control - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). More: [alternatives to TollBit](https://agentreadycrawlers.com/vendors/tollbit/alternatives/), [alternatives to AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/). ## Sources - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "TollBit vs Cloudflare | Agent Ready Crawlers" description: "TollBit vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # TollBit vs Cloudflare | Agent Ready Crawlers > TollBit vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # TollBit vs Cloudflare In short TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. On verifies agents the rubric grades TollBit partial and Cloudflare yes, and on per-purpose policy the rubric grades TollBit partial and Cloudflare yes. ## The sourced difference TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. [Source](https://blog.cloudflare.com/introducing-pay-per-crawl/), read 2026-11-04. ## Side by side | Question | TollBit | Cloudflare | | --- | --- | --- | | Tells a verified agent from a spoof | Partial; issues access tokens to AI companies and depends on CDN or bot-manager partners for detection; Web Bot Auth support not confirmed (checked 2026-11-04) | Yes; Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | | Per-purpose policy (train, search, answer, act) | Partial; rules and rates per bot set by the publisher | Yes; separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | | Charging for access | Yes; bot paywall with publisher-set usage-based pricing | Partial; Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Partial; Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | | Cost | free for publishers to start | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | Reporting | Yes; TollBit Analytics and portfolio analytics across sites | Yes; AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | | Where it runs | integrates at the CDN or bot-manager layer (Fastly, Akamai, DataDome, Arc XP) by redirect | edge: enforced on Cloudflare's reverse proxy, no app code needed | | Points, rank | 6 of 10, rank 6 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | Free for publishers to start ('Get started free'); publishers set their own per-access rates; TollBit's fees to AI companies are not published (checked 2026-11-04). | AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). | ## Verdicts by reader Platform engineer TollBit needs a CDN or bot manager in front (Fastly, Akamai, DataDome); Cloudflare needs your zone on Cloudflare. Security engineer Both rely on upstream identity; TollBit issues tokens to AI companies, Cloudflare requires verified crawlers. Publisher operations TollBit is free to start for publishers and reports bot scrapes across a portfolio; Cloudflare pay per crawl is in private beta. ## What each does not do ### TollBit - Does not detect bots on its own network edge; it relies on CDN and bot-management partners to classify and redirect (Fastly integration blog). - No published fee schedule for AI companies (homepage read 2026-11-04). - Usage statistics are TollBit's own counts, not independently audited (homepage). ### Cloudflare - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. More: [alternatives to TollBit](https://agentreadycrawlers.com/vendors/tollbit/alternatives/), [alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/). ## Sources - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Vercel vs AWS WAF Bot Control | Agent Ready Crawlers" description: "Vercel vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Vercel vs AWS WAF Bot Control | Agent Ready Crawlers > Vercel vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Vercel vs AWS WAF Bot Control In short Vercel splits AI crawlers from AI assistants on all plans for free; AWS WAF labels bots by category and name and can charge them, at USD 10 a month per web ACL plus request fees. On charging the rubric grades Vercel not published and AWS WAF Bot Control yes. For a platform engineer: Choose by where the app is hosted. ## The sourced difference Vercel splits AI crawlers from AI assistants on all plans for free; AWS WAF labels bots by category and name and can charge them, at USD 10 a month per web ACL plus request fees. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. ## Side by side | Question | Vercel | AWS WAF Bot Control | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; IP, reverse DNS and Web Bot Auth; directory at bots.fyi | Yes; Web Bot Auth (rule set Version_4.0+), verified self-identifying bots | | Per-purpose policy (train, search, answer, act) | Yes; AI crawlers and AI assistants handled separately since 2026-10-29 | Yes; rules on bot category and name labels; prices can differ by bot identity and verification status | | Charging for access | Not published; support (checked 2026-11-04) | Yes; HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | | Cost | published: Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Reporting | Yes; Firewall tab BotID filter and Observability Plus metrics | Yes; Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs | | Where it runs | both: managed rulesets at Vercel's edge firewall; BotID verdict read in server code via checkBotId() | edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway | | Points, rank | 6 of 10, rank 6 (tied) | 8 of 10, rank 1 (tied) | | Price read 2026-11-04 | BotID Basic free on all plans; Deep Analysis USD 1 per 1,000 checkBotId() calls on Pro, custom on Enterprise; AI Bots managed ruleset on all plans. | Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge. | ## Verdicts by reader Platform engineer Choose by where the app is hosted; both are managed rules with no app code. Security engineer Both verify Web Bot Auth, so signed agents pass either one. Publisher operations Only AWS can charge AI bots, and only on CloudFront web ACLs. ## What each does not do ### Vercel - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). ### AWS WAF Bot Control - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). More: [alternatives to Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/), [alternatives to AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/). ## Sources - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [vercel.com: vercels bot verification now supports web bot auth](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [vercel.com: verified bots](https://vercel.com/docs/botid/verified-bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Vercel vs Netlify | Agent Ready Crawlers" description: "Vercel vs Netlify: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops." canonical: https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Comparisons" --- # Vercel vs Netlify | Agent Ready Crawlers > Vercel vs Netlify: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. Source: https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Head-to-head # Vercel vs Netlify In short Vercel verifies bots by IP, reverse DNS and Web Bot Auth; Netlify's AI bot control is a User Agent Blocker extension that matches strings only and can be active on one project at a time. On verifies agents the rubric grades Vercel yes and Netlify not published, and on per-purpose policy the rubric grades Vercel yes and Netlify partial. ## The sourced difference Vercel verifies bots by IP, reverse DNS and Web Bot Auth; Netlify's AI bot control is a User Agent Blocker extension that matches strings only and can be active on one project at a time. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04. ## Side by side | Question | Vercel | Netlify | | --- | --- | --- | | Tells a verified agent from a spoof | Yes; IP, reverse DNS and Web Bot Auth; directory at bots.fyi | Not published; support (checked 2026-11-04): blocking matches user-agent strings only | | Per-purpose policy (train, search, answer, act) | Yes; AI crawlers and AI assistants handled separately since 2026-10-29 | Partial; choose AI crawlers versus SEO or search crawlers from a preset list | | Charging for access | Not published; support (checked 2026-11-04) | Not published; support (checked 2026-11-04) | | Lets signed commerce agents through | Not published; Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not published; support (checked 2026-11-04) | | Cost | published: Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | published: platform plans from USD 0; extension price not stated | | Reporting | Yes; Firewall tab BotID filter and Observability Plus metrics | Not published; no AI bot reporting found (checked 2026-11-04) | | Where it runs | both: managed rulesets at Vercel's edge firewall; BotID verdict read in server code via checkBotId() | edge: Netlify Edge Function and edge firewall rules | | Points, rank | 6 of 10, rank 6 (tied) | 1 of 10, rank 15 | | Price read 2026-11-04 | BotID Basic free on all plans; Deep Analysis USD 1 per 1,000 checkBotId() calls on Pro, custom on Enterprise; AI Bots managed ruleset on all plans. | Free USD 0 (300 credits), Personal USD 9/month (1,000 credits), Pro USD 20/month (3,000 credits), Enterprise custom; Firewall Traffic Rules and basic rate limiting listed from Free; User Agent Blocker price not stated. | ## Verdicts by reader Platform engineer Netlify needs an Edge Function or extension; Vercel has a managed ruleset. Security engineer Netlify cannot tell a spoofed GPTBot from a real one; Vercel can. Publisher operations Both are free to start; Vercel separates crawlers from assistants. ## What each does not do ### Vercel - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). ### Netlify - No verified-bot list, reverse DNS or Web Bot Auth checks; user agents can be spoofed (block AI crawlers docs). - Preset list is incomplete by Netlify's own statement (block AI crawlers docs, 2026-09-18). - No AI crawler monetization (docs read 2026-11-04). More: [alternatives to Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/), [alternatives to Netlify](https://agentreadycrawlers.com/vendors/netlify/alternatives/). ## Sources - [docs.netlify.com: block ai crawlers](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [netlify.com: pricing](https://www.netlify.com/pricing/) read 2026-11-04 - [vercel.com: vercels bot verification now supports web bot auth](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [vercel.com: verified bots](https://vercel.com/docs/botid/verified-bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Contact | Agent Ready Crawlers" description: "How to reach the editors of Agent Ready Crawlers: send corrections, a missing crawler or agent, or a source for a vendor grade to the editor mailbox." canonical: https://agentreadycrawlers.com/contact site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Publication" --- # Contact | Agent Ready Crawlers > How to reach the editors of Agent Ready Crawlers: send corrections, a missing crawler or agent, or a source for a vendor grade to the editor mailbox. Source: https://agentreadycrawlers.com/contact Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Contact Write to [editor@agentreadycrawlers.com](mailto:editor@agentreadycrawlers.com). - **An error.** Name the page and give the source that shows the correct fact. It will be corrected and logged on [the corrections page](https://agentreadycrawlers.com/corrections/). - **A crawler or agent that is missing.** Send its exact user-agent string, its robots.txt token if it has one, and the operator's documentation URL. A row needs a source to be added. - **A vendor grade.** Send the public documentation URL that supports a different grade. Grades follow documents, so the change shows with the new URL and date. - **An operator that wants its documentation read again.** Send the URL of the page. ## Sources - [Editorial policy (this site)](https://agentreadycrawlers.com/editorial-policy/) read 2026-11-04 - [Corrections log (this site)](https://agentreadycrawlers.com/corrections/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Corrections and known gaps | Agent Ready Crawlers" description: "Corrections log for Agent Ready Crawlers, with the known gaps in the 2026-11-04 edition: what we could not confirm and where each gap is noted in the Index." canonical: https://agentreadycrawlers.com/corrections site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Publication" --- # Corrections and known gaps | Agent Ready Crawlers > Corrections log for Agent Ready Crawlers, with the known gaps in the 2026-11-04 edition: what we could not confirm and where each gap is noted in the Index. Source: https://agentreadycrawlers.com/corrections Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Corrections and known gaps ## Corrections log No corrections yet. The first edition was published on 2026-11-04. Each correction will appear here with its date, the page and what changed. To report an error, write to [editor@agentreadycrawlers.com](mailto:editor@agentreadycrawlers.com) with the page and the source that shows the correct fact. ## Known gaps in this edition These are the things we looked for on 2026-11-04 and could not confirm. They are listed so that an absence in the Index is not read as a fact. - Cloudflare Radar verified status was confirmed only for GPTBot and ChatGPT agent; other slugs (claudebot, bytespider) returned 404, so the full cross-check needs the Radar API with a token or a browser session. - Cloudflare's managed ruleset names for Visa TAP and Mastercard Agent Pay agents are not in its docs (checked 2026-11-04). - No operator publishes a public list of Visa TAP or Mastercard Agent Pay registered agents; the ledger says so for every row. - Claude Duet's Claude-Agent token, Muse, Comet, Copilot and Gemini auto-browse facts come from detection vendors (Verisoul, Centinel Analytica); Anthropic, Meta, Perplexity, Microsoft and Google publish nothing for site owners on these. - vendor\_rule\_names are filled only where an operator or vendor published them (OpenAI for ChatGPT agent, Perplexity for its bots); Akamai, Netlify and DataDome rule names for other crawlers were not found. ## Sources - [Editorial policy (this site)](https://agentreadycrawlers.com/editorial-policy/) read 2026-11-04 - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Crawler Index: every row | Agent Ready Crawlers" description: "All 70 AI crawlers and agents in the Crawler Index with operator, token, purpose, verification and block rate, and a profile page per row. Read 2026-11-05." canonical: https://agentreadycrawlers.com/crawlers site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Crawlers" --- # Crawler Index: every row | Agent Ready Crawlers > All 70 AI crawlers and agents in the Crawler Index with operator, token, purpose, verification and block rate, and a profile page per row. Read 2026-11-05. Source: https://agentreadycrawlers.com/crawlers Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The full Crawler Index In short The full Crawler Index: 70 rows from 32 operators, one profile page each. By purpose: 9 training, 15 search index, 8 live answer fetch, 20 task agent, 3 shopping agent, 15 other. Search filters the table as you type; the purpose pages are static lists. Block rates are from the 13,071-domain crawl of 2026-11-04. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) Operators with two or more rows have their own page: [Google](https://agentreadycrawlers.com/crawlers/operator/google/), [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/), [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/), [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/), [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/), [Perplexity](https://agentreadycrawlers.com/crawlers/operator/perplexity/), [Microsoft](https://agentreadycrawlers.com/crawlers/operator/microsoft/), [Mistral AI](https://agentreadycrawlers.com/crawlers/operator/mistral-ai/), [Apple](https://agentreadycrawlers.com/crawlers/operator/apple/), [Browserbase](https://agentreadycrawlers.com/crawlers/operator/browserbase/), [DuckDuckGo](https://agentreadycrawlers.com/crawlers/operator/duckduckgo/). All 70 rows, grouped by operator. Block rate: share of top-10,000 sites with a parsed robots.txt that disallow the token at the root. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Agent Ready Crawlers | agent-readiness-crawl | Other | Partly | None published | 4.2% | Profile | | Amazon | Amazonbot | Training | Yes | IP list | 15.6% | Profile | | Amazon | Amzn-SearchBot | Search index | Yes | IP list | not measured | Profile | | Amazon | Amzn-User | Live answer fetch | May not apply | IP list | not measured | Profile | | Amazon | Buy for Me (shopping agent) no token | Shopping agent | No docs | None published | no token | Profile | | Amazon | Rufus (shopping assistant) no token | Shopping agent | No docs | None published | no token | Profile | | Amazon Web Services | Amazon Bedrock AgentCore Browser no token | Task agent | No statement | Web Bot Auth | no token | Profile | | Anchor | Anchor Browser no token | Task agent | No docs | Web Bot Auth | no token | Profile | | Anthropic | anthropic-ai | Training | Legacy | None published | 15.5% | Profile | | Anthropic | Claude computer use (API tool) no token | Task agent | No docs | None published | no token | Profile | | Anthropic | Claude-Agent | Task agent | Reported | None published | not measured | Profile | | Anthropic | Claude in Chrome no token | Task agent | No statement | None published | no token | Profile | | Anthropic | Claude-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Anthropic | Claude-User | Live answer fetch | Yes | IP list | 8.1% | Profile | | Anthropic | ClaudeBot | Training | Yes | IP list | 17.6% | Profile | | Apify | Apify Website Content Crawler no token | Task agent | Setting | Web Bot Auth | no token | Profile | | Apple | Applebot | Search index | Yes | Reverse DNS | 14.8% | Profile | | Apple | Applebot-Extended | Training | Yes | None published | 14.8% | Profile | | Block | Goose (open-source agent) no token | Task agent | No docs | Cloudflare signed | no token | Profile | | Browser Use | Browser Use (library and cloud) no token | Task agent | No docs | None published | no token | Profile | | Browserbase | Browserbase cloud browsers no token | Task agent | No docs | Web Bot Auth | no token | Profile | | Browserbase | Stagehand (open-source framework) no token | Task agent | No docs | None published | no token | Profile | | ByteDance | Bytespider | Other | No docs | None published | 19.2% | Profile | | Cloudflare | Cloudflare Browser Rendering no token | Task agent | No statement | Web Bot Auth | no token | Profile | | Cohere | cohere-ai | Other | No docs | None published | 15.5% | Profile | | Common Crawl | CCBot | Training | Yes | Reverse DNS | 19.4% | Profile | | Diffbot | Diffbot | Other | Setting | None published | 15.7% | Profile | | DuckDuckGo | DuckAssistBot | Live answer fetch | Yes | IP list | 8.1% | Profile | | DuckDuckGo | DuckDuckBot | Search index | Yes | IP list | not measured | Profile | | Exa | ExaSearchBot | Search index | No docs | None published | not measured | Profile | | Firecrawl | FirecrawlAgent | Other | Yes | None published | not measured | Profile | | Google | Gemini in Chrome (auto browse) no token | Task agent | No docs | None published | no token | Profile | | Google | Google-Agent no token | Task agent | May not apply | Web Bot Auth | no token | Profile | | Google | Google-CloudVertexBot | Other | Yes | Reverse DNS | 10.9% | Profile | | Google | Google-Extended | Training | Yes | None published | 16.0% | Profile | | Google | Google-GeminiNotebook (was Google-NotebookLM) no token | Live answer fetch | May not apply | IP list | no token | Profile | | Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile | | Google | GoogleOther | Other | Yes | Reverse DNS | not measured | Profile | | Google | Storebot-Google | Search index | Yes | Reverse DNS | not measured | Profile | | Hive (ImageSift) | ImagesiftBot | Other | Yes | None published | 9.5% | Profile | | Huawei | PetalBot | Search index | Yes | Reverse DNS | 14.1% | Profile | | Manus | Manus cloud browser no token | Task agent | No docs | None published | no token | Profile | | Meta | FacebookBot | Other | Legacy | None published | 10.2% | Profile | | Meta | facebookexternalhit | Other | May not apply | None published | not measured | Profile | | Meta | meta-externalads | Other | Yes | None published | not measured | Profile | | Meta | meta-externalagent | Training | Yes | None published | 16.0% | Profile | | Meta | meta-externalfetcher | Live answer fetch | May not apply | None published | 8.5% | Profile | | Meta | meta-webindexer | Search index | Yes | None published | not measured | Profile | | Meta | Muse (personal agent) no token | Shopping agent | No docs | None published | no token | Profile | | Microsoft | bingbot | Search index | Yes | Reverse DNS | 3.1% | Profile | | Microsoft | Browse with Copilot (formerly Copilot Actions) no token | Task agent | No docs | None published | no token | Profile | | Microsoft | MicrosoftPreview | Other | Yes | Reverse DNS | 4.0% | Profile | | Mistral AI | MistralAI-Index | Search index | Yes | IP list | not measured | Profile | | Mistral AI | MistralAI-Training | Training | Yes | None published | not measured | Profile | | Mistral AI | MistralAI-User | Live answer fetch | Yes | IP list | 7.7% | Profile | | OpenAI | ChatGPT agent (Cloud browser, formerly Operator) no token | Task agent | No statement | Web Bot Auth | no token | Profile | | OpenAI | ChatGPT Atlas (browser agent mode) no token | Task agent | No docs | None published | no token | Profile | | OpenAI | ChatGPT-User | Live answer fetch | May not apply | IP list | 9.7% | Profile | | OpenAI | GPTBot | Training | Yes | IP list | 18.5% | Profile | | OpenAI | OAI-AdsBot | Other | No statement | IP list | not measured | Profile | | OpenAI | OAI-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Ora | OraBot | Other | No statement | Web Bot Auth | not measured | Profile | | Perplexity | Comet (browser assistant) no token | Task agent | No docs | None published | no token | Profile | | Perplexity | Perplexity Computer no token | Task agent | No docs | None published | no token | Profile | | Perplexity | Perplexity-User | Live answer fetch | May not apply | IP list | 8.3% | Profile | | Perplexity | PerplexityBot | Search index | Yes | IP list | 11.2% | Profile | | Timpi | Timpibot | Search index | No docs | None published | 9.8% | Profile | | Webz.io | omgili | Other | Yes | None published | 15.7% | Profile | | xAI | Grok Bot (agent) no token | Task agent | No docs | None published | no token | Profile | | You.com | YouBot | Search index | No docs | None published | 10.3% | Profile | ## Added on 2026-11-05 - [**The names sites write in robots.txt**](https://agentreadycrawlers.com/crawlers/named-in-robots-txt/) The user-agent names most often written in the parsed robots.txt files of the crawl, and which have no row in the Index. - [**Agentic browsers and how they identify**](https://agentreadycrawlers.com/crawlers/agentic-browsers/) Every task-agent row in the Index: token, user-agent string, verification and what the operator says about robots.txt. ## Sources - [Index method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "agent-readiness-crawl: user agent and token | Agent Ready Crawlers" description: "agent-readiness-crawl from Agent Ready Crawlers: exact user-agent string, robots.txt token agent-readiness-crawl, other purpose and how to verify it." canonical: https://agentreadycrawlers.com/crawlers/agent-readiness-crawl site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # agent-readiness-crawl: user agent and token | Agent Ready Crawlers > agent-readiness-crawl from Agent Ready Crawlers: exact user-agent string, robots.txt token agent-readiness-crawl, other purpose and how to verify it. Source: https://agentreadycrawlers.com/crawlers/agent-readiness-crawl Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Agent Ready Crawlers # agent-readiness-crawl (the crawl behind this site) In short agent-readiness-crawl (the crawl behind this site) is Agent Ready Crawlers' bot, filed under Other. The token to write in robots.txt is agent-readiness-crawl. The operator publishes nothing a site can check. 4.2 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Run on 2026-11-04 from a single machine. Operator Agent Ready Crawlers Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)` robots.txt token `agent-readiness-crawl` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Partly, because one pass skipped the check The main pass read /robots.txt first and fetched each of about 18 paths per domain only where the group for agent-readiness-crawl, or the \* group, allowed it. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains with the same user agent but without a per-path robots.txt check; the crawl now checks every path for future runs. [Source](https://agentreadycrawlers.com/method/#own-crawl), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://agentreadycrawlers.com/method/#own-crawl), read 2026-11-04. IP ranges Not published. Rate behaviour Main pass: about 18 plain GET requests per domain (robots.txt, the home page twice with one asking for text/markdown, one made-up missing path, llms.txt, llms-full.txt and 12 discovery paths), up to 100 domains at once, a 10 second timeout, no JavaScript, one request at a time per domain. Follow-up pass: ten more well-known paths on 10,508 reachable domains. Block rate **4.2%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `agent-readiness-crawl` at the root (247 sites). 0.0% name it; 4.1% refuse it only through a \* group. Rank 0 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 3.0% (0.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [agentreadycrawlers.com](https://agentreadycrawlers.com/method/#own-crawl), read 2026-11-04. Note. Run on 2026-11-04 from a single machine. The user-agent string carries no contact URL and the crawl publishes no IP list, so it fails its own verification test: a site owner cannot confirm a request is ours except by this documentation. The follow-up pass skipped the per-path robots.txt check that the main pass applied, which is now fixed in the crawl for future runs. It stores only extracted fields and the first 200 words of a home page. ## Allow and refuse lines ### To allow agent-readiness-crawl User-agent: agent-readiness-crawl Allow: / ### To refuse agent-readiness-crawl User-agent: agent-readiness-crawl Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Sources - [agentreadycrawlers.com](https://agentreadycrawlers.com/method/#own-crawl) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Amazon Bedrock AgentCore Browser profile | Agent Ready Crawlers" description: "Amazon Bedrock AgentCore Browser from Amazon Web Services: no robots.txt token, purpose and what a site can check. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/agentcore-browser site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Amazon Bedrock AgentCore Browser profile | Agent Ready Crawlers > Amazon Bedrock AgentCore Browser from Amazon Web Services: no robots.txt token, purpose and what a site can check. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/agentcore-browser Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Amazon Web Services # Amazon Bedrock AgentCore Browser In short Amazon Bedrock AgentCore Browser is Amazon Web Services' browsing agent. Amazon Bedrock AgentCore Browser has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. Amazon Bedrock AgentCore Browser: site owners keep control and may block, monitor or rate-limit agent traffic regardless of signatures; AWS states no robots.txt behaviour. Operator Amazon Web Services Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Signed with Web Bot Auth when enabled robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No statement AWS: site owners keep control and may block, monitor or rate-limit agent traffic regardless of signatures; AWS states no robots.txt behaviour. [Source](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html), read 2026-11-04. Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the developer running the agent. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [docs.aws.amazon.com](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html), read 2026-11-04. Note. Web Bot Auth support is in preview and off by default (browserSigning setting), per AWS docs and WorkOS. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Amazon Web Services in the Verification Ledger Signs on behalf of customers' agents when enabled. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory Web Bot Auth signing in preview, opt-in (browserSigning) Cloudflare listed by AWS as recognised by Cloudflare, Akamai and others Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [docs.aws.amazon.com: browser web bot auth](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Agentic browsers and how they identify | Agent Ready Crawlers" description: "Every task-agent and shopping-agent row in the Crawler Index: token, user-agent string, verification and what the operator says about robots.txt." canonical: https://agentreadycrawlers.com/crawlers/agentic-browsers site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Crawlers" --- # Agentic browsers and how they identify | Agent Ready Crawlers > Every task-agent and shopping-agent row in the Crawler Index: token, user-agent string, verification and what the operator says about robots.txt. Source: https://agentreadycrawlers.com/crawlers/agentic-browsers Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Agentic browsers and how they identify In short The Index holds 23 agent rows, 20 task agents and 3 shopping agents. 8 sign their requests, 1 names a robots.txt token and 6 look like the visitor's own browser. 15 publish nothing a site can check, so a log line cannot tell them from a person. A crawler arrives with a name. A browsing agent often arrives as a browser: the same window a person uses, driven by a model. The question for a site owner is what, if anything, in the request says an agent sent it. This table gives the answer row by row, from the operator's own pages. The difference between the two kinds of visitor is explained in [the guide on crawlers and browsing agents](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent/). ## How the rows identify themselves Out of 23 agent rows. The groups overlap: an agent can sign its requests and also appear as a browser. Only one row, Claude Duet cloud browser, publishes a robots.txt token, so a robots.txt group can address that agent and no others. For the rest, the lever is the WAF and, where the operator signs requests, the signature. ## Every agent row | Agent | Operator | Token | User agent, as published | Verification | robots.txt | | --- | --- | --- | --- | --- | --- | | ChatGPT agent | OpenAI | none | Requests carry Signature, Signature-Input and Signature-Agent: "https://chatgpt.com" headers (RFC 9421); the user agent string itself is not published as an identifier | Web Bot Auth signature | No statement | | ChatGPT Atlas | OpenAI | none | Ran in the user's own browser with a Chrome user agent; no agent token published | Nothing published | No docs | | Claude in Chrome | Anthropic | none | The user's own Chrome user agent; no agent token | Nothing published | No statement | | Claude Duet cloud browser | Anthropic | Claude-Agent | Linux Chrome string with a "compatible; Claude-Agent/1.0" token, as observed by Verisoul (secondary) | Nothing published | Reported | | Claude computer use | Anthropic | none | Whatever browser the developer runs; no Anthropic identifier | Nothing published | No docs | | Comet | Perplexity | none | Chrome's user agent from the user's own device; no unique identifier | Nothing published | No docs | | Perplexity Computer | Perplexity | none | No declared identity | Nothing published | No docs | | Google-Agent | Google | none | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36 | Web Bot Auth signature | May not apply | | Gemini in Chrome | Google | none | The user's own Chrome user agent; no marker in local mode | Nothing published | No docs | | Browse with Copilot | Microsoft | none | The user's own Edge user agent; no marker documented | Nothing published | No docs | | Buy for Me | Amazon | none | Not published | Nothing published | No docs | | Rufus | Amazon | none | Not applicable to third-party sites | Nothing published | No docs | | Amazon Bedrock AgentCore Browser | Amazon Web Services | none | Signed with Web Bot Auth when enabled | Web Bot Auth signature | No statement | | Muse | Meta | none | Does not identify itself, per Verisoul and Amazon's statement reported by GeekWire | Nothing published | No docs | | Browser Use | Browser Use | none | Chromium driven over the DevTools Protocol; Verisoul reports it names itself only in some setups | Nothing published | No docs | | Apify Website Content Crawler | Apify | none | Signed with Web Bot Auth when signHttpRequests is enabled | Web Bot Auth signature | Setting | | Browserbase cloud browsers | Browserbase | none | Signed with Web Bot Auth for customers who enable it | Web Bot Auth signature | No docs | | Stagehand | Browserbase | none | Inherits the browser it runs in; signed only when run on Browserbase with Web Bot Auth | Nothing published | No docs | | Cloudflare Browser Rendering | Cloudflare | none | Sends signed Web Bot Auth headers | Web Bot Auth signature | No statement | | Anchor Browser | Anchor | none | Can sign with Web Bot Auth depending on customer setup | Web Bot Auth signature | No docs | | Goose | Block | none | Signs with Web Bot Auth under Cloudflare's signed agents program | Cloudflare signed agent | No docs | | Grok Bot | xAI | none | No declared identity since mid-October 2026, per Verisoul | Nothing published | No docs | | Manus cloud browser | Manus | none | Signs cloud-browser page loads, per Verisoul | Nothing published | No docs | ## What a site can do about each group Signed agents 8 rows sign with Web Bot Auth or sit on Cloudflare's signed agent list. Verify the signature and write the rule on the verified identity. [How to verify Web Bot Auth signatures](https://agentreadycrawlers.com/guides/verify-web-bot-auth/). Agents in the visitor's browser 6 rows send the user's own Chrome or Edge user agent from the user's own address. A rule written on the string would refuse people too. Account and checkout pages need authentication, since robots.txt cannot address these agents. Everything else 9 rows publish nothing and do not look like a visitor's browser either. The page linked in each row says what the operator does and does not disclose. ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) read 2026-11-04 - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 - [Task agent rows (this site)](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Buy for Me: user agent, token, block rate | Agent Ready Crawlers" description: "Buy for Me from Amazon: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/amazon-buy-for-me site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Buy for Me: user agent, token, block rate | Agent Ready Crawlers > Buy for Me from Amazon: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/amazon-buy-for-me Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Shopping agent from Amazon # Buy for Me (shopping agent) In short Buy for Me (shopping agent) is Amazon's shopping agent. Buy for Me has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Buy for Me has no operator bot documentation, checked 2026-11-04. Buys from other brands' sites through the Amazon Shopping app. Operator [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/) Vendor profile: [Amazon](https://agentreadycrawlers.com/vendors/amazon/) Purpose Shopping agent [All shopping agent rows](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/). An agent that compares, selects or buys goods for a person. User-agent string No exact string is published. What the operator or a third party says: Not published robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). Amazon's Buy for Me announcement (2025-04-03) does not describe how the agent identifies itself to brand sites. [Source](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands). Note. Buys from other brands' sites through the Amazon Shopping app. GeekWire reports Amazon says Buy for Me identifies itself and lets brands opt out, but no identifier or opt-out mechanism is published. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Amazon in the Verification Ledger Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) - [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) - [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Amazon rows - [Amazonbot](https://agentreadycrawlers.com/crawlers/amazonbot/) Training token Amazonbot - [Amzn-SearchBot](https://agentreadycrawlers.com/crawlers/amzn-searchbot/) Search index token Amzn-SearchBot - [Amzn-User](https://agentreadycrawlers.com/crawlers/amzn-user/) Live answer fetch token Amzn-User - [Rufus (shopping assistant)](https://agentreadycrawlers.com/crawlers/amazon-rufus/) Shopping agent no token ## Sources - [aboutamazon.com: amazon shopping app buy for me brands](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [GeekWire: Amazon blocks Meta's Muse](https://www.geekwire.com/2026/amazon-blocks-metas-muse-ai-assistant-in-new-standoff-over-agentic-shopping/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Rufus: user agent, token, block rate | Agent Ready Crawlers" description: "Rufus from Amazon: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/amazon-rufus site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Rufus: user agent, token, block rate | Agent Ready Crawlers > Rufus from Amazon: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/amazon-rufus Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Shopping agent from Amazon # Rufus (shopping assistant) In short Rufus (shopping assistant) is Amazon's shopping agent. Rufus has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Kept as a row because merchants ask; any web fetch it triggers would surface as Amzn-User if it follows Amazon's documented fetcher. Operator [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/) Vendor profile: [Amazon](https://agentreadycrawlers.com/vendors/amazon/) Purpose Shopping agent [All shopping agent rows](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/). An agent that compares, selects or buys goods for a person. User-agent string No exact string is published. What the operator or a third party says: Not applicable to third-party sites robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). Rufus answers inside Amazon's store; no third-party fetch identity is documented. [Source](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands). Note. Kept as a row because merchants ask; any web fetch it triggers would surface as Amzn-User if it follows Amazon's documented fetcher. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Amazon in the Verification Ledger Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) - [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) - [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Amazon rows - [Amazonbot](https://agentreadycrawlers.com/crawlers/amazonbot/) Training token Amazonbot - [Amzn-SearchBot](https://agentreadycrawlers.com/crawlers/amzn-searchbot/) Search index token Amzn-SearchBot - [Amzn-User](https://agentreadycrawlers.com/crawlers/amzn-user/) Live answer fetch token Amzn-User - [Buy for Me (shopping agent)](https://agentreadycrawlers.com/crawlers/amazon-buy-for-me/) Shopping agent no token ## Sources - [aboutamazon.com: amazon shopping app buy for me brands](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Amazonbot: user agent, token, block rate | Agent Ready Crawlers" description: "Amazonbot from Amazon: exact user-agent string, robots.txt token Amazonbot, training purpose and how to verify it. 15.6% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/amazonbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Amazonbot: user agent, token, block rate | Agent Ready Crawlers > Amazonbot from Amazon: exact user-agent string, robots.txt token Amazonbot, training purpose and how to verify it. 15.6% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/amazonbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Amazon # Amazonbot In short Amazonbot is Amazon's training crawler. The token to write in robots.txt is Amazonbot. Check a visitor against the operator's published IP list. 15.6 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Amazonbot: automated crawling respects the Robots Exclusion Protocol, honouring user-agent and allow/disallow directives; robots.txt may be cached for 30 days. Operator [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/) Vendor profile: [Amazon](https://agentreadycrawlers.com/vendors/amazon/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `Amazonbot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Amazon: automated crawling respects the Robots Exclusion Protocol, honouring user-agent and allow/disallow directives; robots.txt may be cached for 30 days. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://developer.amazon.com/amazonbot/ip-addresses/), read 2026-11-04. IP ranges [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) Rate behaviour Does not support Crawl-delay. Honours rel=nofollow and robots meta noarchive, noindex and none. Block rate **15.6%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Amazonbot` at the root (926 sites). 12.0% name it; 3.6% refuse it only through a \* group. Rank 9 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.7% (9.1% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developer.amazon.com](https://developer.amazon.com/amazonbot), read 2026-11-04. Note. Amazon says Amazonbot improves products and services and may be used to train Amazon AI models. Changes take about 24 hours. ## Allow and refuse lines ### To allow Amazonbot User-agent: Amazonbot Allow: / ### To refuse Amazonbot User-agent: Amazonbot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Amazon in the Verification Ledger Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) - [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) - [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Amazon rows - [Amzn-SearchBot](https://agentreadycrawlers.com/crawlers/amzn-searchbot/) Search index token Amzn-SearchBot - [Amzn-User](https://agentreadycrawlers.com/crawlers/amzn-user/) Live answer fetch token Amzn-User - [Buy for Me (shopping agent)](https://agentreadycrawlers.com/crawlers/amazon-buy-for-me/) Shopping agent no token - [Rufus (shopping assistant)](https://agentreadycrawlers.com/crawlers/amazon-rufus/) Shopping agent no token ## Sources - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [developer.amazon.com: ip addresses](https://developer.amazon.com/amazonbot/ip-addresses/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Amzn-SearchBot: user agent, token, block rate | Agent Ready Crawlers" description: "Amzn-SearchBot from Amazon: exact user-agent string, robots.txt token Amzn-SearchBot, search index purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/amzn-searchbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Amzn-SearchBot: user agent, token, block rate | Agent Ready Crawlers > Amzn-SearchBot from Amazon: exact user-agent string, robots.txt token Amzn-SearchBot, search index purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/amzn-searchbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Amazon # Amzn-SearchBot In short Amzn-SearchBot is Amazon's search index crawler. Check a visitor against the operator's published IP list. The crawl of 2026-11-04 did not measure Amzn-SearchBot. Eligibility for Alexa and other Amazon search experiences. Amzn-SearchBot: if robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the directives given to other search bots. Operator [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/) Vendor profile: [Amazon](https://agentreadycrawlers.com/vendors/amazon/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `Amzn-SearchBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Amazon: if robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the directives given to other search bots. Not used for training. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/), read 2026-11-04. IP ranges [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) Rate behaviour Does not support Crawl-delay. Block rate Not measured: the crawl checked 31 AI and search tokens, and `Amzn-SearchBot` is not one of them. Documentation [developer.amazon.com](https://developer.amazon.com/amazonbot), read 2026-11-04. Note. Eligibility for Alexa and other Amazon search experiences. ## Allow and refuse lines ### To allow Amzn-SearchBot User-agent: Amzn-SearchBot Allow: / ### To refuse Amzn-SearchBot User-agent: Amzn-SearchBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Amazon in the Verification Ledger Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) - [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) - [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Amazon rows - [Amazonbot](https://agentreadycrawlers.com/crawlers/amazonbot/) Training token Amazonbot - [Amzn-User](https://agentreadycrawlers.com/crawlers/amzn-user/) Live answer fetch token Amzn-User - [Buy for Me (shopping agent)](https://agentreadycrawlers.com/crawlers/amazon-buy-for-me/) Shopping agent no token - [Rufus (shopping assistant)](https://agentreadycrawlers.com/crawlers/amazon-rufus/) Shopping agent no token ## Sources - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [developer.amazon.com: searchbot ip addresses](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Amzn-User: user agent, token, block rate | Agent Ready Crawlers" description: "Amzn-User from Amazon: exact user-agent string, robots.txt token Amzn-User, live answer fetch purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/amzn-user site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Amzn-User: user agent, token, block rate | Agent Ready Crawlers > Amzn-User from Amazon: exact user-agent string, robots.txt token Amzn-User, live answer fetch purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/amzn-user Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from Amazon # Amzn-User In short Amzn-User is Amazon's live answer fetcher. The token to write in robots.txt is Amzn-User. Check a visitor against the operator's published IP list. The crawl of 2026-11-04 did not measure Amzn-User. Fetches live information for Alexa answers. Amzn-User: because actions can be initiated by a user, Amzn-User may not follow all robots.txt directives. Operator [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/) Vendor profile: [Amazon](https://agentreadycrawlers.com/vendors/amazon/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `Amzn-User` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt May not apply (user-triggered) Amazon: because actions can be initiated by a user, Amzn-User may not follow all robots.txt directives. Not used for training. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://developer.amazon.com/amazonbot/live-ip-addresses/), read 2026-11-04. IP ranges [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Rate behaviour User-triggered. Block rate Not measured: the crawl checked 31 AI and search tokens, and `Amzn-User` is not one of them. Documentation [developer.amazon.com](https://developer.amazon.com/amazonbot), read 2026-11-04. Note. Fetches live information for Alexa answers. ## Allow and refuse lines ### To allow Amzn-User User-agent: Amzn-User Allow: / ### To refuse Amzn-User User-agent: Amzn-User Disallow: / Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Amazon in the Verification Ledger Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) - [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) - [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Amazon rows - [Amazonbot](https://agentreadycrawlers.com/crawlers/amazonbot/) Training token Amazonbot - [Amzn-SearchBot](https://agentreadycrawlers.com/crawlers/amzn-searchbot/) Search index token Amzn-SearchBot - [Buy for Me (shopping agent)](https://agentreadycrawlers.com/crawlers/amazon-buy-for-me/) Shopping agent no token - [Rufus (shopping assistant)](https://agentreadycrawlers.com/crawlers/amazon-rufus/) Shopping agent no token ## Sources - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [developer.amazon.com: live ip addresses](https://developer.amazon.com/amazonbot/live-ip-addresses/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Anchor Browser: user agent, token, block rate | Agent Ready Crawlers" description: "Anchor Browser from Anchor: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/anchor-browser site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Anchor Browser: user agent, token, block rate | Agent Ready Crawlers > Anchor Browser from Anchor: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/anchor-browser Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Anchor # Anchor Browser In short Anchor Browser is Anchor's browsing agent. Anchor Browser has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. Anchor Browser has no operator bot documentation, checked 2026-11-04. Verisoul reports sessions can also be configured to pass as human. Operator Anchor Vendor profile: [Anchor Browser](https://agentreadycrawlers.com/vendors/anchor/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Can sign with Web Bot Auth depending on customer setup robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) on robots.txt. [Source](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the customer. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/anchor-browser). Note. In Cloudflare's first signed-agents cohort. Verisoul reports sessions can also be configured to pass as human. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Anchor in the Verification Ledger Signing depends on customer configuration. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory signs per customer setup Cloudflare Cloudflare signed agent (cohort of 2025-08-28) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [verisoul.ai: anchor browser](https://www.verisoul.ai/agents/anchor-browser) read 2026-11-04 - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "anthropic-ai: user agent, token, block rate | Agent Ready Crawlers" description: "anthropic-ai from Anthropic: robots.txt token anthropic-ai, training purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/anthropic-ai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # anthropic-ai: user agent, token, block rate | Agent Ready Crawlers > anthropic-ai from Anthropic: robots.txt token anthropic-ai, training purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/anthropic-ai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Anthropic # anthropic-ai (legacy token) In short anthropic-ai (legacy token) is Anthropic's training crawler. The token to write in robots.txt is anthropic-ai. The operator publishes nothing a site can check. 15.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, not listed on Anthropic's current crawler page, which names only ClaudeBot, Claude-User and Claude-SearchBot (checked 2026-11-04). Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string No exact string is published. What the operator or a third party says: No current request string; legacy robots.txt token only robots.txt token `anthropic-ai` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Not on current operator page Not listed on Anthropic's current crawler page, which names only ClaudeBot, Claude-User and Claude-SearchBot (checked 2026-11-04). [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. IP ranges Not published. Rate behaviour Not documented. Block rate **15.5%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `anthropic-ai` at the root (918 sites). 11.7% name it; 3.8% refuse it only through a \* group. Rank 10 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.6% (8.9% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Note. Still common in published robots.txt files; the shared crawl tests it. A Disallow for anthropic-ai alone does not cover ClaudeBot. ## Allow and refuse lines ### To allow anthropic-ai User-agent: anthropic-ai Allow: / ### To refuse anthropic-ai User-agent: anthropic-ai Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Anthropic rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Apify Website Content Crawler profile | Agent Ready Crawlers" description: "Apify Website Content Crawler from Apify: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/apify site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Apify Website Content Crawler profile | Agent Ready Crawlers > Apify Website Content Crawler from Apify: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/apify Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Apify # Apify Website Content Crawler In short Apify Website Content Crawler is Apify's browsing agent. Apify Website Content Crawler has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. On robots.txt, respectRobotsTxtFile is an input option; the Console prefills it, the API does not. Operator Apify Vendor profile: [Apify](https://agentreadycrawlers.com/vendors/apify/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Signed with Web Bot Auth when signHttpRequests is enabled robots.txt token None. No robots.txt line can address this agent. Respects robots.txt Depends on a setting Apify: respectRobotsTxtFile is an input option; the Console prefills it, the API does not. [Source](https://blog.apify.com/ai-agent-infrastructure/), read 2026-11-04. Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://blog.apify.com/ai-agent-infrastructure/), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the Actor run. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [blog.apify.com](https://blog.apify.com/ai-agent-infrastructure/), read 2026-11-04. Note. No key directory at apify.com/.well-known (404, 2026-11-04); verification relies on the signature-agent named in each signed request. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Apify in the Verification Ledger Signing option documented; directory location not found. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory signHttpRequests option; no key directory at apify.com (404, 2026-11-04) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [blog.apify.com: ai agent infrastructure](https://blog.apify.com/ai-agent-infrastructure/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Applebot: user agent, token, block rate | Agent Ready Crawlers" description: "Applebot from Apple: exact user-agent string, robots.txt token Applebot, search index purpose and how to verify it. 14.8% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/applebot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Applebot: user agent, token, block rate | Agent Ready Crawlers > Applebot from Apple: exact user-agent string, robots.txt token Applebot, search index purpose and how to verify it. 14.8% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/applebot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Apple # Applebot In short Applebot is Apple's search index crawler. The token to write in robots.txt is Applebot. Verify it by reverse DNS, then a forward lookup. Mixed-use crawler; Cloudflare classes it Accountable (2026-09-15). Applebot: enabling Applebot in robots.txt allows content to appear in Apple search results; data may also train Apple foundation models unless Applebot-Extended is disallowed. Operator [Apple](https://agentreadycrawlers.com/crawlers/operator/apple/) Vendor profile: [Apple](https://agentreadycrawlers.com/vendors/apple/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)` robots.txt token `Applebot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Apple: enabling Applebot in robots.txt allows content to appear in Apple search results; data may also train Apple foundation models unless Applebot-Extended is disallowed. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://support.apple.com/en-us/119829), read 2026-11-04. IP ranges [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json) Rate behaviour Not published. Block rate **14.8%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Applebot` at the root (877 sites). 11.6% name it; 3.1% refuse it only through a \* group. Rank 12 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.1% (8.8% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.apple.com](https://support.apple.com/en-us/119829), read 2026-11-04. Note. Reverse DNS in \*.applebot.apple.com. Mixed-use crawler; Cloudflare classes it Accountable (2026-09-15). Apple says nosnippet opts content out of broad world knowledge answers. ## Allow and refuse lines ### To allow Applebot User-agent: Applebot Allow: / ### To refuse Applebot User-agent: Applebot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Apple in the Verification Ledger IP list and reverse DNS. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json) Reverse DNS \*.applebot.apple.com Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) ## Other Apple rows - [Applebot-Extended (control token)](https://agentreadycrawlers.com/crawlers/applebot-extended/) Training token Applebot-Extended ## Sources - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [search.developer.apple.com: applebot (JSON)](https://search.developer.apple.com/applebot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Applebot-Extended: user agent and token | Agent Ready Crawlers" description: "Applebot-Extended from Apple: robots.txt token Applebot-Extended, training purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/applebot-extended site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Applebot-Extended: user agent and token | Agent Ready Crawlers > Applebot-Extended from Apple: robots.txt token Applebot-Extended, training purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/applebot-extended Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Apple # Applebot-Extended (control token) In short Applebot-Extended (control token) is Apple's training crawler. The token to write in robots.txt is Applebot-Extended. The operator publishes nothing a site can check. 14.8 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disallowing Applebot-Extended opts content out of training generative foundation models; pages can still appear in search. Operator [Apple](https://agentreadycrawlers.com/crawlers/operator/apple/) Vendor profile: [Apple](https://agentreadycrawlers.com/vendors/apple/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string No exact string is published. What the operator or a third party says: None: Applebot-Extended does not crawl webpages robots.txt token `Applebot-Extended` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Apple: disallowing Applebot-Extended opts content out of training generative foundation models; pages can still appear in search. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://support.apple.com/en-us/119829), read 2026-11-04. IP ranges Not published. Rate behaviour No crawl of its own. Block rate **14.8%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Applebot-Extended` at the root (877 sites). 11.6% name it; 3.1% refuse it only through a \* group. Rank 13 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.1% (8.8% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.apple.com](https://support.apple.com/en-us/119829), read 2026-11-04. Note. Apple says it only determines how data crawled by Applebot is used. ## Allow and refuse lines ### To allow Applebot-Extended User-agent: Applebot-Extended Allow: / ### To refuse Applebot-Extended User-agent: Applebot-Extended Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Apple in the Verification Ledger IP list and reverse DNS. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json) Reverse DNS \*.applebot.apple.com Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Apple rows - [Applebot](https://agentreadycrawlers.com/crawlers/applebot/) Search index token Applebot ## Sources - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Bingbot: user agent, token, block rate | Agent Ready Crawlers" description: "Bingbot from Microsoft: exact user-agent string, robots.txt token bingbot, search index purpose and how to verify it. 3.1% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/bingbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Bingbot: user agent, token, block rate | Agent Ready Crawlers > Bingbot from Microsoft: exact user-agent string, robots.txt token bingbot, search index purpose and how to verify it. 3.1% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/bingbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Microsoft # Bingbot In short Bingbot is Microsoft's search index crawler. The token to write in robots.txt is bingbot. Verify it by reverse DNS, then a forward lookup. Hourly crawl control in Bing Webmaster Tools. Bingbot: robots.txt can be configured to tell Bing crawlers how to interact with a site; crawl rate is set by hour in Bing Webmaster Tools. Operator [Microsoft](https://agentreadycrawlers.com/crawlers/operator/microsoft/) Vendor profile: [Microsoft](https://agentreadycrawlers.com/vendors/microsoft/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `bingbot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Microsoft: robots.txt can be configured to tell Bing crawlers how to interact with a site; crawl rate is set by hour in Bing Webmaster Tools. [Source](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26), read 2026-11-04. IP ranges [bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json) Rate behaviour Hourly crawl control in Bing Webmaster Tools. Block rate **3.1%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Bingbot` at the root (182 sites). 0.4% name it; 2.7% refuse it only through a \* group. Rank 30 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 2.2% (0.2% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [bing.com](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), read 2026-11-04. Note. Verify Bingbot tool at bing.com/webmasters/verifybingbot. Training opt-out today is the NOARCHIVE meta tag; a robots.txt no-training preference is targeted for early 2027, per Cloudflare's 2026-09-15 post. bingbot.json creationTime 2024-01-03. ## Allow and refuse lines ### To allow bingbot User-agent: bingbot Allow: / ### To refuse bingbot User-agent: bingbot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Microsoft in the Verification Ledger bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json) Reverse DNS Verify Bingbot tool and documented methods Key directory not published Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) ## Other Microsoft rows - [MicrosoftPreview](https://agentreadycrawlers.com/crawlers/microsoftpreview/) Other token MicrosoftPreview - [Browse with Copilot (formerly Copilot Actions)](https://agentreadycrawlers.com/crawlers/copilot-actions/) Task agent no token ## Sources - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [bing.com: how to verify bingbot 3905dc26](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) read 2026-11-04 - [bing.com: bingbot (JSON)](https://www.bing.com/toolbox/bingbot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Browser Use: user agent, token, block rate | Agent Ready Crawlers" description: "Browser Use from Browser Use: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/browser-use site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Browser Use: user agent, token, block rate | Agent Ready Crawlers > Browser Use from Browser Use: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/browser-use Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Browser Use # Browser Use (library and cloud) In short Browser Use (library and cloud) is Browser Use's browsing agent. Browser Use has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Browser Use says its cloud created 3.7M sessions in the seven days to 2026-10-14 (vendor figure via Verisoul). Operator Browser Use Vendor profile: [Browser Use](https://agentreadycrawlers.com/vendors/browser-use/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Chromium driven over the DevTools Protocol; Verisoul reports it names itself only in some setups robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) for site owners; the library itself does not apply robots.txt. [Source](https://www.verisoul.ai/agents/browser-use), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/browser-use), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the developer. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/browser-use). Note. MIT-licensed. Browser Use says its cloud created 3.7M sessions in the seven days to 2026-10-14 (vendor figure via Verisoul). No key directory at browser-use.com (404, 2026-11-04). ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Browser Use in the Verification Ledger Publishes nothing for site owners (no key directory at browser-use.com, 404, 2026-11-04). [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [verisoul.ai: browser use](https://www.verisoul.ai/agents/browser-use) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Browserbase cloud browsers profile | Agent Ready Crawlers" description: "Browserbase cloud browsers from Browserbase: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/browserbase site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Browserbase cloud browsers profile | Agent Ready Crawlers > Browserbase cloud browsers from Browserbase: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/browserbase Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Browserbase # Browserbase cloud browsers In short Browserbase cloud browsers is Browserbase's browsing agent. Browserbase cloud browsers has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. Browserbase cloud browsers has no operator bot documentation, checked 2026-11-04. Part of Cloudflare's first signed-agents cohort (2025-08-28). Operator [Browserbase](https://agentreadycrawlers.com/crawlers/operator/browserbase/) Vendor profile: [Browserbase](https://agentreadycrawlers.com/vendors/browserbase/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Signed with Web Bot Auth for customers who enable it robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) on robots.txt; Browserbase sessions follow the customer's code. [Source](https://docs.browserbase.com/platform/identity/overview), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://www.browserbase.com/.well-known/http-message-signatures-directory), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the customer. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://docs.browserbase.com/platform/identity/overview). Note. Part of Cloudflare's first signed-agents cohort (2025-08-28). Key directory served 2026-11-04. Browserbase also sells a "Verified" browser with real fingerprints, so unsigned sessions are hard to tell apart. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Browserbase in the Verification Ledger Key directory; signing is a customer feature. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory https://www.browserbase.com/.well-known/http-message-signatures-directory Cloudflare Cloudflare signed agent (cohort of 2025-08-28) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Browserbase rows - [Stagehand (open-source framework)](https://agentreadycrawlers.com/crawlers/stagehand/) Task agent no token ## Sources - [docs.browserbase.com: overview](https://docs.browserbase.com/platform/identity/overview) read 2026-11-04 - [browserbase.com: http message signatures directory](https://www.browserbase.com/.well-known/http-message-signatures-directory) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Bytespider: user agent, token, block rate | Agent Ready Crawlers" description: "Bytespider from ByteDance: robots.txt token Bytespider, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/bytespider site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Bytespider: user agent, token, block rate | Agent Ready Crawlers > Bytespider from ByteDance: robots.txt token Bytespider, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/bytespider Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from ByteDance # Bytespider In short Bytespider is ByteDance's bot, filed under Other. The token to write in robots.txt is Bytespider. The operator publishes nothing a site can check. 19.2 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Verisoul cites Cloudflare Radar at about 3 percent of global crawler traffic in September 2026. Operator ByteDance Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string No exact string is published. What the operator or a third party says: Contains "Bytespider"; ByteDance publishes no string robots.txt token `Bytespider` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). No ByteDance page documents its purpose, IP ranges or robots.txt handling. [Source](https://www.centinelanalytica.com/bots/bytespider), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.centinelanalytica.com/bots/bytespider), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **19.2%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Bytespider` at the root (1,141 sites). 15.4% name it; 3.8% refuse it only through a \* group. Rank 2 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 15.0% (12.3% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.centinelanalytica.com/bots/bytespider). Note. Verisoul cites Cloudflare Radar at about 3 percent of global crawler traffic in September 2026. Public reports of it ignoring robots.txt exist but are not operator statements. ## Allow and refuse lines ### To allow Bytespider User-agent: Bytespider Allow: / ### To refuse Bytespider User-agent: Bytespider Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## ByteDance in the Verification Ledger Publishes nothing. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not found on Radar (slug bytespider returned 404, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [centinelanalytica.com: bytespider](https://www.centinelanalytica.com/bots/bytespider) read 2026-11-04 - [Verisoul: Bytespider (cites Cloudflare Radar)](https://www.verisoul.ai/agents/bytespider) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "CCBot: user agent, token, block rate | Agent Ready Crawlers" description: "CCBot from Common Crawl: exact user-agent string, robots.txt token CCBot, training purpose and how to verify it. 19.4% of top sites block it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/ccbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # CCBot: user agent, token, block rate | Agent Ready Crawlers > CCBot from Common Crawl: exact user-agent string, robots.txt token CCBot, training purpose and how to verify it. 19.4% of top sites block it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/ccbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Common Crawl # CCBot In short CCBot is Common Crawl's training crawler. The token to write in robots.txt is CCBot. Verify it by reverse DNS, then a forward lookup. 19.4 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: CCBot/2.0 (https://commoncrawl.org/faq/). On robots.txt, Common Crawl documents "User-agent: CCBot" then "Disallow: /" to prevent crawling. Operator Common Crawl Vendor profile: [Common Crawl](https://agentreadycrawlers.com/vendors/common-crawl/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string `CCBot/2.0 (https://commoncrawl.org/faq/)` robots.txt token `CCBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Common Crawl documents "User-agent: CCBot" then "Disallow: /" to prevent crawling. [Source](https://commoncrawl.org/ccbot), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://commoncrawl.org/ccbot), read 2026-11-04. IP ranges [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json) Rate behaviour Not published on the CCBot page. Block rate **19.4%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `CCBot` at the root (1,151 sites). 15.7% name it; 3.7% refuse it only through a \* group. Rank 1 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 14.9% (12.2% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [commoncrawl.org](https://commoncrawl.org/ccbot), read 2026-11-04. Note. Dedicated IP ranges with reverse DNS \*.crawl.commoncrawl.org (IPv4 only). Common Crawl warns of crawlers falsely identifying as CCBot. ccbot.json creationTime 2026-08-11. Its open dataset is widely used to train models. ## Allow and refuse lines ### To allow CCBot User-agent: CCBot Allow: / ### To refuse CCBot User-agent: CCBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Common Crawl in the Verification Ledger IP list with synctoken 20260811134000 and reverse DNS. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json) Reverse DNS \*.crawl.commoncrawl.org (IPv4 only) Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04 - [index.commoncrawl.org: ccbot (JSON)](https://index.commoncrawl.org/ccbot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ChatGPT agent: user agent, token, block rate | Agent Ready Crawlers" description: "ChatGPT agent from OpenAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/chatgpt-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ChatGPT agent: user agent, token, block rate | Agent Ready Crawlers > ChatGPT agent from OpenAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/chatgpt-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from OpenAI # ChatGPT agent (Cloud browser, formerly Operator) In short ChatGPT agent (Cloud browser, formerly Operator) is OpenAI's browsing agent. ChatGPT agent has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. OpenAI's allowlisting article covers signature verification and allowlisting only; it states no robots.txt behaviour for the Cloud browser. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Requests carry Signature, Signature-Input and Signature-Agent: "https://chatgpt.com" headers (RFC 9421); the user agent string itself is not published as an identifier robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No statement OpenAI's allowlisting article covers signature verification and allowlisting only; it states no robots.txt behaviour for the Cloud browser. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://chatgpt.com/.well-known/http-message-signatures-directory), read 2026-11-04. IP ranges Not published. Rate behaviour OpenAI says at launch the Cloud browser cannot sign in to websites or complete payments. Vendor rule names - **Cloudflare:** Signed agent, bot tag chatgpt-agent, Bot Detection ID 129220581 (listed as ChatGPT Operator in the WAF search) - **Vercel:** ChatGPT Agent entry, id chatgpt-operator, verified automatically - **Akamai:** ChatGPT Agent, Artificial Intelligence bots category - **HUMAN Security:** ChatGPT Agent in Known Bots & Crawlers; AI Agents Permissions in AgenticTrust Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [help.openai.com](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. Note. OpenAI's help article was renamed "ChatGPT Work's Cloud browser allowlisting" (updated about 15 days before 2026-11-04). Key directory served 2026-11-04 with one Ed25519 key. Part of Cloudflare's first signed-agents cohort (2025-08-28). Radar lists "ChatGPT agent" as Verified. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) Training token GPTBot - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Search index token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [chatgpt.com: http message signatures directory](https://chatgpt.com/.well-known/http-message-signatures-directory) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Cloudflare Radar: GPTBot](https://radar.cloudflare.com/bots/directory/gptbot) read 2026-11-04 - [Cloudflare: signed agents](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ChatGPT Atlas: user agent, token, block rate | Agent Ready Crawlers" description: "ChatGPT Atlas from OpenAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/chatgpt-atlas site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ChatGPT Atlas: user agent, token, block rate | Agent Ready Crawlers > ChatGPT Atlas from OpenAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/chatgpt-atlas Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from OpenAI # ChatGPT Atlas (browser agent mode) In short ChatGPT Atlas (browser agent mode) is OpenAI's browsing agent. ChatGPT Atlas has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. ChatGPT Atlas has no operator bot documentation, checked 2026-11-04. Secondary source only (Verisoul Agent Atlas). Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Ran in the user's own browser with a Chrome user agent; no agent token published robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) for Atlas agent mode; Verisoul reports OpenAI retired Atlas on 2026-08-09. [Source](https://www.verisoul.ai/agents/chatgpt-atlas), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/chatgpt-atlas), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/chatgpt-atlas). Note. Secondary source only (Verisoul Agent Atlas). Kept as a row so log readers can place historic traffic; the signed Cloud browser row is the live OpenAI agent. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) Training token GPTBot - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Search index token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token ## Sources - [verisoul.ai: chatgpt atlas](https://www.verisoul.ai/agents/chatgpt-atlas) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ChatGPT-User: user agent, token, block rate | Agent Ready Crawlers" description: "ChatGPT-User from OpenAI: exact user-agent string, robots.txt token ChatGPT-User, live answer fetch purpose and how to verify it. 9.7% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/chatgpt-user site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ChatGPT-User: user agent, token, block rate | Agent Ready Crawlers > ChatGPT-User from OpenAI: exact user-agent string, robots.txt token ChatGPT-User, live answer fetch purpose and how to verify it. 9.7% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/chatgpt-user Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from OpenAI # ChatGPT-User In short ChatGPT-User is OpenAI's live answer fetcher. The token to write in robots.txt is ChatGPT-User. Check a visitor against the operator's published IP list. 9.7 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). ChatGPT-User: because these actions are initiated by a user, robots.txt rules may not apply. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot` robots.txt token `ChatGPT-User` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt May not apply (user-triggered) OpenAI: because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to decide search inclusion. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://openai.com/chatgpt-user.json), read 2026-11-04. IP ranges [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) Rate behaviour Not used for automatic crawling, per OpenAI. chatgpt-user.json creationTime read: 2026-10-27. Block rate **9.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `ChatGPT-User` at the root (574 sites). 6.3% name it; 3.3% refuse it only through a \* group. Rank 20 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 6.9% (4.5% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [platform.openai.com](https://platform.openai.com/docs/bots), read 2026-11-04. Note. Also used for GPT Actions and Custom GPT fetches. Use OAI-SearchBot, not ChatGPT-User, to manage search opt-outs. ## Allow and refuse lines ### To allow ChatGPT-User User-agent: ChatGPT-User Allow: / ### To refuse ChatGPT-User User-agent: ChatGPT-User Disallow: / Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-27[OpenAI updates the ChatGPT-User IP list](https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list/) ## Other OpenAI rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) Training token GPTBot - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Search index token OAI-SearchBot - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: chatgpt user (JSON)](https://openai.com/chatgpt-user.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Claude computer use: user agent and token | Agent Ready Crawlers" description: "Claude computer use from Anthropic: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/claude-computer-use site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Claude computer use: user agent and token | Agent Ready Crawlers > Claude computer use from Anthropic: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/claude-computer-use Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Anthropic # Claude computer use (API tool) In short Claude computer use (API tool) is Anthropic's browsing agent. Claude computer use has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Claude computer use has no operator bot documentation, checked 2026-11-04. Hosted browsers it runs inside (Browserbase, AgentCore, Cloudflare Browser Rendering) may sign requests. Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Whatever browser the developer runs; no Anthropic identifier robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). The computer use tool runs in the developer's own environment, so requests come from the developer's infrastructure. [Source](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the developer. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool). Note. Treat as the developer's traffic. Hosted browsers it runs inside (Browserbase, AgentCore, Cloudflare Browser Rendering) may sign requests. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Anthropic rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent ## Sources - [docs.anthropic.com: computer use tool](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Claude Duet cloud browser profile | Agent Ready Crawlers" description: "Claude Duet cloud browser from Anthropic: robots.txt token Claude-Agent, purpose and what a site can check. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/claude-duet site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Claude Duet cloud browser profile | Agent Ready Crawlers > Claude Duet cloud browser from Anthropic: robots.txt token Claude-Agent, purpose and what a site can check. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/claude-duet Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Anthropic # Claude Duet cloud browser In short Claude Duet cloud browser is Anthropic's browsing agent. The token to write in robots.txt is Claude-Agent. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure Claude-Agent. On robots.txt, Verisoul reports Anthropic says the agent aims to follow site rules; Anthropic's crawler page does not list it (checked 2026-11-04). Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Linux Chrome string with a "compatible; Claude-Agent/1.0" token, as observed by Verisoul (secondary) robots.txt token `Claude-Agent` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Reported by a third party Verisoul reports Anthropic says the agent aims to follow site rules; Anthropic's crawler page does not list it (checked 2026-11-04). [Source](https://www.verisoul.ai/agents/claude-duet), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/claude-duet), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `Claude-Agent` is not one of them. Documentation [verisoul.ai](https://www.verisoul.ai/agents/claude-duet), read 2026-11-04. Note. Duet launched 2026-10-21 as a waitlist research preview for US Pro and Max subscribers, per Verisoul. Token taken from observed traffic, not an Anthropic page; treat as unconfirmed. ## Allow and refuse lines ### To allow Claude-Agent User-agent: Claude-Agent Allow: / ### To refuse Claude-Agent User-agent: Claude-Agent Disallow: / Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-21[Anthropic's Claude Duet starts browsing with a Claude-Agent token](https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent/) ## Other Anthropic rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [verisoul.ai: claude duet](https://www.verisoul.ai/agents/claude-duet) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Claude in Chrome: user agent and token | Agent Ready Crawlers" description: "Claude in Chrome from Anthropic: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/claude-in-chrome site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Claude in Chrome: user agent and token | Agent Ready Crawlers > Claude in Chrome from Anthropic: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/claude-in-chrome Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Anthropic # Claude in Chrome In short Claude in Chrome is Anthropic's browsing agent. Claude in Chrome has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Browser extension; Anthropic documents user-side controls only. No robots.txt behaviour or site-side opt-out is documented on the Anthropic pages read (checked 2026-11-04). Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: The user's own Chrome user agent; no agent token robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No statement No robots.txt behaviour or site-side opt-out is documented on the Anthropic pages read (checked 2026-11-04). [Source](https://claude.com/docs/third-party/claude-desktop/browser), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome), read 2026-11-04. IP ranges Not published. Rate behaviour Runs in the user's browser with the user's tabs, sign-ins and IP address. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [claude.com](https://claude.com/docs/third-party/claude-desktop/browser), read 2026-11-04. Note. Browser extension; Anthropic documents user-side controls only. Third-party detection vendors flag it by behaviour, not identity (https://www.centinelanalytica.com/blog/ai-browser-agents). ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Anthropic rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [claude.com: browser](https://claude.com/docs/third-party/claude-desktop/browser) read 2026-11-04 - [support.claude.com: 12012173 getting started with claude in chrome](https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Claude-SearchBot: user agent and token | Agent Ready Crawlers" description: "Claude-SearchBot from Anthropic: robots.txt token Claude-SearchBot, search index purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/claude-searchbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Claude-SearchBot: user agent and token | Agent Ready Crawlers > Claude-SearchBot from Anthropic: robots.txt token Claude-SearchBot, search index purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/claude-searchbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Anthropic # Claude-SearchBot In short Claude-SearchBot is Anthropic's search index crawler. The token to write in robots.txt is Claude-SearchBot. Check a visitor against the operator's published IP list. 7.7 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string No exact string is published. What the operator or a third party says: Contains "Claude-SearchBot"; no full string published robots.txt token `Claude-SearchBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://claude.com/crawling/bots.json), read 2026-11-04. IP ranges [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Rate behaviour Respects Crawl-delay where appropriate, per Anthropic. Block rate **7.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Claude-SearchBot` at the root (460 sites). 4.1% name it; 3.6% refuse it only through a \* group. Rank 27 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.5% (2.9% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Note. Search-index crawler, separate from training. ## Allow and refuse lines ### To allow Claude-SearchBot User-agent: Claude-SearchBot Allow: / ### To refuse Claude-SearchBot User-agent: Claude-SearchBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) ## Other Anthropic rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Claude-User: user agent, token, block rate | Agent Ready Crawlers" description: "Claude-User from Anthropic: robots.txt token Claude-User, live answer fetch purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/claude-user site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Claude-User: user agent, token, block rate | Agent Ready Crawlers > Claude-User from Anthropic: robots.txt token Claude-User, live answer fetch purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/claude-user Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from Anthropic # Claude-User In short Claude-User is Anthropic's live answer fetcher. The token to write in robots.txt is Claude-User. Check a visitor against the operator's published IP list. Unlike OpenAI and Perplexity, Anthropic says its user-triggered fetcher honours robots.txt. On robots.txt, Anthropic: Claude-User allows site owners to control which sites can be accessed through user-initiated requests; disabling it stops retrieval for user queries. Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string No exact string is published. What the operator or a third party says: Contains "Claude-User"; no full string published robots.txt token `Claude-User` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Anthropic: Claude-User allows site owners to control which sites can be accessed through user-initiated requests; disabling it stops retrieval for user queries. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://claude.com/crawling/bots.json), read 2026-11-04. IP ranges [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Rate behaviour Respects Crawl-delay where appropriate, per Anthropic. Block rate **8.1%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Claude-User` at the root (479 sites). 4.5% name it; 3.6% refuse it only through a \* group. Rank 24 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.7% (3.1% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Note. Unlike OpenAI and Perplexity, Anthropic says its user-triggered fetcher honours robots.txt. ## Allow and refuse lines ### To allow Claude-User User-agent: Claude-User Allow: / ### To refuse Claude-User User-agent: Claude-User Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) ## Other Anthropic rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ClaudeBot: user agent, token, block rate | Agent Ready Crawlers" description: "ClaudeBot from Anthropic: robots.txt token ClaudeBot, training purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/claudebot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ClaudeBot: user agent, token, block rate | Agent Ready Crawlers > ClaudeBot from Anthropic: robots.txt token ClaudeBot, training purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/claudebot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Anthropic # ClaudeBot In short ClaudeBot is Anthropic's training crawler. The token to write in robots.txt is ClaudeBot. Check a visitor against the operator's published IP list. Supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" then "Crawl-delay: 1". ClaudeBot: its bots honour robots.txt "do not crawl" directives; restricting ClaudeBot signals that future materials should be excluded from training datasets. Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string No exact string is published. What the operator or a third party says: Contains "ClaudeBot"; Anthropic does not publish a full example string on its help page robots.txt token `ClaudeBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Anthropic: its bots honour robots.txt "do not crawl" directives; restricting ClaudeBot signals that future materials should be excluded from training datasets. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://claude.com/crawling/bots.json), read 2026-11-04. IP ranges [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Rate behaviour Supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" then "Crawl-delay: 1". Anthropic says it will not attempt to bypass CAPTCHAs. Block rate **17.6%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `ClaudeBot` at the root (1,044 sites). 14.0% name it; 3.5% refuse it only through a \* group. Rank 4 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 13.1% (10.6% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Note. Anthropic warns that IP blocking may not work as an opt-out because it stops its bots reading robots.txt. bots.json creationTime read: 2026-10-20 (includes 216.73.216.0/22). Help article dated 2026-04-07. ## Allow and refuse lines ### To allow ClaudeBot User-agent: ClaudeBot Allow: / ### To refuse ClaudeBot User-agent: ClaudeBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) ## Other Anthropic rows - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare Browser Rendering profile | Agent Ready Crawlers" description: "Cloudflare Browser Rendering from Cloudflare: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/cloudflare-browser-rendering site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Cloudflare Browser Rendering profile | Agent Ready Crawlers > Cloudflare Browser Rendering from Cloudflare: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/cloudflare-browser-rendering Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Cloudflare # Cloudflare Browser Rendering In short Cloudflare Browser Rendering is Cloudflare's browsing agent. Cloudflare Browser Rendering has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. Cloudflare documents the automatic request headers; robots.txt handling is up to the customer's script. Operator Cloudflare Vendor profile: [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Sends signed Web Bot Auth headers robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No statement Cloudflare documents the automatic request headers; robots.txt handling is up to the customer's script. [Source](https://developers.cloudflare.com/browser-rendering/reference/automatic-request-headers/), read 2026-11-04. Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://developers.cloudflare.com/browser-rendering/reference/automatic-request-headers/), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the customer. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [blog.cloudflare.com](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. Note. Classed as a signed agent since 2025-08-28; always given bot score 1 for Bot Management customers, per Cloudflare. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Cloudflare in the Verification Ledger Signs Browser Rendering traffic. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory documented in Browser Rendering automatic request headers Cloudflare Cloudflare signed agent Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 - [developers.cloudflare.com: automatic request headers](https://developers.cloudflare.com/browser-rendering/reference/automatic-request-headers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "cohere-ai: user agent, token, block rate | Agent Ready Crawlers" description: "cohere-ai from Cohere: robots.txt token cohere-ai, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/cohere-ai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # cohere-ai: user agent, token, block rate | Agent Ready Crawlers > cohere-ai from Cohere: robots.txt token cohere-ai, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/cohere-ai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Cohere # cohere-ai In short cohere-ai is Cohere's bot, filed under Other. The token to write in robots.txt is cohere-ai. The operator publishes nothing a site can check. 15.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Token appears in many robots.txt files; Cohere's docs URL for a crawler page returned 404 on 2026-11-04. Operator Cohere Vendor profile: [Cohere](https://agentreadycrawlers.com/vendors/cohere/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string No exact string is published. What the operator or a third party says: Not published robots.txt token `cohere-ai` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). [Source](https://technologychecker.io/blog/ai-crawler-list), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://technologychecker.io/blog/ai-crawler-list), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **15.5%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `cohere-ai` at the root (919 sites). 11.6% name it; 3.8% refuse it only through a \* group. Rank 11 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.7% (8.9% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://technologychecker.io/blog/ai-crawler-list). Note. Token appears in many robots.txt files; Cohere's docs URL for a crawler page returned 404 on 2026-11-04. ## Allow and refuse lines ### To allow cohere-ai User-agent: cohere-ai Allow: / ### To refuse cohere-ai User-agent: cohere-ai Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Cohere in the Verification Ledger Publishes nothing. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [technologychecker.io: ai crawler list](https://technologychecker.io/blog/ai-crawler-list) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Browse with Copilot: user agent and token | Agent Ready Crawlers" description: "Browse with Copilot from Microsoft: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/copilot-actions site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Browse with Copilot: user agent and token | Agent Ready Crawlers > Browse with Copilot from Microsoft: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/copilot-actions Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Microsoft # Browse with Copilot (formerly Copilot Actions) In short Browse with Copilot (formerly Copilot Actions) is Microsoft's browsing agent. Browse with Copilot has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Browse with Copilot has no operator bot documentation, checked 2026-11-04. Runs in the current window with the profile's cookies. Operator [Microsoft](https://agentreadycrawlers.com/crawlers/operator/microsoft/) Vendor profile: [Microsoft](https://agentreadycrawlers.com/vendors/microsoft/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: The user's own Edge user agent; no marker documented robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) for site owners; actions run locally in the user's Edge. [Source](https://www.verisoul.ai/agents/copilot-edge), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/copilot-edge), read 2026-11-04. IP ranges Not published. Rate behaviour Runs in the current window with the profile's cookies. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/copilot-edge). Note. Copilot Actions launched as a US preview 2025-10-23 and was renamed Browse with Copilot on 2026-05-13, per Verisoul (secondary). ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Microsoft in the Verification Ledger bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json) Reverse DNS Verify Bingbot tool and documented methods Key directory not published Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Microsoft rows - [Bingbot](https://agentreadycrawlers.com/crawlers/bingbot/) Search index token bingbot - [MicrosoftPreview](https://agentreadycrawlers.com/crawlers/microsoftpreview/) Other token MicrosoftPreview ## Sources - [verisoul.ai: copilot edge](https://www.verisoul.ai/agents/copilot-edge) read 2026-11-04 - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Diffbot: user agent, token, block rate | Agent Ready Crawlers" description: "Diffbot from Diffbot: robots.txt token Diffbot, other purpose and what a site can check; no exact user-agent string is published. 15.7% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/diffbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Diffbot: user agent, token, block rate | Agent Ready Crawlers > Diffbot from Diffbot: robots.txt token Diffbot, other purpose and what a site can check; no exact user-agent string is published. 15.7% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/diffbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Diffbot # Diffbot In short Diffbot is Diffbot's bot, filed under Other. The token to write in robots.txt is Diffbot. The operator publishes nothing a site can check. Knowledge graph and extraction service; robots.txt compliance is a customer setting. Diffbot: its crawler checks robots.txt, but customers can opt to "ignore robots.txt" in crawl settings or via the Crawl API. Operator Diffbot Vendor profile: [Diffbot](https://agentreadycrawlers.com/vendors/diffbot/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string No exact string is published. What the operator or a third party says: Contains "Diffbot"; full string not on the page read robots.txt token `Diffbot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Depends on a setting Diffbot: its crawler checks robots.txt, but customers can opt to "ignore robots.txt" in crawl settings or via the Crawl API. [Source](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems), read 2026-11-04. IP ranges Not published. Rate behaviour Set per customer crawl job. Block rate **15.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Diffbot` at the root (932 sites). 11.9% name it; 3.8% refuse it only through a \* group. Rank 7 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.8% (9.1% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [docs.diffbot.com](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems), read 2026-11-04. Note. Knowledge graph and extraction service; robots.txt compliance is a customer setting. ## Allow and refuse lines ### To allow Diffbot User-agent: Diffbot Allow: / ### To refuse Diffbot User-agent: Diffbot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Diffbot in the Verification Ledger Publishes nothing for verification; robots.txt can be ignored by customer setting. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [docs.diffbot.com: why is my crawl not crawling and other uncommon crawl problems](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DuckAssistBot: user agent, token, block rate | Agent Ready Crawlers" description: "DuckAssistBot from DuckDuckGo: exact user-agent string, robots.txt token DuckAssistBot, live answer fetch purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/duckassistbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # DuckAssistBot: user agent, token, block rate | Agent Ready Crawlers > DuckAssistBot from DuckDuckGo: exact user-agent string, robots.txt token DuckAssistBot, live answer fetch purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/duckassistbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from DuckDuckGo # DuckAssistBot In short DuckAssistBot is DuckDuckGo's live answer fetcher. The token to write in robots.txt is DuckAssistBot. Check a visitor against the operator's published IP list. Verisoul reports it signs requests; DuckDuckGo's page names only the IP list. On robots.txt, disallowing DuckAssistBot takes effect after 72 hours; data is not used to train AI models; opting out does not affect organic rankings. Operator [DuckDuckGo](https://agentreadycrawlers.com/crawlers/operator/duckduckgo/) Vendor profile: [DuckDuckGo](https://agentreadycrawlers.com/vendors/duckduckgo/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)` robots.txt token `DuckAssistBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator DuckDuckGo: disallowing DuckAssistBot takes effect after 72 hours; data is not used to train AI models; opting out does not affect organic rankings. [Source](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://duckduckgo.com/duckassistbot.json), read 2026-11-04. IP ranges [duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json) Rate behaviour Crawls pages in real time for AI-assisted answers. Block rate **8.1%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `DuckAssistBot` at the root (483 sites). 4.3% name it; 3.8% refuse it only through a \* group. Rank 25 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.8% (3.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), read 2026-11-04. Note. Verisoul reports it signs requests; DuckDuckGo's page names only the IP list. ## Allow and refuse lines ### To allow DuckAssistBot User-agent: DuckAssistBot Allow: / ### To refuse DuckAssistBot User-agent: DuckAssistBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## DuckDuckGo in the Verification Ledger Two IP lists. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json) - [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other DuckDuckGo rows - [DuckDuckBot](https://agentreadycrawlers.com/crawlers/duckduckbot/) Search index token DuckDuckBot ## Sources - [duckduckgo.com: duckassistbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) read 2026-11-04 - [duckduckgo.com: duckassistbot (JSON)](https://duckduckgo.com/duckassistbot.json) read 2026-11-04 - [Verisoul: DuckAssistBot](https://www.verisoul.ai/agents/duckassistbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DuckDuckBot: user agent, token, block rate | Agent Ready Crawlers" description: "DuckDuckBot from DuckDuckGo: exact user-agent string, robots.txt token DuckDuckBot, search index purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/duckduckbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # DuckDuckBot: user agent, token, block rate | Agent Ready Crawlers > DuckDuckBot from DuckDuckGo: exact user-agent string, robots.txt token DuckDuckBot, search index purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/duckduckbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from DuckDuckGo # DuckDuckBot In short DuckDuckBot is DuckDuckGo's search index crawler. The token to write in robots.txt is DuckDuckBot. Check a visitor against the operator's published IP list. The crawl of 2026-11-04 did not measure DuckDuckBot. Its user-agent entry: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html). Builds a search index that sends visitors back. Refusing it removes pages from that product. On robots.txt, DuckDuckGo: DuckDuckBot respects robotstxt.org. Operator [DuckDuckGo](https://agentreadycrawlers.com/crawlers/operator/duckduckgo/) Vendor profile: [DuckDuckGo](https://agentreadycrawlers.com/vendors/duckduckgo/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)` robots.txt token `DuckDuckBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator DuckDuckGo: DuckDuckBot respects robotstxt.org. [Source](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://duckduckgo.com/duckduckbot.json), read 2026-11-04. IP ranges [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json) Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `DuckDuckBot` is not one of them. Documentation [duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot), read 2026-11-04. Note. Search crawler, separate from DuckAssistBot. ## Allow and refuse lines ### To allow DuckDuckBot User-agent: DuckDuckBot Allow: / ### To refuse DuckDuckBot User-agent: DuckDuckBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## DuckDuckGo in the Verification Ledger Two IP lists. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json) - [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other DuckDuckGo rows - [DuckAssistBot](https://agentreadycrawlers.com/crawlers/duckassistbot/) Live answer fetch token DuckAssistBot ## Sources - [duckduckgo.com: duckduckbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) read 2026-11-04 - [duckduckgo.com: duckduckbot (JSON)](https://duckduckgo.com/duckduckbot.json) read 2026-11-04 - [duckduckgo.com: duckassistbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ExaSearchBot: user agent, token, block rate | Agent Ready Crawlers" description: "ExaSearchBot from Exa: robots.txt token ExaSearchBot, search index purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/exasearchbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ExaSearchBot: user agent, token, block rate | Agent Ready Crawlers > ExaSearchBot from Exa: robots.txt token ExaSearchBot, search index purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/exasearchbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Exa # ExaSearchBot In short ExaSearchBot is Exa's search index crawler. The token to write in robots.txt is ExaSearchBot. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure ExaSearchBot. ExaSearchBot has no operator bot documentation, checked 2026-11-04. Builds the index behind Exa's AI search API; Verisoul reports it signs requests. Operator Exa Vendor profile: [Exa](https://agentreadycrawlers.com/vendors/exa/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string No exact string is published. What the operator or a third party says: Contains "ExaSearchBot" robots.txt token `ExaSearchBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) found on the pages read. [Source](https://www.verisoul.ai/agents/exasearchbot), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/exasearchbot), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `ExaSearchBot` is not one of them. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/exasearchbot). Note. Builds the index behind Exa's AI search API; Verisoul reports it signs requests. ## Allow and refuse lines ### To allow ExaSearchBot User-agent: ExaSearchBot Allow: / ### To refuse ExaSearchBot User-agent: ExaSearchBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Exa in the Verification Ledger Publishes nothing we could read. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [verisoul.ai: exasearchbot](https://www.verisoul.ai/agents/exasearchbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "FacebookBot: user agent, token, block rate | Agent Ready Crawlers" description: "FacebookBot from Meta: robots.txt token FacebookBot, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/facebookbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # FacebookBot: user agent, token, block rate | Agent Ready Crawlers > FacebookBot from Meta: robots.txt token FacebookBot, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/facebookbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Meta # FacebookBot (legacy token) In short FacebookBot (legacy token) is Meta's bot, filed under Other. The token to write in robots.txt is FacebookBot. The operator publishes nothing a site can check. Still named in many robots.txt files (twitter.com names it in the shared crawl, 2026-11-04). On robots.txt, not listed on Meta's current crawler page, which names FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher (checked 2026-11-04). Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string No exact string is published. What the operator or a third party says: Not listed on Meta's current crawler page robots.txt token `FacebookBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Not on current operator page Not listed on Meta's current crawler page, which names FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher (checked 2026-11-04). [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. IP ranges Not published. Rate behaviour Not documented. Block rate **10.2%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `FacebookBot` at the root (603 sites). 6.4% name it; 3.7% refuse it only through a \* group. Rank 18 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 7.3% (4.6% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Note. Still named in many robots.txt files (twitter.com names it in the shared crawl, 2026-11-04). Kept as a legacy row. ## Allow and refuse lines ### To allow FacebookBot User-agent: FacebookBot Allow: / ### To refuse FacebookBot User-agent: FacebookBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Meta rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Training token meta-externalagent - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) Live answer fetch token meta-externalfetcher - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Search index token meta-webindexer - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) Other token meta-externalads - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) Other token facebookexternalhit - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) Shopping agent no token ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "FacebookExternalHit: user agent and token | Agent Ready Crawlers" description: "FacebookExternalHit from Meta: exact user-agent string, robots.txt token facebookexternalhit, other purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/facebookexternalhit site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # FacebookExternalHit: user agent and token | Agent Ready Crawlers > FacebookExternalHit from Meta: exact user-agent string, robots.txt token facebookexternalhit, other purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/facebookexternalhit Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Meta # FacebookExternalHit In short FacebookExternalHit is Meta's bot, filed under Other. The token to write in robots.txt is facebookexternalhit. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure facebookexternalhit. Meta suggests allowlisting by IP as the more secure option but the page links no IP list. FacebookExternalHit might bypass robots.txt when performing security or integrity checks. Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)` robots.txt token `facebookexternalhit` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt May not apply (user-triggered) Meta: might bypass robots.txt when performing security or integrity checks. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. IP ranges Not published. Rate behaviour Requests may use a Range header; Open Graph tags must sit in the first 1 MB. Block rate Not measured: the crawl checked 31 AI and search tokens, and `facebookexternalhit` is not one of them. Documentation [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Note. Link-preview fetcher for shared links. Meta suggests allowlisting by IP as the more secure option but the page links no IP list. ## Allow and refuse lines ### To allow facebookexternalhit User-agent: facebookexternalhit Allow: / ### To refuse facebookexternalhit User-agent: facebookexternalhit Disallow: / Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Meta rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Training token meta-externalagent - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) Live answer fetch token meta-externalfetcher - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Search index token meta-webindexer - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) Other token meta-externalads - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) Other token FacebookBot - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) Shopping agent no token ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "FirecrawlAgent: user agent, token, block rate | Agent Ready Crawlers" description: "FirecrawlAgent from Firecrawl: robots.txt token FirecrawlAgent, other purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/firecrawl site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # FirecrawlAgent: user agent, token, block rate | Agent Ready Crawlers > FirecrawlAgent from Firecrawl: robots.txt token FirecrawlAgent, other purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/firecrawl Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Firecrawl # FirecrawlAgent In short FirecrawlAgent is Firecrawl's bot, filed under Other. The token to write in robots.txt is FirecrawlAgent. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure FirecrawlAgent. Firecrawl advises allowlisting with robots.txt rather than a WAF rule on User-Agent. FirecrawlAgent reads robots.txt before it crawls and applies rules for FirecrawlAgent and for \*. Operator Firecrawl Vendor profile: [Firecrawl](https://agentreadycrawlers.com/vendors/firecrawl/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string No exact string is published. What the operator or a third party says: Not fixed: Firecrawl says do not assume FirecrawlAgent is the User-Agent header on the fetch robots.txt token `FirecrawlAgent` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Firecrawl: reads robots.txt before it crawls and applies rules for FirecrawlAgent and for \*. [Source](https://docs.firecrawl.dev/advanced-scraping-guide), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://docs.firecrawl.dev/advanced-scraping-guide), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the customer job. Block rate Not measured: the crawl checked 31 AI and search tokens, and `FirecrawlAgent` is not one of them. Documentation [docs.firecrawl.dev](https://docs.firecrawl.dev/advanced-scraping-guide), read 2026-11-04. Note. Firecrawl advises allowlisting with robots.txt rather than a WAF rule on User-Agent. GitHub issue #1169 records complaints that the token never appears in logs. ## Allow and refuse lines ### To allow FirecrawlAgent User-agent: FirecrawlAgent Allow: / ### To refuse FirecrawlAgent User-agent: FirecrawlAgent Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Firecrawl in the Verification Ledger Robots.txt token only; Firecrawl advises against UA-based allowlisting. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [docs.firecrawl.dev: advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Gemini in Chrome: user agent and token | Agent Ready Crawlers" description: "Gemini in Chrome from Google: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/gemini-in-chrome site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Gemini in Chrome: user agent and token | Agent Ready Crawlers > Gemini in Chrome from Google: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/gemini-in-chrome Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Google # Gemini in Chrome (auto browse) In short Gemini in Chrome (auto browse) is Google's browsing agent. Gemini in Chrome has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Verisoul reports caps of 20 tasks a day on AI Pro and 200 on Ultra. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: The user's own Chrome user agent; no marker in local mode robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) for auto browse in the user's own Chrome; Google's cloud agents declare Google-Agent instead. [Source](https://www.verisoul.ai/agents/chrome-auto-browse), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/chrome-auto-browse), read 2026-11-04. IP ranges Not published. Rate behaviour Verisoul reports caps of 20 tasks a day on AI Pro and 200 on Ultra. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/chrome-auto-browse). Note. Auto browse launched 2026-01-28 as a US preview for AI Pro and Ultra subscribers, per Verisoul (secondary). It hands control back to the person to pay or accept terms. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [verisoul.ai: chrome auto browse](https://www.verisoul.ai/agents/chrome-auto-browse) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google-Agent: user agent, token, block rate | Agent Ready Crawlers" description: "Google-Agent from Google: exact user-agent string, no robots.txt token, task agent purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/google-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Google-Agent: user agent, token, block rate | Agent Ready Crawlers > Google-Agent from Google: exact user-agent string, no robots.txt token, task agent purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/google-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Google # Google-Agent In short Google-Agent is Google's browsing agent. Google-Agent has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. Google lists it as a user-triggered fetcher; these fetchers generally ignore robots.txt because the fetch was requested by a user. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36` robots.txt token None. No robots.txt line can address this agent. Respects robots.txt May not apply (user-triggered) Google lists it as a user-triggered fetcher; these fetchers generally ignore robots.txt because the fetch was requested by a user. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04. Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://agent.bot.goog/.well-known/http-message-signatures-directory), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Rate behaviour User-triggered; no rate published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth), read 2026-11-04. Note. Used by agents hosted on Google infrastructure to navigate and act on user request. A subset of requests is signed with Web Bot Auth as https://agent.bot.goog (experimental); Google says keep IP and reverse-DNS checks as not every request is signed. user-triggered-agents.json creationTime 2026-03-03. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-08-19[Google retires the Google-NotebookLM user agent](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 - [agent.bot.goog: http message signatures directory](https://agent.bot.goog/.well-known/http-message-signatures-directory) read 2026-11-04 - [developers.google.com: user triggered agents (JSON)](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google-CloudVertexBot: user agent and token | Agent Ready Crawlers" description: "Google-CloudVertexBot from Google: robots.txt token Google-CloudVertexBot, other purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/google-cloudvertexbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Google-CloudVertexBot: user agent and token | Agent Ready Crawlers > Google-CloudVertexBot from Google: robots.txt token Google-CloudVertexBot, other purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/google-cloudvertexbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Google # Google-CloudVertexBot In short Google-CloudVertexBot is Google's bot, filed under Other. The token to write in robots.txt is Google-CloudVertexBot. Verify it by reverse DNS, then a forward lookup. Also matches the Googlebot token group per Google's table. On robots.txt, preferences addressed to Google-CloudVertexBot affect crawls requested by site owners for building Vertex AI Agents; no effect on Search. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string No exact string is published. What the operator or a third party says: User-Agent substring "Google-CloudVertexBot" robots.txt token `Google-CloudVertexBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Google: preferences addressed to Google-CloudVertexBot affect crawls requested by site owners for building Vertex AI Agents; no effect on Search. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) Rate behaviour Crawls are requested by the site owner. Block rate **10.9%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Google-CloudVertexBot` at the root (645 sites). 7.1% name it; 3.8% refuse it only through a \* group. Rank 16 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 8.3% (5.5% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Note. Also matches the Googlebot token group per Google's table. ## Allow and refuse lines ### To allow Google-CloudVertexBot User-agent: Google-CloudVertexBot Allow: / ### To refuse Google-CloudVertexBot User-agent: Google-CloudVertexBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [developers.google.com: googlebot (JSON)](https://developers.google.com/static/search/apis/ipranges/googlebot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google-Extended: user agent, token, block rate | Agent Ready Crawlers" description: "Google-Extended from Google: robots.txt token Google-Extended, training purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/google-extended site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Google-Extended: user agent, token, block rate | Agent Ready Crawlers > Google-Extended from Google: robots.txt token Google-Extended, training purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/google-extended Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Google # Google-Extended (control token) In short Google-Extended (control token) is Google's training crawler. The token to write in robots.txt is Google-Extended. The operator publishes nothing a site can check. 16.0 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Google says it intends to support the AIPREF training preference signal (user-agent: \_\_usage\_ai-training) in early 2027. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string No exact string is published. What the operator or a third party says: None: crawling uses existing Google user agent strings; the token is a control only robots.txt token `Google-Extended` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Google: Google-Extended manages whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI; it does not affect Search inclusion or ranking. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. IP ranges Not published. Rate behaviour No separate crawl. Block rate **16.0%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Google-Extended` at the root (949 sites). 12.6% name it; 3.4% refuse it only through a \* group. Rank 5 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 12.1% (9.6% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Note. Google says it intends to support the AIPREF training preference signal (user-agent: \_\_usage\_ai-training) in early 2027. ## Allow and refuse lines ### To allow Google-Extended User-agent: Google-Extended Allow: / ### To refuse Google-Extended User-agent: Google-Extended Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google-GeminiNotebook: user agent and token | Agent Ready Crawlers" description: "Google-GeminiNotebook from Google: exact user-agent string, no robots.txt token, live answer fetch purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/google-gemininotebook site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Google-GeminiNotebook: user agent and token | Agent Ready Crawlers > Google-GeminiNotebook from Google: exact user-agent string, no robots.txt token, live answer fetch purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/google-gemininotebook Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from Google # Google-GeminiNotebook (was Google-NotebookLM) In short Google-GeminiNotebook (was Google-NotebookLM) is Google's live answer fetcher. Google-GeminiNotebook has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. Check a visitor against the operator's published IP list. The former Google-NotebookLM agent string was supported until August 2026, per Google (page last updated 2026-08-19). Google-GeminiNotebook generally ignores robots.txt, per Google. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)` robots.txt token None. No robots.txt line can address this agent. Respects robots.txt May not apply (user-triggered) User-triggered fetcher: generally ignores robots.txt, per Google. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) Rate behaviour Fetches individual URLs a user adds as sources. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation [developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04. Note. The former Google-NotebookLM agent string was supported until August 2026, per Google (page last updated 2026-08-19). ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Use the verification method above to decide at the WAF. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-08-19[Google retires the Google-NotebookLM user agent](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token ## Sources - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 - [developers.google.com: user triggered fetchers google (JSON)](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Googlebot: user agent, token, block rate | Agent Ready Crawlers" description: "Googlebot from Google: exact user-agent string, robots.txt token Googlebot, search index purpose and how to verify it. 2.8% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/googlebot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Googlebot: user agent, token, block rate | Agent Ready Crawlers > Googlebot from Google: exact user-agent string, robots.txt token Googlebot, search index purpose and how to verify it. 2.8% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/googlebot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Google # Googlebot In short Googlebot is Google's search index crawler. The token to write in robots.txt is Googlebot. Verify it by reverse DNS, then a forward lookup. 2.8 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Reverse DNS: crawl-\*\*\*-\*\*\*-\*\*\*-\*\*\*.googlebot.com or geo-crawl-\*\*\*.geo.googlebot.com, then forward-confirm. Googlebot: common crawlers always obey robots.txt rules when crawling automatically. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` robots.txt token `Googlebot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Google: common crawlers always obey robots.txt rules when crawling automatically. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) Rate behaviour Crawl rate can be reduced via Google's documented process; Google does not support Crawl-delay. Block rate **2.8%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Googlebot` at the root (166 sites). 0.5% name it; 2.3% refuse it only through a \* group. Rank 31 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 2.1% (0.4% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Note. Reverse DNS: crawl-\*\*\*-\*\*\*-\*\*\*-\*\*\*.googlebot.com or geo-crawl-\*\*\*.geo.googlebot.com, then forward-confirm. Cloudflare classes Googlebot as an Accountable mixed-use crawler (2026-09-15); its "Block" setting now stops Googlebot entirely. ## Allow and refuse lines ### To allow Googlebot User-agent: Googlebot Allow: / ### To refuse Googlebot User-agent: Googlebot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) ## Other Google rows - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [developers.google.com: googlebot (JSON)](https://developers.google.com/static/search/apis/ipranges/googlebot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "GoogleOther: user agent, token, block rate | Agent Ready Crawlers" description: "GoogleOther from Google: exact user-agent string, robots.txt token GoogleOther, other purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/googleother site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # GoogleOther: user agent, token, block rate | Agent Ready Crawlers > GoogleOther from Google: exact user-agent string, robots.txt token GoogleOther, other purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/googleother Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Google # GoogleOther In short GoogleOther is Google's bot, filed under Other. The token to write in robots.txt is GoogleOther. Verify it by reverse DNS, then a forward lookup. The crawl of 2026-11-04 did not measure GoogleOther. Blocking GoogleOther does not affect any specific Google product, per Google. GoogleOther obeys robots.txt when crawling automatically, per Google. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `GoogleOther` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Common crawler: obeys robots.txt when crawling automatically, per Google. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) Rate behaviour Not product specific; used for one-off crawls such as internal research and development. Block rate Not measured: the crawl checked 31 AI and search tokens, and `GoogleOther` is not one of them. Documentation [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Note. Blocking GoogleOther does not affect any specific Google product, per Google. ## Allow and refuse lines ### To allow GoogleOther User-agent: GoogleOther Allow: / ### To refuse GoogleOther User-agent: GoogleOther Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [developers.google.com: googlebot (JSON)](https://developers.google.com/static/search/apis/ipranges/googlebot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Goose: user agent, token, block rate | Agent Ready Crawlers" description: "Goose from Block: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/goose site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Goose: user agent, token, block rate | Agent Ready Crawlers > Goose from Block: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/goose Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Block # Goose (open-source agent) In short Goose (open-source agent) is Block's browsing agent. Goose has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. Cloudflare lists it as a signed agent. Goose has no operator bot documentation, checked 2026-11-04. A browsing agent that clicks, fills forms and finishes a job for a person. In Cloudflare's first signed-agents cohort (2025-08-28). Operator Block Vendor profile: [Block](https://agentreadycrawlers.com/vendors/block/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Signs with Web Bot Auth under Cloudflare's signed agents program robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) on robots.txt. [Source](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Cloudflare signed agent Cloudflare lists the agent as a signed agent; the operator publishes no list of its own. [Documentation](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. IP ranges Not published. Rate behaviour Runs locally by default. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://blog.cloudflare.com/signed-agents/). Note. In Cloudflare's first signed-agents cohort (2025-08-28). ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Use the verification method above to decide at the WAF. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Block in the Verification Ledger Open-source agent. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory signs under Cloudflare's signed agents program Cloudflare Cloudflare signed agent (cohort of 2025-08-28) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "GPTBot: user agent, token, block rate | Agent Ready Crawlers" description: "GPTBot from OpenAI: exact user-agent string, robots.txt token GPTBot, training purpose and how to verify it. 18.5% of top sites block it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/gptbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # GPTBot: user agent, token, block rate | Agent Ready Crawlers > GPTBot from OpenAI: exact user-agent string, robots.txt token GPTBot, training purpose and how to verify it. 18.5% of top sites block it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/gptbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from OpenAI # GPTBot In short GPTBot is OpenAI's training crawler. The token to write in robots.txt is GPTBot. Check a visitor against the operator's published IP list. 18.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot` robots.txt token `GPTBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator OpenAI: disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://openai.com/gptbot.json), read 2026-11-04. IP ranges [openai.com/gptbot.json](https://openai.com/gptbot.json) Rate behaviour OpenAI says changes to robots.txt take about 24 hours to reach its systems, and that if both GPTBot and OAI-SearchBot are allowed it may use one crawl for both uses. gptbot.json creationTime read: 2026-09-22. Block rate **18.5%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `GPTBot` at the root (1,097 sites). 15.2% name it; 3.3% refuse it only through a \* group. Rank 3 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 13.9% (11.5% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [platform.openai.com](https://platform.openai.com/docs/bots), read 2026-11-04. Note. Version 1.4 in the published example string. When fetching robots.txt it may add a "robots.txt" marker to the user agent. Listed as Verified on Cloudflare Radar (radar.cloudflare.com/bots/directory/gptbot, read 2026-11-04). ## Allow and refuse lines ### To allow GPTBot User-agent: GPTBot Allow: / ### To refuse GPTBot User-agent: GPTBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Search index token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: gptbot (JSON)](https://openai.com/gptbot.json) read 2026-11-04 - [Cloudflare Radar: GPTBot](https://radar.cloudflare.com/bots/directory/gptbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Grok Bot: user agent, token, block rate | Agent Ready Crawlers" description: "Grok Bot from xAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/grok-bot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Grok Bot: user agent, token, block rate | Agent Ready Crawlers > Grok Bot from xAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/grok-bot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from xAI # Grok Bot (agent) In short Grok Bot (agent) is xAI's browsing agent. Grok Bot has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Grok Bot has no operator bot documentation, checked 2026-11-04. Verisoul reports Grok Bot stopped signing requests in mid-October 2026 and uses the user's IP. Operator xAI Vendor profile: [xAI](https://agentreadycrawlers.com/vendors/xai/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: No declared identity since mid-October 2026, per Verisoul robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). [Source](https://www.verisoul.ai/agents/xai-bots), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/grok-bot), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/grok-bot). Note. Verisoul reports Grok Bot stopped signing requests in mid-October 2026 and uses the user's IP. Secondary source. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## xAI in the Verification Ledger Publishes nothing; Verisoul reports Grok Bot stopped signing in mid-October 2026. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [verisoul.ai: grok bot](https://www.verisoul.ai/agents/grok-bot) read 2026-11-04 - [verisoul.ai: xai bots](https://www.verisoul.ai/agents/xai-bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ImagesiftBot: user agent, token, block rate | Agent Ready Crawlers" description: "ImagesiftBot from Hive (ImageSift): exact user-agent string, robots.txt token ImagesiftBot, other purpose and how to verify it. 9.5% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/imagesiftbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ImagesiftBot: user agent, token, block rate | Agent Ready Crawlers > ImagesiftBot from Hive (ImageSift): exact user-agent string, robots.txt token ImagesiftBot, other purpose and how to verify it. 9.5% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/imagesiftbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Hive (ImageSift) # ImagesiftBot In short ImagesiftBot is Hive (ImageSift)'s bot, filed under Other. The token to write in robots.txt is ImagesiftBot. The operator publishes nothing a site can check. 9.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Scrapes public images for web intelligence products. On robots.txt, standard directives that target ImagesiftBot are respected. Operator Hive (ImageSift) Vendor profile: [Hive](https://agentreadycrawlers.com/vendors/hive-imagesift/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)` robots.txt token `ImagesiftBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator ImageSift: standard directives that target ImagesiftBot are respected. [Source](https://imagesift.com/about), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://imagesift.com/about), read 2026-11-04. IP ranges Not published. Rate behaviour Supports Crawl-delay as the minimum seconds between request starts. Block rate **9.5%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `ImagesiftBot` at the root (563 sites). 5.5% name it; 3.9% refuse it only through a \* group. Rank 21 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 6.9% (4.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [imagesift.com](https://imagesift.com/about), read 2026-11-04. Note. Scrapes public images for web intelligence products. If no rule targets ImagesiftBot it falls back to rules for other crawlers, per the page. ## Allow and refuse lines ### To allow ImagesiftBot User-agent: ImagesiftBot Allow: / ### To refuse ImagesiftBot User-agent: ImagesiftBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Hive (ImageSift) in the Verification Ledger User agent string and robots.txt behaviour only. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [imagesift.com: about](https://imagesift.com/about) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Manus cloud browser: user agent and token | Agent Ready Crawlers" description: "Manus cloud browser from Manus: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/manus site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Manus cloud browser: user agent and token | Agent Ready Crawlers > Manus cloud browser from Manus: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/manus Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Manus # Manus cloud browser In short Manus cloud browser is Manus' browsing agent. Manus cloud browser has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Manus cloud browser has no operator bot documentation, checked 2026-11-04. Local browser modes look like the user's own traffic, per Verisoul. Operator Manus Vendor profile: [Manus](https://agentreadycrawlers.com/vendors/manus/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Signs cloud-browser page loads, per Verisoul robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). [Source](https://www.verisoul.ai/agents/manus-bot), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/manus-bot), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/manus-bot). Note. Local browser modes look like the user's own traffic, per Verisoul. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Manus in the Verification Ledger Publishes nothing we could read. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [verisoul.ai: manus bot](https://www.verisoul.ai/agents/manus-bot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Meta-ExternalAds: user agent and token | Agent Ready Crawlers" description: "Meta-ExternalAds from Meta: exact user-agent string, robots.txt token meta-externalads, other purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/meta-externalads site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Meta-ExternalAds: user agent and token | Agent Ready Crawlers > Meta-ExternalAds from Meta: exact user-agent string, robots.txt token meta-externalads, other purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/meta-externalads Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Meta # Meta-ExternalAds In short Meta-ExternalAds is Meta's bot, filed under Other. The token to write in robots.txt is meta-externalads. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure meta-externalads. Improves advertising and other business products, per Meta. Its user-agent entry: meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). Meta shows robots.txt groups for meta-externalads in its example file. Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)` robots.txt token `meta-externalads` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Meta shows robots.txt groups for meta-externalads in its example file. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `meta-externalads` is not one of them. Documentation [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Note. Improves advertising and other business products, per Meta. ## Allow and refuse lines ### To allow meta-externalads User-agent: meta-externalads Allow: / ### To refuse meta-externalads User-agent: meta-externalads Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Meta rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Training token meta-externalagent - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) Live answer fetch token meta-externalfetcher - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Search index token meta-webindexer - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) Other token facebookexternalhit - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) Other token FacebookBot - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) Shopping agent no token ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Meta-ExternalAgent: user agent and token | Agent Ready Crawlers" description: "Meta-ExternalAgent from Meta: exact user-agent string, robots.txt token meta-externalagent, training purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/meta-externalagent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Meta-ExternalAgent: user agent and token | Agent Ready Crawlers > Meta-ExternalAgent from Meta: exact user-agent string, robots.txt token meta-externalagent, training purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/meta-externalagent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Meta # Meta-ExternalAgent In short Meta-ExternalAgent is Meta's training crawler. The token to write in robots.txt is meta-externalagent. The operator publishes nothing a site can check. 16.0 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Meta documents blocking with a robots.txt disallow for the crawler; the file may be cached up to 24 hours. Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string `meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)` robots.txt token `meta-externalagent` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Meta documents blocking with a robots.txt disallow for the crawler; robots.txt may be cached up to 24 hours. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **16.0%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Meta-ExternalAgent` at the root (950 sites). 12.3% name it; 3.7% refuse it only through a \* group. Rank 6 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 12.0% (9.4% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Note. Crawls for training foundation AI models or indexing content directly, per Meta. Meta publishes no IP list or reverse-DNS pattern on this page. ## Allow and refuse lines ### To allow meta-externalagent User-agent: meta-externalagent Allow: / ### To refuse meta-externalagent User-agent: meta-externalagent Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-15[DataDome Q3 report: 25 billion AI agent requests](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) ## Other Meta rows - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) Live answer fetch token meta-externalfetcher - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Search index token meta-webindexer - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) Other token meta-externalads - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) Other token facebookexternalhit - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) Other token FacebookBot - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) Shopping agent no token ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Meta-ExternalFetcher: user agent and token | Agent Ready Crawlers" description: "Meta-ExternalFetcher from Meta: exact user-agent string, robots.txt token meta-externalfetcher, live answer fetch purpose and how to verify it." canonical: https://agentreadycrawlers.com/crawlers/meta-externalfetcher site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Meta-ExternalFetcher: user agent and token | Agent Ready Crawlers > Meta-ExternalFetcher from Meta: exact user-agent string, robots.txt token meta-externalfetcher, live answer fetch purpose and how to verify it. Source: https://agentreadycrawlers.com/crawlers/meta-externalfetcher Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from Meta # Meta-ExternalFetcher In short Meta-ExternalFetcher is Meta's live answer fetcher. The operator publishes nothing a site can check. 8.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). Meta-ExternalFetcher may bypass robots.txt because it performs fetches requested by the user. Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)` robots.txt token `meta-externalfetcher` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt May not apply (user-triggered) Meta: may bypass robots.txt because it performs fetches requested by the user. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. IP ranges Not published. Rate behaviour User-triggered. Block rate **8.5%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Meta-ExternalFetcher` at the root (505 sites). 4.7% name it; 3.8% refuse it only through a \* group. Rank 22 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 6.0% (3.3% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Note. Meta says it supports evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks. ## Allow and refuse lines ### To allow meta-externalfetcher User-agent: meta-externalfetcher Allow: / ### To refuse meta-externalfetcher User-agent: meta-externalfetcher Disallow: / Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Meta rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Training token meta-externalagent - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Search index token meta-webindexer - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) Other token meta-externalads - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) Other token facebookexternalhit - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) Other token FacebookBot - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) Shopping agent no token ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Muse: user agent, token, block rate | Agent Ready Crawlers" description: "Muse from Meta: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/meta-muse site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Muse: user agent, token, block rate | Agent Ready Crawlers > Muse from Meta: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/meta-muse Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Shopping agent from Meta # Muse (personal agent) In short Muse (personal agent) is Meta's shopping agent. Muse has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Muse has no operator bot documentation, checked 2026-11-04. Meta launched Muse on 2026-09-08 in the US, per Verisoul. Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Shopping agent [All shopping agent rows](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/). An agent that compares, selects or buys goods for a person. User-agent string No exact string is published. What the operator or a third party says: Does not identify itself, per Verisoul and Amazon's statement reported by GeekWire robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). [Source](https://www.verisoul.ai/agents/meta-muse), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/meta-muse), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/meta-muse). Note. Meta launched Muse on 2026-09-08 in the US, per Verisoul. Amazon blocked it in September 2026. DataDome's Q3 2026 report (via Help Net Security) counted 29.1 million Muse requests in under two weeks without self-identification. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-15[DataDome Q3 report: 25 billion AI agent requests](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) - 2026-09-08[Meta launches Muse, a personal agent that does not identify itself](https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i/) ## Other Meta rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Training token meta-externalagent - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) Live answer fetch token meta-externalfetcher - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Search index token meta-webindexer - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) Other token meta-externalads - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) Other token facebookexternalhit - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) Other token FacebookBot ## Sources - [verisoul.ai: meta muse](https://www.verisoul.ai/agents/meta-muse) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 - [Help Net Security: DataDome AI agent traffic report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report) read 2026-11-04 - [GeekWire: Amazon blocks Meta's Muse](https://www.geekwire.com/2026/amazon-blocks-metas-muse-ai-assistant-in-new-standoff-over-agentic-shopping/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Meta-WebIndexer: user agent, token, block rate | Agent Ready Crawlers" description: "Meta-WebIndexer from Meta: exact user-agent string, robots.txt token meta-webindexer, search index purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/meta-webindexer site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Meta-WebIndexer: user agent, token, block rate | Agent Ready Crawlers > Meta-WebIndexer from Meta: exact user-agent string, robots.txt token meta-webindexer, search index purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/meta-webindexer Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Meta # Meta-WebIndexer In short Meta-WebIndexer is Meta's search index crawler. The token to write in robots.txt is meta-webindexer. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure meta-webindexer. Its user-agent entry: meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). Meta-WebIndexer: allowing Meta-WebIndexer in robots.txt helps Meta cite and link content in Meta AI responses. Operator [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/) Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)` robots.txt token `meta-webindexer` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Meta: allowing Meta-WebIndexer in robots.txt helps Meta cite and link content in Meta AI responses. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `meta-webindexer` is not one of them. Documentation [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. Note. Search-index crawler for Meta AI. ## Allow and refuse lines ### To allow meta-webindexer User-agent: meta-webindexer Allow: / ### To refuse meta-webindexer User-agent: meta-webindexer Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Meta in the Verification Ledger User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-15[DataDome Q3 report: 25 billion AI agent requests](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) ## Other Meta rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Training token meta-externalagent - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) Live answer fetch token meta-externalfetcher - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) Other token meta-externalads - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) Other token facebookexternalhit - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) Other token FacebookBot - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) Shopping agent no token ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "MicrosoftPreview: user agent and token | Agent Ready Crawlers" description: "MicrosoftPreview from Microsoft: exact user-agent string, robots.txt token MicrosoftPreview, other purpose and how to verify it. 4.0% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/microsoftpreview site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # MicrosoftPreview: user agent and token | Agent Ready Crawlers > MicrosoftPreview from Microsoft: exact user-agent string, robots.txt token MicrosoftPreview, other purpose and how to verify it. 4.0% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/microsoftpreview Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Microsoft # MicrosoftPreview In short MicrosoftPreview is Microsoft's bot, filed under Other. The token to write in robots.txt is MicrosoftPreview. Verify it by reverse DNS, then a forward lookup. 4.0 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). MicrosoftPreview is covered by Bing's general robots.txt guidance for its crawlers. Operator [Microsoft](https://agentreadycrawlers.com/crawlers/operator/microsoft/) Vendor profile: [Microsoft](https://agentreadycrawlers.com/vendors/microsoft/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MicrosoftPreview/2.0; +https://aka.ms/MicrosoftPreview) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `MicrosoftPreview` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Covered by Bing's general robots.txt guidance for its crawlers. [Source](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **4.0%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `MicrosoftPreview` at the root (235 sites). 0.0% name it; 3.9% refuse it only through a \* group. Rank 29 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 2.9% (0.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [bing.com](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), read 2026-11-04. Note. Generates page snapshots for Microsoft products; desktop and mobile variants. ## Allow and refuse lines ### To allow MicrosoftPreview User-agent: MicrosoftPreview Allow: / ### To refuse MicrosoftPreview User-agent: MicrosoftPreview Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Microsoft in the Verification Ledger bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json) Reverse DNS Verify Bingbot tool and documented methods Key directory not published Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Microsoft rows - [Bingbot](https://agentreadycrawlers.com/crawlers/bingbot/) Search index token bingbot - [Browse with Copilot (formerly Copilot Actions)](https://agentreadycrawlers.com/crawlers/copilot-actions/) Task agent no token ## Sources - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [bing.com: how to verify bingbot 3905dc26](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "MistralAI-Index: user agent, token, block rate | Agent Ready Crawlers" description: "MistralAI-Index from Mistral AI: exact user-agent string, robots.txt token MistralAI-Index, search index purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/mistralai-index site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # MistralAI-Index: user agent, token, block rate | Agent Ready Crawlers > MistralAI-Index from Mistral AI: exact user-agent string, robots.txt token MistralAI-Index, search index purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/mistralai-index Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Mistral AI # MistralAI-Index In short MistralAI-Index is Mistral AI's search index crawler. The token to write in robots.txt is MistralAI-Index. Check a visitor against the operator's published IP list. The crawl of 2026-11-04 did not measure MistralAI-Index. Its user-agent entry: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots). On robots.txt, indexing only; content is not used for generative AI training of any kind. Operator [Mistral AI](https://agentreadycrawlers.com/crawlers/operator/mistral-ai/) Vendor profile: [Mistral AI](https://agentreadycrawlers.com/vendors/mistral-ai/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)` robots.txt token `MistralAI-Index` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Mistral: indexing only; content is not used for generative AI training of any kind. [Source](https://docs.mistral.ai/robots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://mistral.ai/mistralai-index-ips.json), read 2026-11-04. IP ranges [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json) Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `MistralAI-Index` is not one of them. Documentation [docs.mistral.ai](https://docs.mistral.ai/robots), read 2026-11-04. Note. Indexes for Mistral search. ## Allow and refuse lines ### To allow MistralAI-Index User-agent: MistralAI-Index Allow: / ### To refuse MistralAI-Index User-agent: MistralAI-Index Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Mistral AI in the Verification Ledger IP lists for User and Index; none linked for Training. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) - [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Mistral AI rows - [MistralAI-User](https://agentreadycrawlers.com/crawlers/mistralai-user/) Live answer fetch token MistralAI-User - [MistralAI-Training](https://agentreadycrawlers.com/crawlers/mistralai-training/) Training token MistralAI-Training ## Sources - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-04 - [mistral.ai: mistralai index ips (JSON)](https://mistral.ai/mistralai-index-ips.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "MistralAI-Training: user agent and token | Agent Ready Crawlers" description: "MistralAI-Training from Mistral AI: exact user-agent string, robots.txt token MistralAI-Training, training purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/mistralai-training site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # MistralAI-Training: user agent and token | Agent Ready Crawlers > MistralAI-Training from Mistral AI: exact user-agent string, robots.txt token MistralAI-Training, training purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/mistralai-training Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Mistral AI # MistralAI-Training In short MistralAI-Training is Mistral AI's training crawler. The token to write in robots.txt is MistralAI-Training. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure MistralAI-Training. Collects content to train or tune a model. Refusing it is the training opt-out. MistralAI-Training: webmasters can disallow this user agent in robots.txt. Operator [Mistral AI](https://agentreadycrawlers.com/crawlers/operator/mistral-ai/) Vendor profile: [Mistral AI](https://agentreadycrawlers.com/vendors/mistral-ai/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)` robots.txt token `MistralAI-Training` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Mistral: webmasters can disallow this user agent in robots.txt. [Source](https://docs.mistral.ai/robots), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://docs.mistral.ai/robots), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate Not measured: the crawl checked 31 AI and search tokens, and `MistralAI-Training` is not one of them. Documentation [docs.mistral.ai](https://docs.mistral.ai/robots), read 2026-11-04. Note. No IP list linked for the training crawler on Mistral's page (checked 2026-11-04). ## Allow and refuse lines ### To allow MistralAI-Training User-agent: MistralAI-Training Allow: / ### To refuse MistralAI-Training User-agent: MistralAI-Training Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Mistral AI in the Verification Ledger IP lists for User and Index; none linked for Training. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) - [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Mistral AI rows - [MistralAI-User](https://agentreadycrawlers.com/crawlers/mistralai-user/) Live answer fetch token MistralAI-User - [MistralAI-Index](https://agentreadycrawlers.com/crawlers/mistralai-index/) Search index token MistralAI-Index ## Sources - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "MistralAI-User: user agent, token, block rate | Agent Ready Crawlers" description: "MistralAI-User from Mistral AI: exact user-agent string, robots.txt token MistralAI-User, live answer fetch purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/mistralai-user site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # MistralAI-User: user agent, token, block rate | Agent Ready Crawlers > MistralAI-User from Mistral AI: exact user-agent string, robots.txt token MistralAI-User, live answer fetch purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/mistralai-user Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from Mistral AI # MistralAI-User In short MistralAI-User is Mistral AI's live answer fetcher. The token to write in robots.txt is MistralAI-User. Check a visitor against the operator's published IP list. User actions in Vibe (Le Chat). On robots.txt, Mistral: MistralAI-User governs which sites user requests can be made to; not used for automatic crawling or training. Operator [Mistral AI](https://agentreadycrawlers.com/crawlers/operator/mistral-ai/) Vendor profile: [Mistral AI](https://agentreadycrawlers.com/vendors/mistral-ai/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)` robots.txt token `MistralAI-User` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Mistral: MistralAI-User governs which sites user requests can be made to; not used for automatic crawling or training. [Source](https://docs.mistral.ai/robots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://mistral.ai/mistralai-user-ips.json), read 2026-11-04. IP ranges [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) Rate behaviour User-triggered. Block rate **7.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `MistralAI-User` at the root (459 sites). 3.8% name it; 3.9% refuse it only through a \* group. Rank 28 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.5% (2.6% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [docs.mistral.ai](https://docs.mistral.ai/robots), read 2026-11-04. Note. User actions in Vibe (Le Chat). ## Allow and refuse lines ### To allow MistralAI-User User-agent: MistralAI-User Allow: / ### To refuse MistralAI-User User-agent: MistralAI-User Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Mistral AI in the Verification Ledger IP lists for User and Index; none linked for Training. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) - [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Mistral AI rows - [MistralAI-Index](https://agentreadycrawlers.com/crawlers/mistralai-index/) Search index token MistralAI-Index - [MistralAI-Training](https://agentreadycrawlers.com/crawlers/mistralai-training/) Training token MistralAI-Training ## Sources - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-04 - [mistral.ai: mistralai user ips (JSON)](https://mistral.ai/mistralai-user-ips.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "The names sites write in robots.txt | Agent Ready Crawlers" description: "The user-agent names sites write most often in robots.txt, from 5,938 top-10,000 files, and which have no row in the Crawler Index. Crawl of 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/named-in-robots-txt site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Crawlers" --- # The names sites write in robots.txt | Agent Ready Crawlers > The user-agent names sites write most often in robots.txt, from 5,938 top-10,000 files, and which have no row in the Crawler Index. Crawl of 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/named-in-robots-txt Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The names sites write in robots.txt In short Sites write 3,034 distinct user-agent names in the 5,938 top-10,000 robots.txt files the crawl parsed. gptbot is first at 20.7 percent, and 13 of the 40 most written names have no row in the Crawler Index. No file among them names OraBot. A robots.txt group begins with one or more `User-agent` lines. Counting those names across the crawl shows which agents site owners think about, as opposed to which ones operators document. A name counts once per site, case-folded, and the `*` group is left out. The population is the 5,938 top-10,000 sites with a parsed robots.txt, out of 8,624 parsed in all 13,071 crawled domains. ## The fifteen most written names Share of 5,938 top-10,000 sites that write each name, crawl of 2026-11-04. The table below carries the first 40 names with the all-crawled counts. | Rank | Name | Sites, top 10,000 | Share | Sites, all crawled | In the Index | | --- | --- | --- | --- | --- | --- | | 1 | gptbot | 1,229 | 20.7% | 1,822 | GPTBot | | 2 | claudebot | 1,097 | 18.5% | 1,652 | ClaudeBot | | 3 | ccbot | 1,082 | 18.2% | 1,486 | CCBot | | 4 | google-extended | 1,029 | 17.3% | 1,561 | Google-Extended | | 5 | bytespider | 995 | 16.8% | 1,305 | Bytespider | | 6 | amazonbot | 830 | 14.0% | 1,092 | Amazonbot | | 7 | meta-externalagent | 820 | 13.8% | 1,091 | Meta-ExternalAgent | | 8 | applebot-extended | 815 | 13.7% | 1,196 | Applebot-Extended | | 9 | anthropic-ai | 802 | 13.5% | 1,111 | anthropic-ai legacy row | | 10 | perplexitybot | 757 | 12.7% | 1,269 | PerplexityBot | | 11 | cohere-ai | 745 | 12.5% | 977 | cohere-ai | | 12 | diffbot | 715 | 12.0% | 835 | Diffbot | | 13 | chatgpt-user | 714 | 12.0% | 1,185 | ChatGPT-User | | 14 | claude-web | 665 | 11.2% | 887 | No row | | 15 | omgili | 640 | 10.8% | 728 | omgili | | 16 | petalbot | 608 | 10.2% | 713 | PetalBot | | 17 | oai-searchbot | 590 | 9.9% | 995 | OAI-SearchBot | | 18 | googlebot | 536 | 9.0% | 769 | Googlebot | | 19 | baiduspider | 531 | 8.9% | 630 | No row | | 20 | google-cloudvertexbot | 443 | 7.5% | 514 | Google-CloudVertexBot | | 21 | facebookbot | 440 | 7.4% | 526 | FacebookBot legacy row | | 22 | yandex | 433 | 7.3% | 449 | No row | | 23 | bingbot | 428 | 7.2% | 670 | Bingbot | | 24 | adsbot-google | 424 | 7.1% | 512 | No row | | 25 | claude-user | 412 | 6.9% | 633 | Claude-User | | 26 | twitterbot | 406 | 6.8% | 490 | No row | | 27 | googleother | 398 | 6.7% | 513 | GoogleOther | | 28 | claude-searchbot | 396 | 6.7% | 611 | Claude-SearchBot | | 29 | ahrefsbot | 394 | 6.6% | 463 | No row | | 30 | magpie-crawler | 391 | 6.6% | 454 | No row | | 31 | icc-crawler | 380 | 6.4% | 437 | No row | | 32 | perplexity-user | 379 | 6.4% | 610 | Perplexity-User | | 33 | facebookexternalhit | 378 | 6.4% | 465 | FacebookExternalHit | | 34 | omgilibot | 370 | 6.2% | 403 | No row | | 35 | awariosmartbot | 369 | 6.2% | 428 | No row | | 36 | youbot | 357 | 6.0% | 456 | YouBot | | 37 | awariorssbot | 356 | 6.0% | 415 | No row | | 38 | cotoyogi | 341 | 5.7% | 398 | No row | | 39 | mediapartners-google | 323 | 5.4% | 328 | No row | | 40 | duckassistbot | 319 | 5.4% | 415 | DuckAssistBot | [Download the top 60 names for both populations as CSV](https://agentreadycrawlers.com/data/named-agents-2026-11-04.csv). ## Names with no row in the Index 13 of the 40 names have no row here: claude-web, baiduspider, yandex, adsbot-google, twitterbot, ahrefsbot, magpie-crawler, icc-crawler, omgilibot, awariosmartbot, awariorssbot, cotoyogi and mediapartners-google. The Index covers AI crawlers and agents, so search engines and SEO tools belong outside it. For the rest the gap is ours to close: a name that this many sites write is one more owners are asking about, and a row needs an operator page to be read first, so nothing is added without one. ## OraBot Of the 8,624 parsed files, none write the name `OraBot`, the user agent of [Ora's scanner](https://agentreadycrawlers.com/crawlers/orabot/). A site in the crawl can refuse it only through a \* group. That is expected for a bot whose page, read 2026-11-05, describes scans on demand and names no robots.txt route. ## Reading the counts A written name is a decision by a site owner or a copied list, and the crawl cannot tell which. A name can be written to allow as well as to refuse; the [Block Rate Report](https://agentreadycrawlers.com/block-rates/) has the refusals. The long tail is long: 3,034 distinct names in the top-10,000 files, and the 40 above are the part of it where more than a few percent of sites agree. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [The Crawler Index (this site)](https://agentreadycrawlers.com/crawlers/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "OAI-AdsBot: user agent, token, block rate | Agent Ready Crawlers" description: "OAI-AdsBot from OpenAI: exact user-agent string, robots.txt token OAI-AdsBot, other purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/oai-adsbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # OAI-AdsBot: user agent, token, block rate | Agent Ready Crawlers > OAI-AdsBot from OpenAI: exact user-agent string, robots.txt token OAI-AdsBot, other purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/oai-adsbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from OpenAI # OAI-AdsBot In short OAI-AdsBot is OpenAI's bot, filed under Other. The token to write in robots.txt is OAI-AdsBot. Check a visitor against the operator's published IP list. The crawl of 2026-11-04 did not measure OAI-AdsBot. OpenAI: OAI-AdsBot only visits pages submitted as ads on ChatGPT; OpenAI does not state robots.txt handling for it. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot` robots.txt token `OAI-AdsBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No statement OpenAI: OAI-AdsBot only visits pages submitted as ads on ChatGPT; OpenAI does not state robots.txt handling for it. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://openai.com/adsbot.json), read 2026-11-04. IP ranges [openai.com/adsbot.json](https://openai.com/adsbot.json) Rate behaviour Visits only landing pages submitted as ads, per OpenAI. Block rate Not measured: the crawl checked 31 AI and search tokens, and `OAI-AdsBot` is not one of them. Documentation [platform.openai.com](https://platform.openai.com/docs/bots), read 2026-11-04. Note. Data collected by OAI-AdsBot is not used to train foundation models, per OpenAI. Blocking it affects ad review, not search or training. ## Allow and refuse lines ### To allow OAI-AdsBot User-agent: OAI-AdsBot Allow: / ### To refuse OAI-AdsBot User-agent: OAI-AdsBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) Training token GPTBot - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Search index token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: adsbot (JSON)](https://openai.com/adsbot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "OAI-SearchBot: user agent, token, block rate | Agent Ready Crawlers" description: "OAI-SearchBot from OpenAI: exact user-agent string, robots.txt token OAI-SearchBot, search index purpose and how to verify it. 7.7% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/oai-searchbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # OAI-SearchBot: user agent, token, block rate | Agent Ready Crawlers > OAI-SearchBot from OpenAI: exact user-agent string, robots.txt token OAI-SearchBot, search index purpose and how to verify it. 7.7% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/oai-searchbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from OpenAI # OAI-SearchBot In short OAI-SearchBot is OpenAI's search index crawler. The token to write in robots.txt is OAI-SearchBot. Check a visitor against the operator's published IP list. About 24 hours from a robots.txt change to OpenAI's systems adjusting, per OpenAI. On robots.txt, sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot` robots.txt token `OAI-SearchBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://openai.com/searchbot.json), read 2026-11-04. IP ranges [openai.com/searchbot.json](https://openai.com/searchbot.json) Rate behaviour About 24 hours from a robots.txt change to OpenAI's systems adjusting, per OpenAI. Block rate **7.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `OAI-SearchBot` at the root (458 sites). 4.4% name it; 3.3% refuse it only through a \* group. Rank 26 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.4% (3.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [platform.openai.com](https://platform.openai.com/docs/bots), read 2026-11-04. Note. Robots.txt fetches may carry the extra marker "; robots.txt;" in the user agent string so they can be told apart in logs that omit paths. ## Allow and refuse lines ### To allow OAI-SearchBot User-agent: OAI-SearchBot Allow: / ### To refuse OAI-SearchBot User-agent: OAI-SearchBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) Training token GPTBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: searchbot (JSON)](https://openai.com/searchbot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "omgili: user agent, token, block rate | Agent Ready Crawlers" description: "omgili from Webz.io: exact user-agent string, robots.txt token omgili, other purpose and how to verify it. 15.7% of top sites block it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/omgili site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # omgili: user agent, token, block rate | Agent Ready Crawlers > omgili from Webz.io: exact user-agent string, robots.txt token omgili, other purpose and how to verify it. 15.7% of top sites block it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/omgili Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Bot from Webz.io # omgili (Omgilibot) In short omgili (Omgilibot) is Webz.io's bot, filed under Other. The token to write in robots.txt is omgili. The operator publishes nothing a site can check. 15.7 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Webz.io says its newer crawlers adhere to robots.txt exclusions (2024-07-17 post introducing the "Webzio Duo"). Operator Webz.io Vendor profile: [Webz.io](https://agentreadycrawlers.com/vendors/webz-io/) Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `omgili/0.5 +https://omgili.com` robots.txt token `omgili` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Webz.io says its newer crawlers adhere to robots.txt exclusions (2024-07-17 post introducing the "Webzio Duo"). [Source](https://webz.io/bot.html), read 2026-11-04. Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **15.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `omgili` at the root (933 sites). 11.8% name it; 3.9% refuse it only through a \* group. Rank 8 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 11.8% (9.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [webz.io](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/), read 2026-11-04. Note. Web data vendor whose feeds are sold for AI uses. The 2017 post gives the omgili/0.5 string. ## Allow and refuse lines ### To allow omgili User-agent: omgili Allow: / ### To refuse omgili User-agent: omgili Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Webz.io in the Verification Ledger Publishes nothing for verification. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [webz.io: what is the omgili bot and why is it crawling your website](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/) read 2026-11-04 - [webz.io: bot](https://webz.io/bot.html) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Amazon crawlers and agents | Agent Ready Crawlers" description: "Amazon in the Crawler Index: 5 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/amazon site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Amazon crawlers and agents | Agent Ready Crawlers > Amazon in the Crawler Index: 5 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/amazon Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Amazon: 5 rows In short Amazon has 5 rows in the Index: Amazonbot, Amzn-SearchBot, Amzn-User, Buy for Me and Rufus. They cover 4 purposes: training, search index, live answer fetch, shopping agent. 3 of them have a robots.txt token and 2 publish nothing a site can verify. The most refused is Amazonbot, disallowed at the root by 15.6 percent of top-10,000 sites. Amazon: 5 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Amazon | Amazonbot | Training | Yes | IP list | 15.6% | Profile | | Amazon | Amzn-SearchBot | Search index | Yes | IP list | not measured | Profile | | Amazon | Amzn-User | Live answer fetch | May not apply | IP list | not measured | Profile | | Amazon | Buy for Me (shopping agent) no token | Shopping agent | No docs | None published | no token | Profile | | Amazon | Rufus (shopping assistant) no token | Shopping agent | No docs | None published | no token | Profile | Vendor profile: [Amazon](https://agentreadycrawlers.com/vendors/amazon/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. IP lists - [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/) - [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) - [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Amazon tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Amazonbot | 15.6% | 12.0% | 3.6% | 11.7% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## Sources - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [developer.amazon.com: ip addresses](https://developer.amazon.com/amazonbot/ip-addresses/) read 2026-11-04 - [developer.amazon.com: searchbot ip addresses](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) read 2026-11-04 - [developer.amazon.com: live ip addresses](https://developer.amazon.com/amazonbot/live-ip-addresses/) read 2026-11-04 - [aboutamazon.com: amazon shopping app buy for me brands](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Anthropic crawlers and agents | Agent Ready Crawlers" description: "Anthropic in the Crawler Index: 7 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/anthropic site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Anthropic crawlers and agents | Agent Ready Crawlers > Anthropic in the Crawler Index: 7 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/anthropic Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Anthropic: 7 rows In short Anthropic has 7 rows in the Index: ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Claude in Chrome, Claude Duet cloud browser and Claude computer use. They cover 4 purposes: training, live answer fetch, search index, task agent. 5 of them have a robots.txt token and 4 publish nothing a site can verify. Anthropic: 7 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Anthropic | ClaudeBot | Training | Yes | IP list | 17.6% | Profile | | Anthropic | Claude-User | Live answer fetch | Yes | IP list | 8.1% | Profile | | Anthropic | Claude-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Anthropic | anthropic-ai | Training | Legacy | None published | 15.5% | Profile | | Anthropic | Claude in Chrome no token | Task agent | No statement | None published | no token | Profile | | Anthropic | Claude-Agent | Task agent | Reported | None published | not measured | Profile | | Anthropic | Claude computer use (API tool) no token | Task agent | No docs | None published | no token | Profile | Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Anthropic tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | ClaudeBot | 17.6% | 14.0% | 3.5% | 13.1% | | anthropic-ai | 15.5% | 11.7% | 3.8% | 11.6% | | Claude-User | 8.1% | 4.5% | 3.6% | 5.7% | | Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5.5% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## In the news - 2026-10-21[Anthropic's Claude Duet starts browsing with a Claude-Agent token](https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent/) - 2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 - [claude.com: browser](https://claude.com/docs/third-party/claude-desktop/browser) read 2026-11-04 - [support.claude.com: 12012173 getting started with claude in chrome](https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Apple crawlers and agents | Agent Ready Crawlers" description: "Apple in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. IP list and reverse DNS." canonical: https://agentreadycrawlers.com/crawlers/operator/apple site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Apple crawlers and agents | Agent Ready Crawlers > Apple in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. IP list and reverse DNS. Source: https://agentreadycrawlers.com/crawlers/operator/apple Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Apple: 2 rows In short Apple has 2 rows in the Index: Applebot and Applebot-Extended. They cover 2 purposes: search index, training. 2 of them have a robots.txt token and 1 publish nothing a site can verify. The most refused is Applebot, disallowed at the root by 14.8 percent of top-10,000 sites. IP list and reverse DNS. Apple: 2 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Apple | Applebot | Search index | Yes | Reverse DNS | 14.8% | Profile | | Apple | Applebot-Extended | Training | Yes | None published | 14.8% | Profile | Vendor profile: [Apple](https://agentreadycrawlers.com/vendors/apple/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary IP list and reverse DNS. IP lists - [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json) Reverse DNS \*.applebot.apple.com Key directory publishes nothing Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Apple tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Applebot | 14.8% | 11.6% | 3.1% | 11.1% | | Applebot-Extended | 14.8% | 11.6% | 3.1% | 11.1% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) ## Sources - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Browserbase crawlers and agents | Agent Ready Crawlers" description: "Browserbase in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/browserbase site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Browserbase crawlers and agents | Agent Ready Crawlers > Browserbase in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/browserbase Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Browserbase: 2 rows In short Browserbase has 2 rows in the Index: Browserbase cloud browsers and Stagehand. They cover 1 purpose: task agent. 0 of them have a robots.txt token and 1 publish nothing a site can verify. The crawl measured none of its tokens. Key directory; signing is a customer feature. Browserbase: 2 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Browserbase | Browserbase cloud browsers no token | Task agent | No docs | Web Bot Auth | no token | Profile | | Browserbase | Stagehand (open-source framework) no token | Task agent | No docs | None published | no token | Profile | Vendor profile: [Browserbase](https://agentreadycrawlers.com/vendors/browserbase/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary Key directory; signing is a customer feature. IP lists publishes nothing Reverse DNS publishes nothing Key directory https://www.browserbase.com/.well-known/http-message-signatures-directory Cloudflare Cloudflare signed agent (cohort of 2025-08-28) Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Sources - [docs.browserbase.com: overview](https://docs.browserbase.com/platform/identity/overview) read 2026-11-04 - [browserbase.com: http message signatures directory](https://www.browserbase.com/.well-known/http-message-signatures-directory) read 2026-11-04 - [docs.stagehand.dev: docs.stagehand.dev](https://docs.stagehand.dev/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DuckDuckGo crawlers and agents | Agent Ready Crawlers" description: "DuckDuckGo in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Two IP lists." canonical: https://agentreadycrawlers.com/crawlers/operator/duckduckgo site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # DuckDuckGo crawlers and agents | Agent Ready Crawlers > DuckDuckGo in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Two IP lists. Source: https://agentreadycrawlers.com/crawlers/operator/duckduckgo Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # DuckDuckGo: 2 rows In short DuckDuckGo has 2 rows in the Index: DuckAssistBot and DuckDuckBot. They cover 2 purposes: live answer fetch, search index. 2 of them have a robots.txt token and 0 publish nothing a site can verify. The most refused is DuckAssistBot, disallowed at the root by 8.1 percent of top-10,000 sites. Two IP lists. DuckDuckGo: 2 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | DuckDuckGo | DuckAssistBot | Live answer fetch | Yes | IP list | 8.1% | Profile | | DuckDuckGo | DuckDuckBot | Search index | Yes | IP list | not measured | Profile | Vendor profile: [DuckDuckGo](https://agentreadycrawlers.com/vendors/duckduckgo/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary Two IP lists. IP lists - [duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json) - [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for DuckDuckGo tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | DuckAssistBot | 8.1% | 4.3% | 3.8% | 5.8% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## Sources - [duckduckgo.com: duckassistbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) read 2026-11-04 - [duckduckgo.com: duckassistbot (JSON)](https://duckduckgo.com/duckassistbot.json) read 2026-11-04 - [duckduckgo.com: duckduckbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) read 2026-11-04 - [duckduckgo.com: duckduckbot (JSON)](https://duckduckgo.com/duckduckbot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google crawlers and agents | Agent Ready Crawlers" description: "Google in the Crawler Index: 8 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/google site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Google crawlers and agents | Agent Ready Crawlers > Google in the Crawler Index: 8 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/google Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Google: 8 rows In short Google has 8 rows in the Index: Googlebot, Google-Extended, Google-CloudVertexBot, GoogleOther, Storebot-Google, Google-Agent, Gemini in Chrome and Google-GeminiNotebook. They cover 5 purposes: search index, training, other, task agent, live answer fetch. 5 of them have a robots.txt token and 2 publish nothing a site can verify. Google: 8 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile | | Google | Google-Extended | Training | Yes | None published | 16.0% | Profile | | Google | Google-CloudVertexBot | Other | Yes | Reverse DNS | 10.9% | Profile | | Google | GoogleOther | Other | Yes | Reverse DNS | not measured | Profile | | Google | Storebot-Google | Search index | Yes | Reverse DNS | not measured | Profile | | Google | Google-Agent no token | Task agent | May not apply | Web Bot Auth | no token | Profile | | Google | Gemini in Chrome (auto browse) no token | Task agent | No docs | None published | no token | Profile | | Google | Google-GeminiNotebook (was Google-NotebookLM) no token | Live answer fetch | May not apply | IP list | no token | Profile | Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Google tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Google-Extended | 16.0% | 12.6% | 3.4% | 12.1% | | Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 8.3% | | Googlebot | 2.8% | 0.5% | 2.3% | 2.1% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) - 2026-08-19[Google retires the Google-NotebookLM user agent](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/) ## Sources - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Meta crawlers and agents | Agent Ready Crawlers" description: "Meta in the Crawler Index: 7 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/meta site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Meta crawlers and agents | Agent Ready Crawlers > Meta in the Crawler Index: 7 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/meta Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Meta: 7 rows In short Meta has 7 rows in the Index: Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds, FacebookExternalHit, FacebookBot and Muse. They cover 5 purposes: training, live answer fetch, search index, other, shopping agent. 6 of them have a robots.txt token and 7 publish nothing a site can verify. The most refused is Meta-ExternalAgent, disallowed at the root by 16.0 percent of top-10,000 sites. Meta: 7 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Meta | meta-externalagent | Training | Yes | None published | 16.0% | Profile | | Meta | meta-externalfetcher | Live answer fetch | May not apply | None published | 8.5% | Profile | | Meta | meta-webindexer | Search index | Yes | None published | not measured | Profile | | Meta | meta-externalads | Other | Yes | None published | not measured | Profile | | Meta | facebookexternalhit | Other | May not apply | None published | not measured | Profile | | Meta | FacebookBot | Other | Legacy | None published | 10.2% | Profile | | Meta | Muse (personal agent) no token | Shopping agent | No docs | None published | no token | Profile | Vendor profile: [Meta](https://agentreadycrawlers.com/vendors/meta/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Meta tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 12.0% | | FacebookBot | 10.2% | 6.4% | 3.7% | 7.3% | | Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 6.0% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## In the news - 2026-10-15[DataDome Q3 report: 25 billion AI agent requests](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) - 2026-09-08[Meta launches Muse, a personal agent that does not identify itself](https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i/) ## Sources - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Microsoft crawlers and agents | Agent Ready Crawlers" description: "Microsoft in the Crawler Index: 3 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/microsoft site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Microsoft crawlers and agents | Agent Ready Crawlers > Microsoft in the Crawler Index: 3 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/microsoft Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Microsoft: 3 rows In short Microsoft has 3 rows in the Index: Bingbot, MicrosoftPreview and Browse with Copilot. They cover 3 purposes: search index, other, task agent. 2 of them have a robots.txt token and 1 publish nothing a site can verify. The most refused is MicrosoftPreview, disallowed at the root by 4.0 percent of top-10,000 sites. Microsoft: 3 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Microsoft | bingbot | Search index | Yes | Reverse DNS | 3.1% | Profile | | Microsoft | MicrosoftPreview | Other | Yes | Reverse DNS | 4.0% | Profile | | Microsoft | Browse with Copilot (formerly Copilot Actions) no token | Task agent | No docs | None published | no token | Profile | Vendor profile: [Microsoft](https://agentreadycrawlers.com/vendors/microsoft/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. IP lists - [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json) Reverse DNS Verify Bingbot tool and documented methods Key directory not published Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Microsoft tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | MicrosoftPreview | 4.0% | 0.0% | 3.9% | 2.9% | | Bingbot | 3.1% | 0.4% | 2.7% | 2.2% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) ## Sources - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [bing.com: how to verify bingbot 3905dc26](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) read 2026-11-04 - [verisoul.ai: copilot edge](https://www.verisoul.ai/agents/copilot-edge) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Mistral AI crawlers and agents | Agent Ready Crawlers" description: "Mistral AI in the Crawler Index: 3 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/mistral-ai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Mistral AI crawlers and agents | Agent Ready Crawlers > Mistral AI in the Crawler Index: 3 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/mistral-ai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Mistral AI: 3 rows In short Mistral AI has 3 rows in the Index: MistralAI-User, MistralAI-Index and MistralAI-Training. They cover 3 purposes: live answer fetch, search index, training. 3 of them have a robots.txt token and 1 publish nothing a site can verify. The most refused is MistralAI-User, disallowed at the root by 7.7 percent of top-10,000 sites. Mistral AI: 3 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Mistral AI | MistralAI-User | Live answer fetch | Yes | IP list | 7.7% | Profile | | Mistral AI | MistralAI-Index | Search index | Yes | IP list | not measured | Profile | | Mistral AI | MistralAI-Training | Training | Yes | None published | not measured | Profile | Vendor profile: [Mistral AI](https://agentreadycrawlers.com/vendors/mistral-ai/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary IP lists for User and Index; none linked for Training. IP lists - [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) - [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Mistral AI tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | MistralAI-User | 7.7% | 3.8% | 3.9% | 5.5% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## Sources - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-04 - [mistral.ai: mistralai user ips (JSON)](https://mistral.ai/mistralai-user-ips.json) read 2026-11-04 - [mistral.ai: mistralai index ips (JSON)](https://mistral.ai/mistralai-index-ips.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "OpenAI crawlers and agents | Agent Ready Crawlers" description: "OpenAI in the Crawler Index: 6 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/openai site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # OpenAI crawlers and agents | Agent Ready Crawlers > OpenAI in the Crawler Index: 6 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/openai Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # OpenAI: 6 rows In short OpenAI has 6 rows in the Index: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ChatGPT agent and ChatGPT Atlas. They cover 5 purposes: training, search index, live answer fetch, other, task agent. 4 of them have a robots.txt token and 1 publish nothing a site can verify. The most refused is GPTBot, disallowed at the root by 18.5 percent of top-10,000 sites. OpenAI: 6 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | GPTBot | Training | Yes | IP list | 18.5% | Profile | | OpenAI | OAI-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | OpenAI | ChatGPT-User | Live answer fetch | May not apply | IP list | 9.7% | Profile | | OpenAI | OAI-AdsBot | Other | No statement | IP list | not measured | Profile | | OpenAI | ChatGPT agent (Cloud browser, formerly Operator) no token | Task agent | No statement | Web Bot Auth | no token | Profile | | OpenAI | ChatGPT Atlas (browser agent mode) no token | Task agent | No docs | None published | no token | Profile | Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for OpenAI tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | GPTBot | 18.5% | 15.2% | 3.3% | 13.9% | | ChatGPT-User | 9.7% | 6.3% | 3.3% | 6.9% | | OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5.4% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## In the news - 2026-10-27[OpenAI updates the ChatGPT-User IP list](https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list/) ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: gptbot (JSON)](https://openai.com/gptbot.json) read 2026-11-04 - [openai.com: searchbot (JSON)](https://openai.com/searchbot.json) read 2026-11-04 - [openai.com: chatgpt user (JSON)](https://openai.com/chatgpt-user.json) read 2026-11-04 - [openai.com: adsbot (JSON)](https://openai.com/adsbot.json) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [chatgpt.com: http message signatures directory](https://chatgpt.com/.well-known/http-message-signatures-directory) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Perplexity crawlers and agents | Agent Ready Crawlers" description: "Perplexity in the Crawler Index: 4 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/operator/perplexity site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Perplexity crawlers and agents | Agent Ready Crawlers > Perplexity in the Crawler Index: 4 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/operator/perplexity Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Perplexity: 4 rows In short Perplexity has 4 rows in the Index: PerplexityBot, Perplexity-User, Comet and Perplexity Computer. They cover 3 purposes: search index, live answer fetch, task agent. 2 of them have a robots.txt token and 2 publish nothing a site can verify. The most refused is PerplexityBot, disallowed at the root by 11.2 percent of top-10,000 sites. Perplexity: 4 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Perplexity | PerplexityBot | Search index | Yes | IP list | 11.2% | Profile | | Perplexity | Perplexity-User | Live answer fetch | May not apply | IP list | 8.3% | Profile | | Perplexity | Comet (browser assistant) no token | Task agent | No docs | None published | no token | Profile | | Perplexity | Perplexity Computer no token | Task agent | No docs | None published | no token | Profile | Vendor profile: [Perplexity](https://agentreadycrawlers.com/vendors/perplexity/), with its products, standards, our probes and what it does not do. ## Verification Ledger entry Summary Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. IP lists - [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) - [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa and Mastercard Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04). ## Block rates for Perplexity tokens | Token | Refused, top 10,000 | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | PerplexityBot | 11.2% | 7.9% | 3.4% | 7.9% | | Perplexity-User | 8.3% | 4.6% | 3.7% | 5.9% | Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04. ## Sources - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [perplexity.ai: perplexitybot (JSON)](https://www.perplexity.ai/perplexitybot.json) read 2026-11-04 - [perplexity.ai: perplexity user (JSON)](https://www.perplexity.ai/perplexity-user.json) read 2026-11-04 - [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04 - [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04 - [verisoul.ai: perplexity computer](https://www.verisoul.ai/agents/perplexity-computer) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "OraBot: user agent, token, block rate | Agent Ready Crawlers" description: "OraBot from Ora: exact user-agent string, robots.txt token OraBot, other purpose and how to verify it. Verification: Web Bot Auth. Read 2026-11-05." canonical: https://agentreadycrawlers.com/crawlers/orabot site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Crawlers" --- # OraBot: user agent, token, block rate | Agent Ready Crawlers > OraBot from Ora: exact user-agent string, robots.txt token OraBot, other purpose and how to verify it. Verification: Web Bot Auth. Read 2026-11-05. Source: https://agentreadycrawlers.com/crawlers/orabot Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot from Ora # OraBot (Ora's scanner) In short OraBot is the scanner Ora runs on demand to score a site, filed under Other. Ora says it signs every request with Web Bot Auth, so a site can check a signature against Ora's key directory. Ora's bot page says nothing about how OraBot treats robots.txt, and no site in the crawl names it. Operator Ora Purpose Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. User-agent string `OraBot/1.0 (+https://ora.ai/bot)` robots.txt token `OraBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No statement Ora's bot page does not say whether OraBot reads robots.txt rules before it fetches a page. It says a site owner can ask Ora to exclude a domain from scans, and that a verified owner can turn active probes off in Ora Studio. [Source](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile), read 2026-11-05. Verification Web Bot Auth signature Requests carry an HTTP message signature that can be checked against the operator's published key directory. [Documentation](https://ora.ai/.well-known/http-message-signatures-directory?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile), read 2026-11-05. IP ranges Not published. Rate behaviour On demand, not on a schedule: a scan is a bounded burst of requests with hard timeouts and body-size caps. Ora limits its Personal Agent Protocol probes to one round per target domain every 6 hours and 7 requests per round. Block rate Not measured: the crawl checked 31 AI and search tokens, and `OraBot` is not one of them. Documentation [Ora's scanner bot page](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile), read 2026-11-05. ## Allow and refuse lines The operator documents no robots.txt lines for this token. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity. Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Ora in the Verification Ledger Signs every OraBot request with Web Bot Auth; the key directory answered on 2026-11-05. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory https://ora.ai/.well-known/http-message-signatures-directory Cloudflare not checked Visa TAP registry not applicable: Ora's bot page describes a scanner that does not shop or pay (read 2026-11-05) Mastercard Agent Pay not applicable: Ora's bot page describes a scanner that does not shop or pay (read 2026-11-05) ## What a scan sends and how to check it Signed requests Ora says every request that carries the OraBot user agent is signed with HTTP Message Signatures (RFC 9421, the Web Bot Auth profile) using an Ed25519 key, and carries `Signature`, `Signature-Input` and `Signature-Agent` headers. It tells site owners to treat unsigned traffic on that user agent like any other unverified bot ([Ora's bot page](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile), read 2026-11-05). Key directory [ora.ai/.well-known/http-message-signatures-directory](https://ora.ai/.well-known/http-message-signatures-directory?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile) answered 200 as `application/http-message-signatures-directory+json` on 2026-11-05 and lists one Ed25519 key, valid from 2026-08-27 to 2027-08-27. The directory is the one place a site can check a signature against; the user-agent string proves nothing, since anyone can copy it. Unsigned requests During a scan Ora also sends a handful of requests to the home page that carry other crawlers' user agents, such as GPTBot, ClaudeBot and PerplexityBot, and records how the site answers. Ora says these stay unsigned, because signing them as Ora would contradict the user agent they carry. A site that checks a claimed GPTBot against OpenAI's published addresses will see an address that is not OpenAI's. Personal Agent Protocol reads Every scan requests `/.well-known/poppy.json` on the scanned domain, and on its www or apex sibling when the host has none and its home page resolved to the sibling. Where that file lists them, OraBot also reads the issuer's RFC 8414 metadata and up to three each of other domains' `poppy.json` files, OpenAPI documents and MCP servers' RFC 9728 metadata. Junk-request probes to the endpoints the file lists run only when the scan asks for them. Ora says its scan page and report rescans ask, and that its MCP tools, scheduled jobs and bulk scans never do. Limits Ora states One probe round per target domain every 6 hours, at most 7 requests in a round and a 10 second budget for all of that work in one scan. A scan is on demand, not scheduled. Turning it off Ora says a verified owner can turn active probes off in Ora Studio, and that anyone else can write to the contact address on its bot page, after which Ora stops them for the domain or excludes the domain from scans. The page names no robots.txt route. What the crawl shows None of the 8,624 robots.txt files the crawl parsed names OraBot, so a site in the crawl can refuse it only through a \* group that refuses everything. For scale, 245 of 5,938 top-10,000 sites, 4.1 percent, refuse the crawl's own control token through a \* group alone (crawl of 2026-11-04). The crawl did not measure OraBot as a token of its own. The ora-scanner row in [the Block Rate Report](https://agentreadycrawlers.com/block-rates/) is a control name that is not Ora's user agent, and says nothing about OraBot. ## Sources - [Ora's scanner bot page (vendor)](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile) read 2026-11-05 - [Ora's key directory (vendor)](https://ora.ai/.well-known/http-message-signatures-directory?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=profile) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Comet: user agent, token, block rate | Agent Ready Crawlers" description: "Comet from Perplexity: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/perplexity-comet site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Comet: user agent, token, block rate | Agent Ready Crawlers > Comet from Perplexity: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/perplexity-comet Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Perplexity # Comet (browser assistant) In short Comet (browser assistant) is Perplexity's browsing agent. Comet has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Comet has no operator bot documentation, checked 2026-11-04. Runs on the user's device and IP. Operator [Perplexity](https://agentreadycrawlers.com/crawlers/operator/perplexity/) Vendor profile: [Perplexity](https://agentreadycrawlers.com/vendors/perplexity/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Chrome's user agent from the user's own device; no unique identifier robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) for Comet's agent; it browses as the user's Chromium browser. [Source](https://www.verisoul.ai/agents/perplexity-comet), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.centinelanalytica.com/blog/ai-browser-agents), read 2026-11-04. IP ranges Not published. Rate behaviour Runs on the user's device and IP. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/perplexity-comet). Note. Amazon's 2025-11-04 suit argued Comet sends Chrome's user agent with no unique identifier; the Ninth Circuit vacated the preliminary injunction on 2026-08-04, per Verisoul's summary of the record. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Perplexity in the Verification Ledger Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) - [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Perplexity rows - [PerplexityBot](https://agentreadycrawlers.com/crawlers/perplexitybot/) Search index token PerplexityBot - [Perplexity-User](https://agentreadycrawlers.com/crawlers/perplexity-user/) Live answer fetch token Perplexity-User - [Perplexity Computer](https://agentreadycrawlers.com/crawlers/perplexity-computer/) Task agent no token ## Sources - [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04 - [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Perplexity Computer: user agent and token | Agent Ready Crawlers" description: "Perplexity Computer from Perplexity: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published." canonical: https://agentreadycrawlers.com/crawlers/perplexity-computer site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Perplexity Computer: user agent and token | Agent Ready Crawlers > Perplexity Computer from Perplexity: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Source: https://agentreadycrawlers.com/crawlers/perplexity-computer Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Perplexity # Perplexity Computer In short Perplexity Computer is Perplexity's browsing agent. Perplexity Computer has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Verisoul reports it works in a cloud browser with no declared identity and now also inside users' own Chrome tabs. Operator [Perplexity](https://agentreadycrawlers.com/crawlers/operator/perplexity/) Vendor profile: [Perplexity](https://agentreadycrawlers.com/vendors/perplexity/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: No declared identity robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). [Source](https://www.verisoul.ai/agents/perplexity-computer), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/perplexity-computer), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/perplexity-computer). Note. Verisoul reports it works in a cloud browser with no declared identity and now also inside users' own Chrome tabs. Secondary source only. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Perplexity in the Verification Ledger Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) - [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Perplexity rows - [PerplexityBot](https://agentreadycrawlers.com/crawlers/perplexitybot/) Search index token PerplexityBot - [Perplexity-User](https://agentreadycrawlers.com/crawlers/perplexity-user/) Live answer fetch token Perplexity-User - [Comet (browser assistant)](https://agentreadycrawlers.com/crawlers/perplexity-comet/) Task agent no token ## Sources - [verisoul.ai: perplexity computer](https://www.verisoul.ai/agents/perplexity-computer) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Perplexity-User: user agent, token, block rate | Agent Ready Crawlers" description: "Perplexity-User from Perplexity: exact user-agent string, robots.txt token Perplexity-User, live answer fetch purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/perplexity-user site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Perplexity-User: user agent, token, block rate | Agent Ready Crawlers > Perplexity-User from Perplexity: exact user-agent string, robots.txt token Perplexity-User, live answer fetch purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/perplexity-user Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Live answer fetcher from Perplexity # Perplexity-User In short Perplexity-User is Perplexity's live answer fetcher. Check a visitor against the operator's published IP list. 8.3 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user). Perplexity-User: since a user requested the fetch, this fetcher generally ignores robots.txt rules. Operator [Perplexity](https://agentreadycrawlers.com/crawlers/operator/perplexity/) Vendor profile: [Perplexity](https://agentreadycrawlers.com/vendors/perplexity/) Purpose Live answer fetch [All live answer fetch rows](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/). Fetches a page because a person asked an assistant about it, one request at a time. User-agent string `Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` robots.txt token `Perplexity-User` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt May not apply (user-triggered) Perplexity: since a user requested the fetch, this fetcher generally ignores robots.txt rules. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://www.perplexity.ai/perplexity-user.json), read 2026-11-04. IP ranges [perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) Rate behaviour User-triggered; no crawl rate published. Vendor rule names - **Cloudflare:** Custom rule: (http.user\_agent contains "Perplexity-User" and ip.src in {ranges}) - **AWS WAF:** IP set plus User-Agent string match Block rate **8.3%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Perplexity-User` at the root (495 sites). 4.6% name it; 3.7% refuse it only through a \* group. Rank 23 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.9% (3.2% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [docs.perplexity.ai](https://docs.perplexity.ai/guides/bots), read 2026-11-04. Note. Robots.txt lines for Perplexity-User are a preference Perplexity says it generally does not apply; enforce at the WAF if needed. ## Allow and refuse lines ### To allow Perplexity-User User-agent: Perplexity-User Allow: / ### To refuse Perplexity-User User-agent: Perplexity-User Disallow: / Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Perplexity in the Verification Ledger Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) - [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Perplexity rows - [PerplexityBot](https://agentreadycrawlers.com/crawlers/perplexitybot/) Search index token PerplexityBot - [Comet (browser assistant)](https://agentreadycrawlers.com/crawlers/perplexity-comet/) Task agent no token - [Perplexity Computer](https://agentreadycrawlers.com/crawlers/perplexity-computer/) Task agent no token ## Sources - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [perplexity.ai: perplexity user (JSON)](https://www.perplexity.ai/perplexity-user.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "PerplexityBot: user agent, token, block rate | Agent Ready Crawlers" description: "PerplexityBot from Perplexity: exact user-agent string, robots.txt token PerplexityBot, search index purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/perplexitybot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # PerplexityBot: user agent, token, block rate | Agent Ready Crawlers > PerplexityBot from Perplexity: exact user-agent string, robots.txt token PerplexityBot, search index purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/perplexitybot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Perplexity # PerplexityBot In short PerplexityBot is Perplexity's search index crawler. The token to write in robots.txt is PerplexityBot. Check a visitor against the operator's published IP list. Changes may take up to 24 hours to reflect, per Perplexity. Perplexity recommends allowing PerplexityBot in robots.txt to appear in results and says it is not used to crawl content for AI foundation models. Operator [Perplexity](https://agentreadycrawlers.com/crawlers/operator/perplexity/) Vendor profile: [Perplexity](https://agentreadycrawlers.com/vendors/perplexity/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` robots.txt token `PerplexityBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Perplexity recommends allowing PerplexityBot in robots.txt to appear in results and says it is not used to crawl content for AI foundation models. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://www.perplexity.ai/perplexitybot.json), read 2026-11-04. IP ranges [perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) Rate behaviour Changes may take up to 24 hours to reflect, per Perplexity. Vendor rule names - **Cloudflare:** Custom rule: (http.user\_agent contains "PerplexityBot" and ip.src in {ranges}), action Skip; Perplexity notes Bot Fight Mode cannot be skipped - **AWS WAF:** IP set plus User-Agent string match, action Allow Block rate **11.2%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `PerplexityBot` at the root (667 sites). 7.9% name it; 3.4% refuse it only through a \* group. Rank 15 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 7.9% (5.5% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [docs.perplexity.ai](https://docs.perplexity.ai/guides/bots), read 2026-11-04. Note. Perplexity publishes WAF recipes for Cloudflare and AWS that pair the user agent with its IP list. ## Allow and refuse lines ### To allow PerplexityBot User-agent: PerplexityBot Allow: / ### To refuse PerplexityBot User-agent: PerplexityBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Perplexity in the Verification Ledger Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) - [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Perplexity rows - [Perplexity-User](https://agentreadycrawlers.com/crawlers/perplexity-user/) Live answer fetch token Perplexity-User - [Comet (browser assistant)](https://agentreadycrawlers.com/crawlers/perplexity-comet/) Task agent no token - [Perplexity Computer](https://agentreadycrawlers.com/crawlers/perplexity-computer/) Task agent no token ## Sources - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [perplexity.ai: perplexitybot (JSON)](https://www.perplexity.ai/perplexitybot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "PetalBot: user agent, token, block rate | Agent Ready Crawlers" description: "PetalBot from Huawei: exact user-agent string, robots.txt token PetalBot, search index purpose and how to verify it. 14.1% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/petalbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # PetalBot: user agent, token, block rate | Agent Ready Crawlers > PetalBot from Huawei: exact user-agent string, robots.txt token PetalBot, search index purpose and how to verify it. 14.1% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/petalbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Huawei # PetalBot In short PetalBot is Huawei's search index crawler. The token to write in robots.txt is PetalBot. Verify it by reverse DNS, then a forward lookup. 14.1 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, Huawei: PetalBot complies with the robots protocol; blocking it removes the site from Petal search. Operator Huawei Vendor profile: [Huawei](https://agentreadycrawlers.com/vendors/huawei/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)` robots.txt token `PetalBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Huawei: PetalBot complies with the robots protocol; blocking it removes the site from Petal search. [Source](https://webmaster.petalsearch.com/site/petalbot), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://webmaster.petalsearch.com/site/petalbot), read 2026-11-04. IP ranges Not published. Rate behaviour Huawei says it adjusts crawl volume by server capacity and site quality. Block rate **14.1%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `PetalBot` at the root (835 sites). 10.3% name it; 3.8% refuse it only through a \* group. Rank 14 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 10.9% (8.1% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot), read 2026-11-04. Note. Reverse DNS in petalsearch.com, then forward-confirm. Also feeds Huawei Assistant and AI Search. ## Allow and refuse lines ### To allow PetalBot User-agent: PetalBot Allow: / ### To refuse PetalBot User-agent: PetalBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Huawei in the Verification Ledger Reverse DNS only. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS petalbot-\*.petalsearch.com Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [webmaster.petalsearch.com: petalbot](https://webmaster.petalsearch.com/site/petalbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Live answer fetch crawlers and agents | Agent Ready Crawlers" description: "Live answer fetch in the Crawler Index: 8 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Live answer fetch crawlers and agents | Agent Ready Crawlers > Live answer fetch in the Crawler Index: 8 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Live answer fetch: 8 rows In short 8 rows in the Index are filed under live answer fetch, from 8 operators. 1 publish nothing a site can verify and 1 have no robots.txt token. Of the 6 tokens the crawl measured, ChatGPT-User is refused by 9.7 percent of top-10,000 sites and MistralAI-User by 7.7 percent. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Fetches a page because a person asked an assistant about it, one request at a time. Live answer fetch: 8 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | ChatGPT-User | Live answer fetch | May not apply | IP list | 9.7% | Profile | | Anthropic | Claude-User | Live answer fetch | Yes | IP list | 8.1% | Profile | | Perplexity | Perplexity-User | Live answer fetch | May not apply | IP list | 8.3% | Profile | | Google | Google-GeminiNotebook (was Google-NotebookLM) no token | Live answer fetch | May not apply | IP list | no token | Profile | | Amazon | Amzn-User | Live answer fetch | May not apply | IP list | not measured | Profile | | Meta | meta-externalfetcher | Live answer fetch | May not apply | None published | 8.5% | Profile | | DuckDuckGo | DuckAssistBot | Live answer fetch | Yes | IP list | 8.1% | Profile | | Mistral AI | MistralAI-User | Live answer fetch | Yes | IP list | 7.7% | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | ChatGPT-User | 9.7% | 6.3% | 3.3% | 6.9% | | Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 6.0% | | Perplexity-User | 8.3% | 4.6% | 3.7% | 5.9% | | Claude-User | 8.1% | 4.5% | 3.6% | 5.7% | | DuckAssistBot | 8.1% | 4.3% | 3.8% | 5.8% | | MistralAI-User | 7.7% | 3.8% | 3.9% | 5.5% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Other crawlers and agents | Agent Ready Crawlers" description: "Other in the Crawler Index: 15 AI crawlers and agents from 12 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/other site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Other crawlers and agents | Agent Ready Crawlers > Other in the Crawler Index: 15 AI crawlers and agents from 12 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/other Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Other: 15 rows In short 15 rows in the Index are filed under other, from 12 operators. 10 publish nothing a site can verify and 0 have no robots.txt token. Of the 9 tokens the crawl measured, Bytespider is refused by 19.2 percent of top-10,000 sites and MicrosoftPreview by 4.0 percent. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. Other: 15 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | OAI-AdsBot | Other | No statement | IP list | not measured | Profile | | Google | Google-CloudVertexBot | Other | Yes | Reverse DNS | 10.9% | Profile | | Google | GoogleOther | Other | Yes | Reverse DNS | not measured | Profile | | Microsoft | MicrosoftPreview | Other | Yes | Reverse DNS | 4.0% | Profile | | Meta | meta-externalads | Other | Yes | None published | not measured | Profile | | Meta | facebookexternalhit | Other | May not apply | None published | not measured | Profile | | Meta | FacebookBot | Other | Legacy | None published | 10.2% | Profile | | ByteDance | Bytespider | Other | No docs | None published | 19.2% | Profile | | Cohere | cohere-ai | Other | No docs | None published | 15.5% | Profile | | Diffbot | Diffbot | Other | Setting | None published | 15.7% | Profile | | Webz.io | omgili | Other | Yes | None published | 15.7% | Profile | | Hive (ImageSift) | ImagesiftBot | Other | Yes | None published | 9.5% | Profile | | Firecrawl | FirecrawlAgent | Other | Yes | None published | not measured | Profile | | Ora | OraBot | Other | No statement | Web Bot Auth | not measured | Profile | | Agent Ready Crawlers | agent-readiness-crawl | Other | Partly | None published | 4.2% | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Bytespider | 19.2% | 15.4% | 3.8% | 15.0% | | Diffbot | 15.7% | 11.9% | 3.8% | 11.8% | | omgili | 15.7% | 11.8% | 3.9% | 11.8% | | cohere-ai | 15.5% | 11.6% | 3.8% | 11.7% | | Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 8.3% | | FacebookBot | 10.2% | 6.4% | 3.7% | 7.3% | | ImagesiftBot | 9.5% | 5.5% | 3.9% | 6.9% | | agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 3.0% | | MicrosoftPreview | 4.0% | 0.0% | 3.9% | 2.9% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Search index crawlers and agents | Agent Ready Crawlers" description: "Search index in the Crawler Index: 15 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/search-index site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Search index crawlers and agents | Agent Ready Crawlers > Search index in the Crawler Index: 15 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/search-index Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Search index: 15 rows In short 15 rows in the Index are filed under search index, from 14 operators. 4 publish nothing a site can verify and 0 have no robots.txt token. Of the 9 tokens the crawl measured, Applebot is refused by 14.8 percent of top-10,000 sites and Googlebot by 2.8 percent. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Builds a search index that sends visitors back. Refusing it removes pages from that product. Search index: 15 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | OAI-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Anthropic | Claude-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Perplexity | PerplexityBot | Search index | Yes | IP list | 11.2% | Profile | | Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile | | Google | Storebot-Google | Search index | Yes | Reverse DNS | not measured | Profile | | Apple | Applebot | Search index | Yes | Reverse DNS | 14.8% | Profile | | Microsoft | bingbot | Search index | Yes | Reverse DNS | 3.1% | Profile | | Amazon | Amzn-SearchBot | Search index | Yes | IP list | not measured | Profile | | Meta | meta-webindexer | Search index | Yes | None published | not measured | Profile | | DuckDuckGo | DuckDuckBot | Search index | Yes | IP list | not measured | Profile | | Mistral AI | MistralAI-Index | Search index | Yes | IP list | not measured | Profile | | You.com | YouBot | Search index | No docs | None published | 10.3% | Profile | | Timpi | Timpibot | Search index | No docs | None published | 9.8% | Profile | | Huawei | PetalBot | Search index | Yes | Reverse DNS | 14.1% | Profile | | Exa | ExaSearchBot | Search index | No docs | None published | not measured | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Applebot | 14.8% | 11.6% | 3.1% | 11.1% | | PetalBot | 14.1% | 10.3% | 3.8% | 10.9% | | PerplexityBot | 11.2% | 7.9% | 3.4% | 7.9% | | YouBot | 10.3% | 6.4% | 3.8% | 7.3% | | Timpibot | 9.8% | 5.8% | 4.0% | 7.0% | | OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5.4% | | Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5.5% | | Bingbot | 3.1% | 0.4% | 2.7% | 2.2% | | Googlebot | 2.8% | 0.5% | 2.3% | 2.1% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Shopping agent crawlers and agents | Agent Ready Crawlers" description: "Shopping agent in the Crawler Index: 3 AI crawlers and agents from 2 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/shopping-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Shopping agent crawlers and agents | Agent Ready Crawlers > Shopping agent in the Crawler Index: 3 AI crawlers and agents from 2 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/shopping-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Shopping agent: 3 rows In short 3 rows in the Index are filed under shopping agent, from 2 operators. 3 publish nothing a site can verify and 3 have no robots.txt token. The crawl measured 0 of these tokens. An agent that compares, selects or buys goods for a person. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) An agent that compares, selects or buys goods for a person. Shopping agent: 3 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | Amazon | Buy for Me (shopping agent) no token | Shopping agent | No docs | None published | no token | Profile | | Amazon | Rufus (shopping assistant) no token | Shopping agent | No docs | None published | no token | Profile | | Meta | Muse (personal agent) no token | Shopping agent | No docs | None published | no token | Profile | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [aboutamazon.com: amazon shopping app buy for me brands](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands) read 2026-11-04 - [verisoul.ai: meta muse](https://www.verisoul.ai/agents/meta-muse) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Task agent crawlers and agents | Agent Ready Crawlers" description: "Task agent in the Crawler Index: 20 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/task-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Task agent crawlers and agents | Agent Ready Crawlers > Task agent in the Crawler Index: 20 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/task-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Task agent: 20 rows In short 20 rows in the Index are filed under task agent, from 14 operators. 12 publish nothing a site can verify and 19 have no robots.txt token. The crawl measured 0 of these tokens. A browsing agent that clicks, fills forms and finishes a job for a person. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) A browsing agent that clicks, fills forms and finishes a job for a person. Task agent: 20 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | ChatGPT agent (Cloud browser, formerly Operator) no token | Task agent | No statement | Web Bot Auth | no token | Profile | | OpenAI | ChatGPT Atlas (browser agent mode) no token | Task agent | No docs | None published | no token | Profile | | Anthropic | Claude in Chrome no token | Task agent | No statement | None published | no token | Profile | | Anthropic | Claude-Agent | Task agent | Reported | None published | not measured | Profile | | Anthropic | Claude computer use (API tool) no token | Task agent | No docs | None published | no token | Profile | | Perplexity | Comet (browser assistant) no token | Task agent | No docs | None published | no token | Profile | | Perplexity | Perplexity Computer no token | Task agent | No docs | None published | no token | Profile | | Google | Google-Agent no token | Task agent | May not apply | Web Bot Auth | no token | Profile | | Google | Gemini in Chrome (auto browse) no token | Task agent | No docs | None published | no token | Profile | | Microsoft | Browse with Copilot (formerly Copilot Actions) no token | Task agent | No docs | None published | no token | Profile | | Amazon Web Services | Amazon Bedrock AgentCore Browser no token | Task agent | No statement | Web Bot Auth | no token | Profile | | Browser Use | Browser Use (library and cloud) no token | Task agent | No docs | None published | no token | Profile | | Apify | Apify Website Content Crawler no token | Task agent | Setting | Web Bot Auth | no token | Profile | | Browserbase | Browserbase cloud browsers no token | Task agent | No docs | Web Bot Auth | no token | Profile | | Browserbase | Stagehand (open-source framework) no token | Task agent | No docs | None published | no token | Profile | | Cloudflare | Cloudflare Browser Rendering no token | Task agent | No statement | Web Bot Auth | no token | Profile | | Anchor | Anchor Browser no token | Task agent | No docs | Web Bot Auth | no token | Profile | | Block | Goose (open-source agent) no token | Task agent | No docs | Cloudflare signed | no token | Profile | | xAI | Grok Bot (agent) no token | Task agent | No docs | None published | no token | Profile | | Manus | Manus cloud browser no token | Task agent | No docs | None published | no token | Profile | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [verisoul.ai: chatgpt atlas](https://www.verisoul.ai/agents/chatgpt-atlas) read 2026-11-04 - [claude.com: browser](https://claude.com/docs/third-party/claude-desktop/browser) read 2026-11-04 - [verisoul.ai: claude duet](https://www.verisoul.ai/agents/claude-duet) read 2026-11-04 - [docs.anthropic.com: computer use tool](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool) read 2026-11-04 - [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Training crawlers and agents | Agent Ready Crawlers" description: "Training in the Crawler Index: 9 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/training site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Training crawlers and agents | Agent Ready Crawlers > Training in the Crawler Index: 9 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/training Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Training: 9 rows In short 9 rows in the Index are filed under training, from 8 operators. 5 publish nothing a site can verify and 0 have no robots.txt token. Of the 8 tokens the crawl measured, CCBot is refused by 19.4 percent of top-10,000 sites and Applebot-Extended by 14.8 percent. Collects content to train or tune a model. Refusing it is the training opt-out. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Collects content to train or tune a model. Refusing it is the training opt-out. Training: 9 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | GPTBot | Training | Yes | IP list | 18.5% | Profile | | Anthropic | ClaudeBot | Training | Yes | IP list | 17.6% | Profile | | Anthropic | anthropic-ai | Training | Legacy | None published | 15.5% | Profile | | Google | Google-Extended | Training | Yes | None published | 16.0% | Profile | | Apple | Applebot-Extended | Training | Yes | None published | 14.8% | Profile | | Amazon | Amazonbot | Training | Yes | IP list | 15.6% | Profile | | Meta | meta-externalagent | Training | Yes | None published | 16.0% | Profile | | Common Crawl | CCBot | Training | Yes | Reverse DNS | 19.4% | Profile | | Mistral AI | MistralAI-Training | Training | Yes | None published | not measured | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | CCBot | 19.4% | 15.7% | 3.7% | 14.9% | | GPTBot | 18.5% | 15.2% | 3.3% | 13.9% | | ClaudeBot | 17.6% | 14.0% | 3.5% | 13.1% | | Google-Extended | 16.0% | 12.6% | 3.4% | 12.1% | | Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 12.0% | | Amazonbot | 15.6% | 12.0% | 3.6% | 11.7% | | anthropic-ai | 15.5% | 11.7% | 3.8% | 11.6% | | Applebot-Extended | 14.8% | 11.6% | 3.1% | 11.1% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Stagehand: user agent, token, block rate | Agent Ready Crawlers" description: "Stagehand from Browserbase: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/stagehand site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Stagehand: user agent, token, block rate | Agent Ready Crawlers > Stagehand from Browserbase: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/stagehand Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Browsing agent from Browserbase # Stagehand (open-source framework) In short Stagehand (open-source framework) is Browserbase's browsing agent. Stagehand has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. The operator publishes nothing a site can check. Stagehand has no operator bot documentation, checked 2026-11-04. Identity comes from the hosting browser, not the framework. Operator [Browserbase](https://agentreadycrawlers.com/crawlers/operator/browserbase/) Vendor profile: [Browserbase](https://agentreadycrawlers.com/vendors/browserbase/) Purpose Task agent [All task agent rows](https://agentreadycrawlers.com/crawlers/purpose/task-agent/). A browsing agent that clicks, fills forms and finishes a job for a person. User-agent string No exact string is published. What the operator or a third party says: Inherits the browser it runs in; signed only when run on Browserbase with Web Bot Auth robots.txt token None. No robots.txt line can address this agent. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04) for Stagehand itself. [Source](https://docs.stagehand.dev/), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://docs.stagehand.dev/), read 2026-11-04. IP ranges Not published. Rate behaviour Set by the developer. Block rate None: the agent has no token, so a robots.txt block rate does not exist. Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://docs.stagehand.dev/). Note. Identity comes from the hosting browser, not the framework. ## Allow and refuse lines This agent has no robots.txt token, so there are no lines to copy. Without a token, a published list or a signature, a site owner cannot tell it from an ordinary visitor by request alone. Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF. ## Browserbase in the Verification Ledger Key directory; signing is a customer feature. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory https://www.browserbase.com/.well-known/http-message-signatures-directory Cloudflare Cloudflare signed agent (cohort of 2025-08-28) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Browserbase rows - [Browserbase cloud browsers](https://agentreadycrawlers.com/crawlers/browserbase/) Task agent no token ## Sources - [docs.stagehand.dev: docs.stagehand.dev](https://docs.stagehand.dev/) read 2026-11-04 - [docs.browserbase.com: overview](https://docs.browserbase.com/platform/identity/overview) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Storebot-Google: user agent, token, block rate | Agent Ready Crawlers" description: "Storebot-Google from Google: exact user-agent string, robots.txt token Storebot-Google, search index purpose and how to verify it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/storebot-google site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Storebot-Google: user agent, token, block rate | Agent Ready Crawlers > Storebot-Google from Google: exact user-agent string, robots.txt token Storebot-Google, search index purpose and how to verify it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/storebot-google Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Google # Storebot-Google In short Storebot-Google is Google's search index crawler. The token to write in robots.txt is Storebot-Google. Verify it by reverse DNS, then a forward lookup. The crawl of 2026-11-04 did not measure Storebot-Google. Relevant to commerce sites: blocking it removes Shopping surfaces. Its user-agent entry: Mozilla/5.0 (X11; Linux x86\_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36. Storebot-Google obeys robots.txt, per Google. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36` robots.txt token `Storebot-Google` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Common crawler: obeys robots.txt, per Google. Preferences affect all surfaces of Google Shopping. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) Rate behaviour Not published separately. Block rate Not measured: the crawl checked 31 AI and search tokens, and `Storebot-Google` is not one of them. Documentation [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Note. Relevant to commerce sites: blocking it removes Shopping surfaces. ## Allow and refuse lines ### To allow Storebot-Google User-agent: Storebot-Google Allow: / ### To refuse Storebot-Google User-agent: Storebot-Google Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other Google rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Search index token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [developers.google.com: googlebot (JSON)](https://developers.google.com/static/search/apis/ipranges/googlebot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Timpibot: user agent, token, block rate | Agent Ready Crawlers" description: "Timpibot from Timpi: robots.txt token Timpibot, search index purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/timpibot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Timpibot: user agent, token, block rate | Agent Ready Crawlers > Timpibot from Timpi: robots.txt token Timpibot, search index purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/timpibot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Timpi # Timpibot In short Timpibot is Timpi's search index crawler. The token to write in robots.txt is Timpibot. The operator publishes nothing a site can check. 9.8 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Timpibot's operator publishes no robots.txt statement; we checked on 2026-11-04. Operator Timpi Vendor profile: [Timpi](https://agentreadycrawlers.com/vendors/timpi/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string No exact string is published. What the operator or a third party says: Not published robots.txt token `Timpibot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). timpi.io/crawler and /timpibot returned 404 on 2026-11-04. [Source](https://timpi.io/), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://timpi.io/), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **9.8%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Timpibot` at the root (579 sites). 5.8% name it; 4.0% refuse it only through a \* group. Rank 19 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 7.0% (4.2% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://timpi.io/). Note. Decentralised search index operator. ## Allow and refuse lines ### To allow Timpibot User-agent: Timpibot Allow: / ### To refuse Timpibot User-agent: Timpibot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Timpi in the Verification Ledger Publishes nothing. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [timpi.io: timpi.io](https://timpi.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "YouBot: user agent, token, block rate | Agent Ready Crawlers" description: "YouBot from You.com: robots.txt token YouBot, search index purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/youbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # YouBot: user agent, token, block rate | Agent Ready Crawlers > YouBot from You.com: robots.txt token YouBot, search index purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/youbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from You.com # YouBot In short YouBot is You.com's search index crawler. The token to write in robots.txt is YouBot. The operator publishes nothing a site can check. 10.3 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Verisoul reports YouBot signs its requests, but You.com publishes no key directory or documentation we could read. Operator You.com Vendor profile: [You.com](https://agentreadycrawlers.com/vendors/you-com/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string No exact string is published. What the operator or a third party says: Not published by You.com robots.txt token `YouBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt No documentation operator publishes no bot documentation (checked 2026-11-04). you.com/youbot redirects to a sign-in page. [Source](https://technologychecker.io/blog/ai-crawler-list), read 2026-11-04. Operator publishes no bot documentation (checked 2026-11-04). Verification Nothing published The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.verisoul.ai/agents/youbot), read 2026-11-04. IP ranges Not published. Rate behaviour Not published. Block rate **10.3%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `YouBot` at the root (610 sites). 6.4% name it; 3.8% refuse it only through a \* group. Rank 17 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 7.3% (4.6% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.verisoul.ai/agents/youbot). Note. Verisoul reports YouBot signs its requests, but You.com publishes no key directory or documentation we could read. ## Allow and refuse lines ### To allow YouBot User-agent: YouBot Allow: / ### To refuse YouBot User-agent: YouBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## You.com in the Verification Ledger Publishes nothing we could read; a detection vendor reports signed requests. [All operators](https://agentreadycrawlers.com/verification/). IP lists publishes nothing Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Sources - [verisoul.ai: youbot](https://www.verisoul.ai/agents/youbot) read 2026-11-04 - [technologychecker.io: ai crawler list](https://technologychecker.io/blog/ai-crawler-list) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Data downloads: CSV and JSON | Agent Ready Crawlers" description: "Download the Crawler Index, the Block Rate Report, the Verification Ledger and two crawl slices as CSV and JSON. 13,071 crawl rows." canonical: https://agentreadycrawlers.com/data site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Data" --- # Data downloads: CSV and JSON | Agent Ready Crawlers > Download the Crawler Index, the Block Rate Report, the Verification Ledger and two crawl slices as CSV and JSON. 13,071 crawl rows. Source: https://agentreadycrawlers.com/data Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Data downloads In short Five datasets, each as CSV and some as JSON: the Crawler Index with 70 rows, the Block Rate Report from a crawl of 13,071 domains, the Verification Ledger with 33 operators, and two slices of the same crawl. Every row carries a source URL and the date read. Edition 2026-11-04. ## Crawler Index 70 AI crawlers and agents: operator, user-agent string, robots.txt token, purpose, operator statement, verification, allow and refuse lines, block rate. - [crawler-index-2026-11-04.csv](https://agentreadycrawlers.com/data/crawler-index-2026-11-04.csv) text/csv, about 47 KB - [crawler-index-2026-11-04.json](https://agentreadycrawlers.com/data/crawler-index-2026-11-04.json) application/json, about 91 KB **Cite this.** Agent Ready Crawlers, "Crawler Index", edition 2026-11-04, https://agentreadycrawlers.com/data/. ## Block Rate Report Per-token robots.txt block rates by site list, rank band and Ora category, the 401 and 403 share of home pages and bot-management header prevalence, from 13,071 crawled domains. - [block-rates-2026-11-04.csv](https://agentreadycrawlers.com/data/block-rates-2026-11-04.csv) text/csv, about 62 KB - [block-rates-2026-11-04.json](https://agentreadycrawlers.com/data/block-rates-2026-11-04.json) application/json, about 203 KB - [home-status-2026-11-04.csv](https://agentreadycrawlers.com/data/home-status-2026-11-04.csv) text/csv, about 1 KB - [bot-signals-2026-11-04.csv](https://agentreadycrawlers.com/data/bot-signals-2026-11-04.csv) text/csv, about 1 KB **Cite this.** Agent Ready Crawlers, "Block Rate Report", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/. Crawl of 13,071 rows, read 2026-11-04. ## Verification Ledger 33 operators and programmes: IP lists, reverse DNS, Web Bot Auth key directories, Cloudflare status and card-network registries, each with the URL and the date read. - [verification-ledger-2026-11-04.csv](https://agentreadycrawlers.com/data/verification-ledger-2026-11-04.csv) text/csv, about 14 KB - [verification-ledger-2026-11-04.json](https://agentreadycrawlers.com/data/verification-ledger-2026-11-04.json) application/json, about 22 KB **Cite this.** Agent Ready Crawlers, "Verification Ledger", edition 2026-11-04, https://agentreadycrawlers.com/verification/. ## Names written in robots.txt The most written user-agent names in the parsed robots.txt files of the crawl, top 10,000 and all crawled, with the share of sites that write each. - [named-agents-2026-11-04.csv](https://agentreadycrawlers.com/data/named-agents-2026-11-04.csv) text/csv, about 6 KB **Cite this.** Agent Ready Crawlers, "Names written in robots.txt", edition 2026-11-04, https://agentreadycrawlers.com/crawlers/named-in-robots-txt/. ## Training and search policy mix How parsed robots.txt files combine refusals of the six training tokens and the three AI search tokens, top 10,000 and all crawled. - [policy-mix-2026-11-04.csv](https://agentreadycrawlers.com/data/policy-mix-2026-11-04.csv) text/csv, about 2 KB **Cite this.** Agent Ready Crawlers, "Training and search policy mix", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/training-and-search/. ## Terms of use The data is free to use with the cite-this line. Operator statements are quoted from the operators' own pages and keep their source URL. See [the terms](https://agentreadycrawlers.com/terms/) and [the corrections log](https://agentreadycrawlers.com/corrections/). ## Sources - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "API and MCP server | Agent Ready Crawlers" description: "How to query the crawler data: the MCP server at /mcp, the REST API described by openapi.json and the NLWeb ask endpoint, all read-only and open." canonical: https://agentreadycrawlers.com/developers site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Developers" --- # API and MCP server | Agent Ready Crawlers > How to query the crawler data: the MCP server at /mcp, the REST API described by openapi.json and the NLWeb ask endpoint, all read-only and open. Source: https://agentreadycrawlers.com/developers Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # API and MCP server In short The Crawler Index, the Block Rate Report and the Verification Ledger are also data an agent can query: an MCP server at /mcp with 5 read-only tools, a REST API under /api/v1 described by openapi.json, and an NLWeb /ask endpoint. No credential, 120 requests a minute, edition 2026-11-04. ## Connect the MCP server The server speaks the Streamable HTTP transport at `https://agentreadycrawlers.com/mcp`. Send JSON-RPC 2.0 messages with POST and read the JSON answer. It keeps no session and has no stream the server starts, so one request gets one answer. A client that accepts a URL connects with that address alone; the card at [/mcp/server-card](https://agentreadycrawlers.com/mcp/server-card) and the manifest at [/.well-known/mcp.json](https://agentreadycrawlers.com/.well-known/mcp.json) carry the same facts in the forms clients read. ```bash curl -s https://agentreadycrawlers.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"my-agent","version":"1.0"}}}' ``` ```bash curl -s https://agentreadycrawlers.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_robots_lines","arguments":{"tokens":["GPTBot","ClaudeBot"],"action":"refuse"}}}' ``` A tool answers with its result as `structuredContent` and the same JSON as text. A wrong argument answers `isError: true` with the parameter list, and an unknown tool answers JSON-RPC error -32602. `resources/list` names the page index, the methodology, this page and the dataset files, which `resources/read` returns. ## The tools | Tool | What it returns | Parameters | | --- | --- | --- | | list_crawlers | Lists rows of the Crawler Index (id, name, operator, robots.txt token, purpose, how the operator says it treats robots.txt, verification method, block rates, profile URL). Filter by operator, purpose, verification method, robots.txt reading or words. Call it to find a crawler id before get_crawler, or to answer which crawlers do X. | operator?: Operator name as in the index, case-insensitive, for example OpenAI, Anthropic, Google or Perplexity. purpose?: training, search-index, live-answer-fetch, task-agent, shopping-agent or other. verification_method?: How the operator lets a site verify a request: ip-json, reverse-dns, web-bot-auth, cloudflare-verified or none-published. respects_robots?: The reading of the operator statement on robots.txt, for example "Yes, per operator", "May not apply" or "No documentation". q?: Words to look for in the id, name, operator, token or user-agent string, for example "agent" or "Mozilla". sort?: block_rate_top10000 or block_rate_all (highest first, rows without a rate last) or name. Default is the order of the Crawler Index. limit?: Page size, 1 to 100 (default 20). cursor?: Opaque cursor from next_cursor of the previous page. | | get_crawler | Returns one Crawler Index row by id, robots.txt token or name (for example gptbot, GPTBot or ClaudeBot): operator, exact user-agent string, purpose, the operator statement on robots.txt with its source URL, verification method and IP list URL, rate behaviour, the allow and refuse lines, the vendor rule names, both block rates and the profile page URL to cite. | id: Crawler id (gptbot), robots.txt token (GPTBot) or name, case-insensitive. | | get_block_rate | Returns how many sites in the crawl refuse one robots.txt token (for example GPTBot or ClaudeBot) at the path /: the parsed-site count, the refused count, the part where a robots.txt group names the token, the shares, and the 401 and 403 share of home pages in the same cut. Without stratum it returns all four site lists; with category it returns that category. | token: A robots.txt token such as GPTBot, or a crawler id such as gptbot, case-insensitive. stratum?: all (every crawled domain with a parsed robots.txt), tranco (top 10,000), tranco_1_1000 or tranco_1001_10000. Omit for all four. category?: One of the report's categories, for example "Developer Tools" or "Healthcare"; only categories with at least 30 parsed sites are published. Cannot be combined with stratum. | | get_verification | Returns one Verification Ledger row: the IP list URLs, the reverse DNS pattern, the Web Bot Auth key directory, the Cloudflare verified status, the card-network registry lines, a summary, the source URL and the date read, plus the operator's crawlers with their verification method. Call it to check a request is really from an operator. | operator: Operator name as in the ledger (OpenAI, Anthropic, Google, Perplexity), or the id or robots.txt token of one of its crawlers, case-insensitive. | | get_robots_lines | Returns the allow lines or the refuse lines the Crawler Index holds for up to 20 crawlers (robots.txt tokens or crawler ids), per crawler and joined into one robots.txt body in the order asked. It only joins the rows' own lines; crawlers without a robots.txt token are listed as unmatched. Call it when writing a robots.txt policy for named crawlers. | tokens: robots.txt tokens or crawler ids, for example ["GPTBot", "ClaudeBot"]; in a query string, a comma-separated list. 1 to 20 entries. action: allow returns the Allow lines, refuse returns the Disallow lines. | A question mark after a parameter name marks it optional. ## The REST API Each tool is also one GET endpoint under `/api/v1/`, with the parameters in the query string and the same JSON in the answer. [/api](https://agentreadycrawlers.com/api) is the index. [/openapi.json](https://agentreadycrawlers.com/openapi.json) describes every operation with its parameters, response schema and the Error schema, and [/.well-known/api-catalog](https://agentreadycrawlers.com/.well-known/api-catalog) links that description the way RFC 9727 says. List endpoints take `limit` (1 to 100) and the `next_cursor` of the previous page as `cursor`. A list parameter is a comma-separated value. - [/api/v1/list\_crawlers?purpose=training](https://agentreadycrawlers.com/api/v1/list_crawlers?purpose=training) List crawlers and agents. Paginated. - [/api/v1/get\_crawler?id=gptbot](https://agentreadycrawlers.com/api/v1/get_crawler?id=gptbot) One crawler in full. - [/api/v1/get\_block\_rate?token=GPTBot](https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot) Block rate of one robots.txt token. - [/api/v1/get\_verification?operator=OpenAI](https://agentreadycrawlers.com/api/v1/get_verification?operator=OpenAI) How an operator lets you verify its requests. - [/api/v1/get\_robots\_lines?tokens=GPTBot,ClaudeBot&action=refuse](https://agentreadycrawlers.com/api/v1/get_robots_lines?tokens=GPTBot,ClaudeBot&action=refuse) The robots.txt lines for a set of crawlers. - [/api](https://agentreadycrawlers.com/api) The index: endpoints, datasets, limits and the versioning policy. ```bash curl -s 'https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot&stratum=tranco' | jq '.rates[0].pct_effective' ``` ## Ask in plain words The `/ask` endpoint follows NLWeb 0.55. It searches the crawler rows, the block-rate tokens, the ledger rows and every page, and returns schema.org objects that each carry a `grounding.source` URL to cite. GET takes `query`; POST takes the NLWeb request object. Add `streaming=true`, or `prefer.streaming`, for server-sent events named start, result and complete. `mode=summarize` adds one SearchSummary item assembled from the results by template, with no model involved. ```bash curl -s 'https://agentreadycrawlers.com/ask?query=ClaudeBot+block+rate' ``` ```bash curl -s https://agentreadycrawlers.com/ask -H 'Content-Type: application/json' -d '{"query":{"text":"which operators publish a Web Bot Auth key directory"},"prefer":{"mode":"list"},"meta":{"version":"0.55"}}' ``` A query with no match answers `_meta.response_type: failure` with code NO\_RESULTS, and a missing query answers 400 with INVALID\_QUERY. Context objects are accepted and ignored, because every answer comes from the data and not from a conversation. ## Discovery files - [/openapi.json](https://agentreadycrawlers.com/openapi.json) OpenAPI 3.1 for the REST API and /ask, linked from every page as rel=service-desc and from the Link header of the home page. - [/.well-known/api-catalog](https://agentreadycrawlers.com/.well-known/api-catalog) RFC 9727 linkset: the API, the MCP server and the ask endpoint with their descriptions. - [/mcp/server-card](https://agentreadycrawlers.com/mcp/server-card) The MCP server card (name, version, remote URL, protocol versions), also at /.well-known/mcp/server-card.json. - [/.well-known/mcp.json](https://agentreadycrawlers.com/.well-known/mcp.json) The manifest: endpoint, transport and the tools with their input schemas. - [/.well-known/ai-catalog.json](https://agentreadycrawlers.com/.well-known/ai-catalog.json) AI Catalog 1.0 with entries for the MCP card, the OpenAPI document, the ask endpoint and every document; the same file is at /.well-known/ard.json. - [/schemamap.xml](https://agentreadycrawlers.com/schemamap.xml) Schema feeds: the rows as JSON-LD lines under /feeds/, named in robots.txt. ## Access, limits and errors Credential None. The OpenAPI document declares `security: []` and no security schemes, and [auth.md](https://agentreadycrawlers.com/auth.md) says the same. The site publishes no OAuth metadata, because no endpoint takes a token and metadata for an authorization server that does not exist would point agents at nothing. Rate limit 120 requests a minute per client address, counted per running instance and announced in `RateLimit` and `RateLimit-Policy` headers. A 429 carries `Retry-After`, and the host may add limits of its own above that. Each dataset is one download from [/data/](https://agentreadycrawlers.com/data/), so a bulk reader never needs the API. Errors JSON and never HTML: `{"error":{"code","message","hint","status","docs"}}` with the codes invalid\_params, not\_found, method\_not\_allowed, unsupported\_media\_type, rate\_limited and internal. A missing endpoint under /api answers 404 and lists the endpoints in the hint. Writes There are none, so every call is safe to retry and no idempotency key exists. Every call finishes in one response; nothing runs long enough to need a job pattern. CORS Every answer carries `Access-Control-Allow-Origin: *`, so an agent that runs in a browser can call the API and the MCP server. ## Versioning The path prefix `/api/v1` is the contract. A breaking change, such as a field removed or renamed or a tool removed, ships under `/api/v2` while `/api/v1` keeps answering for at least 180 days, announced on this page and in the `info.version` of openapi.json. Additive changes, such as a new field or tool, keep the prefix. The MCP server follows the same rule, and its card and `serverInfo.version` carry the version number of the OpenAPI document. The data behind every answer is edition 2026-11-04, named in the `X-Dataset-Edition` header and in each tool result. ## Cite the data Every result names the page it comes from; cite that URL. The data files are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) and the cite-this line for each dataset is on the [data page](https://agentreadycrawlers.com/data/). ## What the surface does not do It does not write, hold accounts, generate text with a model or search beyond this site's own rows and pages. A block rate is a count from one crawl and a robots.txt line is a statement of preference, so the API reports both as the pages do. The crawler pages stay the record; the [method page](https://agentreadycrawlers.com/method/) explains how the rows are made. ## Sources - [MCP specification, Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) read 2026-11-04 - [NLWeb specification v0.55](https://github.com/nlweb-ai/website/blob/main/NLWEBSPEC.md) read 2026-11-04 - [RFC 9727, api-catalog](https://www.rfc-editor.org/rfc/rfc9727) read 2026-11-04 - [OpenAPI 3.1.0](https://spec.openapis.org/oas/v3.1.0) read 2026-11-04 - [Schema Feeds specification 0.1](https://github.com/nlweb-ai/website/blob/main/SCHEMA_SPEC.md) read 2026-11-04 - [Data downloads (this site)](https://agentreadycrawlers.com/data/) read 2026-11-04 - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Editorial policy | Agent Ready Crawlers" description: "Editorial policy: how sources are chosen and dated, how vendor claims and Ora's findings are labelled, how grades change and how corrections are logged." canonical: https://agentreadycrawlers.com/editorial-policy site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Publication" --- # Editorial policy | Agent Ready Crawlers > Editorial policy: how sources are chosen and dated, how vendor claims and Ora's findings are labelled, how grades change and how corrections are logged. Source: https://agentreadycrawlers.com/editorial-policy Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Editorial policy In short Every fact on this site carries a source URL and the date it was read. Operators' own pages come first, third-party detection reports are labelled secondary, and a vendor's description of itself is labelled a vendor claim. Grades follow the published rubric and change only when the evidence changes. Errors are corrected in place and logged. ## Sources The order of preference is the operator's own documentation, then a standard or a repository, then a detection vendor or a news report. Third-party reports are marked secondary. A fact with no source is not published; where an operator publishes nothing, the row says so with the date we checked. ## Dates Every page shows the date its facts were read. An edition is dated the day its crawl or its reading was done and is not backdated. Prices and scores carry the date read. ## Claims and findings What a vendor says about itself is a vendor claim and is labelled. Findings from a vendor's lab, Ora's included, are written as "Ora's lab found (date)", never as a general result. We have not tested the products we grade. ## Grades and rank The rubric is on [the method page](https://agentreadycrawlers.com/method/#vendor-rubric). A grade changes when the public documentation it rests on changes, and the new URL and date replace the old. Equal points share a rank. ## Writing Plain and specific, in the register of an operator who reads server logs. No marketing adjectives, no unsourced superlatives, and no question headings; the FAQ asks its questions as questions. ## Corrections We correct the page and add a line to [the corrections log](https://agentreadycrawlers.com/corrections/) with the date and what changed. Report an error to [editor@agentreadycrawlers.com](mailto:editor@agentreadycrawlers.com) with the page and a source. ## Sources - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Corrections log (this site)](https://agentreadycrawlers.com/corrections/) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "FAQ: AI crawlers and agents | Agent Ready Crawlers" description: "33 questions on AI crawlers and agents: ClaudeBot, GPTBot, tokens, verification, pay per crawl and bot management, each answered with a source." canonical: https://agentreadycrawlers.com/faq site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "FAQ" --- # FAQ: AI crawlers and agents | Agent Ready Crawlers > 33 questions on AI crawlers and agents: ClaudeBot, GPTBot, tokens, verification, pay per crawl and bot management, each answered with a source. Source: https://agentreadycrawlers.com/faq Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Questions and answers In short 33 questions about AI crawlers and agents, answered in two to four sentences each: ClaudeBot, GPTBot, robots.txt tokens, verification, pay per crawl, Content-Signals and bot management. Every answer links its source and the date it was read, 2026-11-04, and none is longer than a short paragraph. ## All 33 answers ### What is ClaudeBot? ClaudeBot is Anthropic's training crawler. Restricting it in robots.txt signals that future content should be excluded from Anthropic training datasets. Anthropic runs two other bots: Claude-User for user-requested fetches and Claude-SearchBot for search indexing. Source addresses are listed at claude.com/crawling/bots.json. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. ### How do I block GPTBot without leaving ChatGPT search? Add "User-agent: GPTBot" then "Disallow: /" and keep OAI-SearchBot allowed. OpenAI treats the two settings as independent, and says robots.txt changes take about 24 hours to reach its systems. ChatGPT-User is a separate user-triggered fetcher that may not apply robots.txt. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. ### Does ChatGPT-User obey robots.txt? Not necessarily. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and it is not used to decide search inclusion. To refuse it you need a WAF rule that matches its user agent and the ranges in chatgpt-user.json. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. ### Which AI fetchers say they ignore robots.txt for user requests? OpenAI (ChatGPT-User, rules may not apply), Perplexity (Perplexity-User generally ignores robots.txt), Google (user-triggered fetchers generally ignore it), Amazon (Amzn-User may not follow all directives) and Meta (Meta-ExternalFetcher may bypass it). Anthropic is the exception: it says Claude-User lets site owners control user-initiated access. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. ### How can I tell a real Googlebot from a fake one? Run a reverse DNS lookup on the IP; it must end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the same IP. For volume, match against Google's published JSON lists for common crawlers, special crawlers, user-triggered fetchers and user-triggered agents. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. ### What is Web Bot Auth? Web Bot Auth is an IETF protocol where a bot signs each request with HTTP Message Signatures (RFC 9421) and names its key directory in a Signature-Agent header. The site fetches the public keys and verifies the signature. The working group draft draft-ietf-webbotauth-httpsig-protocol reached revision -01 on 2026-10-29. [Source](https://datatracker.ietf.org/wg/webbotauth/documents/), read 2026-11-04. ### Which operators sign their agent traffic today? Read on 2026-11-04: OpenAI's Cloud browser (keys at chatgpt.com), Google-Agent for a subset of requests (agent.bot.goog, experimental), Browserbase (keys at browserbase.com), Cloudflare Browser Rendering, AWS AgentCore Browser (preview, opt-in) and Apify crawlers when signHttpRequests is on. Anthropic, Perplexity, Meta and Microsoft publish no key directory for their agents. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ### Can I block Claude in Chrome or Comet? Not by identity. Both run inside the user's own browser and send that browser's user agent, cookies and IP, and neither publishes a token or signature for site owners. Detection vendors flag them by behaviour. Rules aimed at them will also hit the human using the browser. [Source](https://www.centinelanalytica.com/blog/ai-browser-agents), read 2026-11-04. ### What does Google-Agent do? Google-Agent is the user-triggered agent Google uses for agents hosted on its infrastructure that navigate the web and act on user request. It comes from the ranges in user-triggered-agents.json, and a subset of its requests is signed with Web Bot Auth as https://agent.bot.goog. Google calls the signing experimental. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04. ### Does blocking Google-Extended hurt my Google rankings? Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether crawled content trains Gemini models and grounds Gemini Apps and Vertex AI answers. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ### How do I stop Apple training on my site but stay in Siri and Spotlight? Disallow Applebot-Extended in robots.txt and keep Applebot allowed. Applebot-Extended crawls nothing; it only governs how Applebot data is used. Apple also reads nosnippet to keep content out of broad world knowledge answers. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. ### How do I opt out of Bing training? There is no robots.txt token for it yet. Use the NOARCHIVE meta tag, which Microsoft says does not affect ranking. Cloudflare reports Microsoft is building robots.txt support for a no-training preference, targeted for early 2027. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ### What changed in Cloudflare's AI crawler settings on 15 September 2026? Cloudflare added Disallow AI Training, made Block and Block on pages with ads apply to mixed-use crawlers such as Googlebot, Bingbot and Applebot, deprecated the Block AI Bots toggle in favour of Search, Training and Agent controls, and replaced Managed Robots.txt with Bot Preference Sync. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ### Is Cloudflare pay per crawl available to everyone? No. Cloudflare's docs (updated 2026-07-28) say pay per crawl is in closed beta and site owners must apply or go through their account executive. Crawlers get HTTP 402 with a crawler-price header and Cloudflare acts as Merchant of Record. [Source](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/), read 2026-11-04. ### Can AWS WAF charge AI bots? Yes, since 2026-06-15. AWS WAF AI traffic monetization returns an x402 HTTP 402 with prices and terms, settles through Coinbase in stablecoins, and works only on web ACLs attached to CloudFront. AWS charges nothing extra for it beyond WAF fees. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. ### Does Meta publish IP ranges for its AI crawlers? Not on its crawler page read on 2026-11-04. Meta lists the user agent strings for FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher and says IP allowlisting is more secure, but links no IP file or reverse DNS pattern. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. ### What is the difference between Meta-ExternalAgent and Meta-ExternalFetcher? Meta-ExternalAgent crawls for uses such as training foundation models or indexing content. Meta-ExternalFetcher fetches individual links at a user's request, including helping AI complete tasks, and Meta says it may bypass robots.txt. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. ### Does Bytespider respect robots.txt? ByteDance publishes no documentation for Bytespider, so there is no operator statement to check against (checked 2026-11-04). Treat any claim about its behaviour as third-party observation, and enforce at the WAF if you need it gone. [Source](https://www.centinelanalytica.com/bots/bytespider), read 2026-11-04. ### How do I verify CCBot? Common Crawl runs CCBot from dedicated ranges with reverse DNS under crawl.commoncrawl.org (IPv4 only) and publishes the ranges at index.commoncrawl.org/ccbot.json. It warns that other crawlers falsely identify as CCBot. [Source](https://commoncrawl.org/ccbot), read 2026-11-04. ### Which crawlers support Crawl-delay? Anthropic supports it ("User-agent: ClaudeBot" then "Crawl-delay: 1"); ImagesiftBot supports it as minimum seconds between request starts; Amazon says Amazonbot, Amzn-SearchBot and Amzn-User do not. RFC 9309 does not define Crawl-delay. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. ### How long do robots.txt changes take to apply? OpenAI and Perplexity say about 24 hours; Amazon says about 24 hours and may use a cached copy up to 30 days old; Meta says up to 24 hours; DuckDuckGo says 72 hours for DuckAssistBot. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. ### What does a Cloudflare signed agent mean for my rules? Signed agents are agents directed by end users whose requests are signed with Web Bot Auth and validated by Cloudflare. Since 2026-07-01 they sit under the verified bot umbrella, so you can allow or skip them with rules on verified status or a specific detection ID. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04. ### How do I allow ChatGPT's Cloud browser on Cloudflare? OpenAI says create a WAF custom rule on Bot Detection ID equals 129220581 (bot tag chatgpt-agent) and set it to allow or skip. Cloudflare verifies the signature itself. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ### Does Vercel block AI crawlers? Vercel's AI Bots managed ruleset can log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) separately, on all plans, since 2026-10-29. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. ### What is the Verification Ledger on this site? A table of which operators publish verification material (IP JSON, reverse DNS pattern, Web Bot Auth key directory, Cloudflare verified status, Visa TAP or Mastercard Agent Pay registration), with the exact URL and the date it was read, and which publish nothing. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. ### What is Visa Trusted Agent Protocol? A Visa specification for AI agents to present a cryptographic signature with timestamps to merchants, so a merchant can tell a credentialed agent with user permission from an anonymous bot. Visa publishes reference code on GitHub; Cloudflare says it is built on Web Bot Auth. [Source](https://github.com/visa/trusted-agent-protocol), read 2026-11-04. ### What does a robots.txt Content-Signal line do? It states preferences for search, ai-input and ai-train under the Content Signals Policy, for example "Content-Signal: search=yes, ai-train=no". It identifies nobody and stops nobody; Cloudflare says enforcement needs a network that identifies the crawler. [Source](https://blog.cloudflare.com/content-signals-policy/), read 2026-11-04. ### Does returning 403 to AI agents hurt how assistants describe my business? Ora's lab found (2026-10-19) that a 403 to agent user agents left both test clients with 0 of 5 answers retrieved, one of only two changes that broke retrieval in its experiment. That is a vendor finding on one docs site; test your own pages. [Source](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=faq), read 2026-11-04. ### Which AI crawler sends the most traffic? It depends on whose network you ask. DataDome's Q3 2026 report put Meta-ExternalAgent and Meta-WebIndexer at 49.7 percent of nearly 25 billion AI agent requests; HUMAN's August 2026 data put Comet at 40.7 percent of agentic browser traffic. Both are vendor figures from their own customers. [Source](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report), read 2026-11-04. ### Where do the block rates on this site come from? From the network's shared crawl of 2026-11-04: a plain fetch of robots.txt for the Tranco top 10,000 plus leaderboard and vendor domains, parsed per token. The crawl stores whether each token is disallowed at the root and whether it is named explicitly. [Source](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04. ### How do I test whether AI agents can reach my site? Read your robots.txt for the tokens that matter, request the home page once with each agent's user agent and once as a browser, and compare the status codes. Then look for the real agents in your logs and run a scan such as Ora's. A copied user agent is not a real agent, so a 403 on the copy is a prompt to check the logs before changing a rule. [Source](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=faq), read 2026-11-05. ### Does Ora's scanner sign its requests? Ora says every request that carries the OraBot user agent is signed with HTTP Message Signatures using an Ed25519 key published in its key directory. During a scan it also sends a few unsigned home page requests that carry other crawlers' user agents, such as GPTBot and ClaudeBot, to see how a site answers them. [Source](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=faq), read 2026-11-05. ### Which AI bots should a publisher allow? A publisher can refuse the training tokens and keep AI search and user-triggered fetches open. OpenAI treats OAI-SearchBot and GPTBot as independent settings, and says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers. The site-type guide writes a robots.txt for five kinds of site from the Crawler Index. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 - [centinelanalytica.com: bytespider](https://www.centinelanalytica.com/bots/bytespider) read 2026-11-04 - [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 - [ora.ai: agentready launch](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=faq) read 2026-11-04 - [helpnetsecurity.com: datadome ai agent traffic report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [ora.ai](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=faq) read 2026-11-04 - [ora.ai](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=faq) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Glossary of crawler and agent terms | Agent Ready Crawlers" description: "41 glossary terms for AI crawler traffic: tokens, verification, Web Bot Auth, Content-Signals, pay per crawl and bot management, each with a sourced example." canonical: https://agentreadycrawlers.com/glossary site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Glossary of crawler and agent terms | Agent Ready Crawlers > 41 glossary terms for AI crawler traffic: tokens, verification, Web Bot Auth, Content-Signals, pay per crawl and bot management, each with a sourced example. Source: https://agentreadycrawlers.com/glossary Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Glossary In short 41 terms for reading robots.txt, crawler logs and bot-management settings, from Applebot-Extended to Web Bot Auth. Each has a sourced example, read 2026-11-04, and a note on how it shows up in the Crawler Index or the crawl behind the Block Rate Report. [Accountable designation](https://agentreadycrawlers.com/glossary/accountable-designation/) Cloudflare's label for crawler operators that offer a training opt-out, an AI-summary opt-out, URL-level visibility, and an assurance that opting out of training does not affect search. [Agentic browser](https://agentreadycrawlers.com/glossary/agentic-browser/) A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot. [ai-input](https://agentreadycrawlers.com/glossary/ai-input/) The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval. [AIPREF training signal](https://agentreadycrawlers.com/glossary/aipref-training-signal/) An IETF AI Preferences vocabulary for usage preferences. [Applebot-Extended](https://agentreadycrawlers.com/glossary/applebot-extended/) Apple's control token for whether Applebot data trains Apple foundation models. [Bot management](https://agentreadycrawlers.com/glossary/bot-management/) Products that classify automated traffic and apply allow, challenge, rate-limit or block decisions, increasingly with separate handling for verified AI crawlers and agents. [Bot paywall](https://agentreadycrawlers.com/glossary/bot-paywall/) A redirect target where AI bots are told the price for content and must present a valid token to get it. [Bot Preference Sync](https://agentreadycrawlers.com/glossary/bot-preference-sync/) The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves, replacing Managed Robots.txt from 2026-09-15. [Bot score](https://agentreadycrawlers.com/glossary/bot-score/) A numeric likelihood that a request is automated, used in rules. [BotBase](https://agentreadycrawlers.com/glossary/botbase/) Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits. [Browsing agent](https://agentreadycrawlers.com/glossary/browsing-agent/) Software that drives a browser to complete a task for a person: clicking, typing and checking out. [Content-Signal](https://agentreadycrawlers.com/glossary/content-signal/) A robots.txt line from the Content Signals Policy that states yes or no for three uses: search, ai-input and ai-train. [Crawl-delay](https://agentreadycrawlers.com/glossary/crawl-delay/) A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. [crawler-price header](https://agentreadycrawlers.com/glossary/crawler-price-header/) The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. [Disallow AI Training](https://agentreadycrawlers.com/glossary/disallow-ai-training/) A Cloudflare Training-control setting launched 2026-09-15 that publishes a no-training preference in robots.txt, keeps Accountable mixed-use crawlers for search and blocks other training crawlers. [Google-Extended](https://agentreadycrawlers.com/glossary/google-extended/) Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. [HTTP 402 Payment Required](https://agentreadycrawlers.com/glossary/http-402/) An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment. [HTTP Message Signatures](https://agentreadycrawlers.com/glossary/http-message-signatures/) RFC 9421: a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. [IP range JSON](https://agentreadycrawlers.com/glossary/ip-range-json/) A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. [Key directory](https://agentreadycrawlers.com/glossary/key-directory/) A JSON document at /.well-known/http-message-signatures-directory listing a signer's public keys for Web Bot Auth, served with the media type application/http-message-signatures-directory+json. [Know Your Agent (KYA)](https://agentreadycrawlers.com/glossary/know-your-agent/) An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token. [KYAPay](https://agentreadycrawlers.com/glossary/kyapay/) The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step. [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/) Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs can verify it. [Mixed-use crawler](https://agentreadycrawlers.com/glossary/mixed-use-crawler/) A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. [noarchive](https://agentreadycrawlers.com/glossary/noarchive/) A robots meta directive. [nosnippet](https://agentreadycrawlers.com/glossary/nosnippet/) A robots meta directive that limits snippets. [Pay per crawl](https://agentreadycrawlers.com/glossary/pay-per-crawl/) Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record. [Reverse DNS verification](https://agentreadycrawlers.com/glossary/reverse-dns-verification/) Checking a client IP by looking up its hostname, confirming the operator's domain, then looking the hostname up again to confirm it maps back to the same IP (forward-confirmed reverse DNS). [RFC 9309](https://agentreadycrawlers.com/glossary/rfc-9309/) The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching. [robots.txt](https://agentreadycrawlers.com/glossary/robots-txt/) A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. [Robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/) The name a crawler answers to in a User-agent line. [Search-index crawler](https://agentreadycrawlers.com/glossary/search-index-crawler/) A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training. [Signature-Agent header](https://agentreadycrawlers.com/glossary/signature-agent-header/) The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys. [Signed agent](https://agentreadycrawlers.com/glossary/signed-agent/) Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy. [Spoofed crawler](https://agentreadycrawlers.com/glossary/spoofed-crawler/) A request that copies a known crawler's user agent to get treatment meant for the real bot. [Training crawler](https://agentreadycrawlers.com/glossary/training-crawler/) A crawler whose fetched content may be used to train AI models. [User-agent string](https://agentreadycrawlers.com/glossary/user-agent-string/) The User-Agent HTTP header a client sends. [User-triggered fetcher](https://agentreadycrawlers.com/glossary/user-triggered-fetcher/) A fetch made because a person asked an assistant something, such as reading a linked page. [Verified bot](https://agentreadycrawlers.com/glossary/verified-bot/) A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature, and lists in a directory site owners can write rules against. [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/) Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/) An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys, so sites can verify the operator without trusting IPs or user agents. ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04 - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Accountable designation: definition | Agent Ready Crawlers" description: "Accountable designation: Cloudflare's label for crawlers whose operators offer a training opt-out and say it does not affect search. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/accountable-designation site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Accountable designation: definition | Agent Ready Crawlers > Accountable designation: Cloudflare's label for crawlers whose operators offer a training opt-out and say it does not affect search. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/accountable-designation Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Accountable designation In short Cloudflare's label for crawler operators that offer a training opt-out, an AI-summary opt-out, URL-level visibility, and an assurance that opting out of training does not affect search. Cloudflare applied the label on 2026-09-15 to Apple, Google and Microsoft as mixed-use crawlers, and to Amazon, Anthropic, Meta and OpenAI as operators of split crawlers. ## Definition Cloudflare's label for crawler operators that offer a training opt-out, an AI-summary opt-out, URL-level visibility, and an assurance that opting out of training does not affect search. ## Sourced example Cloudflare says Apple, Google and Microsoft meet the Accountable qualifications. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## How it shows up here Cloudflare applied the label on 2026-09-15 to Apple, Google and Microsoft as mixed-use crawlers, and to Amazon, Anthropic, Meta and OpenAI as operators of split crawlers. - [Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) Previous: [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/). Next: [Agentic browser](https://agentreadycrawlers.com/glossary/agentic-browser/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Agentic browser: definition | Agent Ready Crawlers" description: "Agentic browser: A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot." canonical: https://agentreadycrawlers.com/glossary/agentic-browser site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Agentic browser: definition | Agent Ready Crawlers > Agentic browser: A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot. Source: https://agentreadycrawlers.com/glossary/agentic-browser Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Agentic browser In short A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot. The Index covers 5 of them: Comet, Claude in Chrome, ChatGPT Atlas, Gemini in Chrome and Browse with Copilot. None publishes a verification method. Example: Verisoul reports Comet became a free download on 2025-10-02. ## Definition A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot. ## Sourced example Verisoul reports Comet became a free download on 2025-10-02. [Source](https://www.verisoul.ai/agents/perplexity-comet), read 2026-11-04. ## How it shows up here The Index covers 5 of them: Comet, Claude in Chrome, ChatGPT Atlas, Gemini in Chrome and Browse with Copilot. None publishes a verification method. - [Comet profile](https://agentreadycrawlers.com/crawlers/perplexity-comet/) - [Gemini in Chrome profile](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Previous: [Accountable designation](https://agentreadycrawlers.com/glossary/accountable-designation/). Next: [ai-input](https://agentreadycrawlers.com/glossary/ai-input/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "ai-input: definition | Agent Ready Crawlers" description: "ai-input: The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval. Leaving it blank states no preference." canonical: https://agentreadycrawlers.com/glossary/ai-input site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # ai-input: definition | Agent Ready Crawlers > ai-input: The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval. Leaving it blank states no preference. Source: https://agentreadycrawlers.com/glossary/ai-input Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # ai-input In short The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval. Leaving it blank states no preference. The builder sets it from the Live answer fetch choice: allow gives ai-input=yes and refuse gives ai-input=no. Example: Cloudflare serves no ai-input value for managed robots.txt customers because it does not know their preference. ## Definition The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval. Leaving it blank states no preference. ## Sourced example Cloudflare serves no ai-input value for managed robots.txt customers because it does not know their preference. [Source](https://blog.cloudflare.com/content-signals-policy/), read 2026-11-04. ## How it shows up here The builder sets it from the Live answer fetch choice: allow gives ai-input=yes and refuse gives ai-input=no. - [What Content-Signals changes](https://agentreadycrawlers.com/guides/what-content-signals-change/) - [Content-Signal](https://agentreadycrawlers.com/glossary/content-signal/) Previous: [Agentic browser](https://agentreadycrawlers.com/glossary/agentic-browser/). Next: [AIPREF training signal](https://agentreadycrawlers.com/glossary/aipref-training-signal/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "AIPREF training signal: definition | Agent Ready Crawlers" description: "AIPREF training signal: An IETF AI Preferences vocabulary for usage preferences. The robots.txt builder on this site does not write it." canonical: https://agentreadycrawlers.com/glossary/aipref-training-signal site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # AIPREF training signal: definition | Agent Ready Crawlers > AIPREF training signal: An IETF AI Preferences vocabulary for usage preferences. The robots.txt builder on this site does not write it. Source: https://agentreadycrawlers.com/glossary/aipref-training-signal Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # AIPREF training signal In short An IETF AI Preferences vocabulary for usage preferences. Google says it intends to read the training preference written as a user-agent line, \_\_usage\_ai-training, in early 2027. The robots.txt builder on this site does not write it. Related: What Content-Signals changes and Google-Extended profile. ## Definition An IETF AI Preferences vocabulary for usage preferences. Google says it intends to read the training preference written as a user-agent line, \_\_usage\_ai-training, in early 2027. ## Sourced example twitter.com's robots.txt already names \_\_usage\_ai-training (shared crawl, 2026-11-04); Google plans support in early 2027. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ## How it shows up here The robots.txt builder on this site does not write it. - [What Content-Signals changes](https://agentreadycrawlers.com/guides/what-content-signals-change/) - [Google-Extended profile](https://agentreadycrawlers.com/crawlers/google-extended/) Previous: [ai-input](https://agentreadycrawlers.com/glossary/ai-input/). Next: [Applebot-Extended](https://agentreadycrawlers.com/glossary/applebot-extended/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Applebot-Extended: definition | Agent Ready Crawlers" description: "Applebot-Extended: Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself." canonical: https://agentreadycrawlers.com/glossary/applebot-extended site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Applebot-Extended: definition | Agent Ready Crawlers > Applebot-Extended: Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself. Source: https://agentreadycrawlers.com/glossary/applebot-extended Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Applebot-Extended In short Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself. 14.8 percent of top-10,000 sites refuse it, 11.6 percent by naming it. Example: Apple says pages that disallow Applebot-Extended can still be included in search results. Related: Applebot-Extended profile and Allow search, refuse training. ## Definition Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself. ## Sourced example Apple says pages that disallow Applebot-Extended can still be included in search results. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. ## How it shows up here 14.8 percent of top-10,000 sites refuse it, 11.6 percent by naming it. - [Applebot-Extended profile](https://agentreadycrawlers.com/crawlers/applebot-extended/) - [Allow search, refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) Previous: [AIPREF training signal](https://agentreadycrawlers.com/glossary/aipref-training-signal/). Next: [Bot management](https://agentreadycrawlers.com/glossary/bot-management/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Bot management: definition | Agent Ready Crawlers" description: "Bot management: products that classify automated traffic and allow, challenge, rate-limit or block it, now with separate handling for AI crawlers and agents." canonical: https://agentreadycrawlers.com/glossary/bot-management site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Bot management: definition | Agent Ready Crawlers > Bot management: products that classify automated traffic and allow, challenge, rate-limit or block it, now with separate handling for AI crawlers and agents. Source: https://agentreadycrawlers.com/glossary/bot-management Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Bot management In short Products that classify automated traffic and apply allow, challenge, rate-limit or block decisions, increasingly with separate handling for verified AI crawlers and agents. 1.3 percent of home pages carried a header or cookie of a bot product (DataDome, HUMAN, Imperva, Kasada or Arcjet) and 6.1 percent a Cloudflare challenge header. Example: Fastly exposes VERIFIED-BOT.AI-CRAWLER and SUSPECTED-BOT.AI-FETCHER signals. ## Definition Products that classify automated traffic and apply allow, challenge, rate-limit or block decisions, increasingly with separate handling for verified AI crawlers and agents. ## Sourced example Fastly exposes VERIFIED-BOT.AI-CRAWLER and SUSPECTED-BOT.AI-FETCHER signals. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. ## How it shows up here 1.3 percent of home pages carried a header or cookie of a bot product (DataDome, HUMAN, Imperva, Kasada or Arcjet) and 6.1 percent a Cloudflare challenge header. - [Vendor ranking](https://agentreadycrawlers.com/vendors/) - [Header table in the edition page](https://agentreadycrawlers.com/block-rates/2026-11-04/) Previous: [Applebot-Extended](https://agentreadycrawlers.com/glossary/applebot-extended/). Next: [Bot paywall](https://agentreadycrawlers.com/glossary/bot-paywall/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Bot paywall: definition | Agent Ready Crawlers" description: "Bot paywall: A redirect target where AI bots are told the price for content and must present a valid token to get it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/bot-paywall site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Bot paywall: definition | Agent Ready Crawlers > Bot paywall: A redirect target where AI bots are told the price for content and must present a valid token to get it. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/bot-paywall Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Bot paywall In short A redirect target where AI bots are told the price for content and must present a valid token to get it. TollBit sells one; the rubric grades it yes on charging and partial on verification because it relies on CDN partners to classify bots. Example: Fastly classifies an AI bot and redirects it to the publisher's TollBit Bot Paywall. ## Definition A redirect target where AI bots are told the price for content and must present a valid token to get it. ## Sourced example Fastly classifies an AI bot and redirects it to the publisher's TollBit Bot Paywall. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. ## How it shows up here TollBit sells one; the rubric grades it yes on charging and partial on verification because it relies on CDN partners to classify bots. - [TollBit profile](https://agentreadycrawlers.com/vendors/tollbit/) - [Fastly profile](https://agentreadycrawlers.com/vendors/fastly/) Previous: [Bot management](https://agentreadycrawlers.com/glossary/bot-management/). Next: [Bot Preference Sync](https://agentreadycrawlers.com/glossary/bot-preference-sync/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Bot Preference Sync: definition | Agent Ready Crawlers" description: "Bot Preference Sync: The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/bot-preference-sync site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Bot Preference Sync: definition | Agent Ready Crawlers > Bot Preference Sync: The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/bot-preference-sync Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Bot Preference Sync In short The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves, replacing Managed Robots.txt from 2026-09-15. It replaced Managed Robots.txt on 2026-09-15, so a Cloudflare-served robots.txt may differ from the file in your repository. Example: Customers who enabled Managed Robots.txt migrate to Bot Preference Sync, per Cloudflare. ## Definition The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves, replacing Managed Robots.txt from 2026-09-15. ## Sourced example Customers who enabled Managed Robots.txt migrate to Bot Preference Sync, per Cloudflare. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## How it shows up here It replaced Managed Robots.txt on 2026-09-15, so a Cloudflare-served robots.txt may differ from the file in your repository. - [Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) Previous: [Bot paywall](https://agentreadycrawlers.com/glossary/bot-paywall/). Next: [Bot score](https://agentreadycrawlers.com/glossary/bot-score/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Bot score: definition | Agent Ready Crawlers" description: "Bot score: A numeric likelihood that a request is automated, used in rules. Signed traffic can be given a fixed score. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/bot-score site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Bot score: definition | Agent Ready Crawlers > Bot score: A numeric likelihood that a request is automated, used in rules. Signed traffic can be given a fixed score. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/bot-score Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Bot score In short A numeric likelihood that a request is automated, used in rules. Signed traffic can be given a fixed score. Vendors expose it differently: DataDome scores each agent on identity strength and fraud history, per its pricing documents. Example: Cloudflare Browser Rendering's signed requests are always given bot score 1 for Bot Management customers. Related: DataDome profile and Cloudflare profile. ## Definition A numeric likelihood that a request is automated, used in rules. Signed traffic can be given a fixed score. ## Sourced example Cloudflare Browser Rendering's signed requests are always given bot score 1 for Bot Management customers. [Source](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. ## How it shows up here Vendors expose it differently: DataDome scores each agent on identity strength and fraud history, per its pricing documents. - [DataDome profile](https://agentreadycrawlers.com/vendors/datadome/) - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) Previous: [Bot Preference Sync](https://agentreadycrawlers.com/glossary/bot-preference-sync/). Next: [BotBase](https://agentreadycrawlers.com/glossary/botbase/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "BotBase: definition | Agent Ready Crawlers" description: "BotBase: Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits." canonical: https://agentreadycrawlers.com/glossary/botbase site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # BotBase: definition | Agent Ready Crawlers > BotBase: Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits. Source: https://agentreadycrawlers.com/glossary/botbase Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # BotBase In short Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits. Cloudflare opened BotBase for Operators on 2026-08-28, giving operators submission status, editing and a behaviour model. Example: Operators can now edit a submission when moving from an IP list to Web Bot Auth signing. ## Definition Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits. ## Sourced example Operators can now edit a submission when moving from an IP list to Web Bot Auth signing. [Source](https://blog.cloudflare.com/botbase-for-operators/), read 2026-11-04. ## How it shows up here Cloudflare opened BotBase for Operators on 2026-08-28, giving operators submission status, editing and a behaviour model. - [Cloudflare opens BotBase for Operators](https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators/) - [How the verified bot list works](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/) Previous: [Bot score](https://agentreadycrawlers.com/glossary/bot-score/). Next: [Browsing agent](https://agentreadycrawlers.com/glossary/browsing-agent/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [BotBase for Operators](https://blog.cloudflare.com/botbase-for-operators/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Browsing agent: definition | Agent Ready Crawlers" description: "Browsing agent: Software that drives a browser to complete a task for a person, clicking, typing and checking out. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/browsing-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Browsing agent: definition | Agent Ready Crawlers > Browsing agent: Software that drives a browser to complete a task for a person, clicking, typing and checking out. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/browsing-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Browsing agent In short Software that drives a browser to complete a task for a person: clicking, typing and checking out. It may run in the operator's cloud or in the user's own browser. 20 Index rows are task agents, and 19 of them have no robots.txt token. Example: ChatGPT's Cloud browser runs in OpenAI's cloud and signs its requests. ## Definition Software that drives a browser to complete a task for a person: clicking, typing and checking out. It may run in the operator's cloud or in the user's own browser. ## Sourced example ChatGPT's Cloud browser runs in OpenAI's cloud and signs its requests; Claude in Chrome runs in the user's Chrome. [Source](https://www.centinelanalytica.com/blog/ai-browser-agents), read 2026-11-04. ## How it shows up here 20 Index rows are task agents, and 19 of them have no robots.txt token. - [All task agents](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Crawler versus browsing agent](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent/) Previous: [BotBase](https://agentreadycrawlers.com/glossary/botbase/). Next: [Content-Signal](https://agentreadycrawlers.com/glossary/content-signal/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Content-Signal: definition | Agent Ready Crawlers" description: "Content-Signal: A robots.txt line from the Content Signals Policy that states yes or no for three uses, search, ai-input and ai-train." canonical: https://agentreadycrawlers.com/glossary/content-signal site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Content-Signal: definition | Agent Ready Crawlers > Content-Signal: A robots.txt line from the Content Signals Policy that states yes or no for three uses, search, ai-input and ai-train. Source: https://agentreadycrawlers.com/glossary/content-signal Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Content-Signal In short A robots.txt line from the Content Signals Policy that states yes or no for three uses: search, ai-input and ai-train. 6.4 percent of top-10,000 robots.txt files (380 of 5,938) carry a Content-Signal line. Example: Cloudflare's example: "Content-Signal: search=yes, ai-train=no". Related: What Content-Signals changes and robots.txt builder. ## Definition A robots.txt line from the Content Signals Policy that states yes or no for three uses: search, ai-input and ai-train. ## Sourced example Cloudflare's example: "Content-Signal: search=yes, ai-train=no". [Source](https://blog.cloudflare.com/content-signals-policy/), read 2026-11-04. ## How it shows up here 6.4 percent of top-10,000 robots.txt files (380 of 5,938) carry a Content-Signal line. - [What Content-Signals changes](https://agentreadycrawlers.com/guides/what-content-signals-change/) - [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) Previous: [Browsing agent](https://agentreadycrawlers.com/glossary/browsing-agent/). Next: [Crawl-delay](https://agentreadycrawlers.com/glossary/crawl-delay/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Crawl-delay: definition | Agent Ready Crawlers" description: "Crawl-delay: A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/crawl-delay site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Crawl-delay: definition | Agent Ready Crawlers > Crawl-delay: A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/crawl-delay Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Crawl-delay In short A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. RFC 9309 does not define it; support varies by operator. The Index records rate behaviour as the operator documents it, in words, because operators describe it differently; the crawl does not measure Crawl-delay. Example: Anthropic supports Crawl-delay. Related: Amazonbot profile and ClaudeBot profile. ## Definition A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. RFC 9309 does not define it; support varies by operator. ## Sourced example Anthropic supports Crawl-delay; Amazon says its bots do not; Google ignores it. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. ## How it shows up here The Index records rate behaviour as the operator documents it, in words, because operators describe it differently; the crawl does not measure Crawl-delay. - [Amazonbot profile](https://agentreadycrawlers.com/crawlers/amazonbot/) - [ClaudeBot profile](https://agentreadycrawlers.com/crawlers/claudebot/) Previous: [Content-Signal](https://agentreadycrawlers.com/glossary/content-signal/). Next: [crawler-price header](https://agentreadycrawlers.com/glossary/crawler-price-header/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "crawler-price header: definition | Agent Ready Crawlers" description: "crawler-price header: The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/crawler-price-header site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # crawler-price header: definition | Agent Ready Crawlers > crawler-price header: The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/crawler-price-header Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # crawler-price header In short The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. It is how a crawler learns the price before it retries; AWS WAF uses an x402 response instead. Example: HTTP 402 Payment Required with crawler-price: USD XX.XX. Related: What pay per crawl pays and Cloudflare profile. ## Definition The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. ## Sourced example HTTP 402 Payment Required with crawler-price: USD XX.XX. [Source](https://blog.cloudflare.com/introducing-pay-per-crawl/), read 2026-11-04. ## How it shows up here It is how a crawler learns the price before it retries; AWS WAF uses an x402 response instead. - [What pay per crawl pays](https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays/) - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) Previous: [Crawl-delay](https://agentreadycrawlers.com/glossary/crawl-delay/). Next: [Disallow AI Training](https://agentreadycrawlers.com/glossary/disallow-ai-training/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Disallow AI Training: definition | Agent Ready Crawlers" description: "Disallow AI Training: a Cloudflare setting, launched 2026-09-15, that publishes a no-training preference in robots.txt and keeps search crawlers allowed." canonical: https://agentreadycrawlers.com/glossary/disallow-ai-training site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Disallow AI Training: definition | Agent Ready Crawlers > Disallow AI Training: a Cloudflare setting, launched 2026-09-15, that publishes a no-training preference in robots.txt and keeps search crawlers allowed. Source: https://agentreadycrawlers.com/glossary/disallow-ai-training Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Disallow AI Training In short A Cloudflare Training-control setting launched 2026-09-15 that publishes a no-training preference in robots.txt, keeps Accountable mixed-use crawlers for search and blocks other training crawlers. It is one of the Cloudflare controls dated 2026-09-15 and the reason Block now stops Googlebot, Bingbot and Applebot while Disallow AI Training does not. ## Definition A Cloudflare Training-control setting launched 2026-09-15 that publishes a no-training preference in robots.txt, keeps Accountable mixed-use crawlers for search and blocks other training crawlers. ## Sourced example Under Disallow AI Training, training-only crawlers from Amazon, Anthropic, Meta and OpenAI are blocked. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## How it shows up here It is one of the Cloudflare controls dated 2026-09-15 and the reason Block now stops Googlebot, Bingbot and Applebot while Disallow AI Training does not. - [Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) - [Allow search, refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) Previous: [crawler-price header](https://agentreadycrawlers.com/glossary/crawler-price-header/). Next: [Google-Extended](https://agentreadycrawlers.com/glossary/google-extended/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google-Extended: definition | Agent Ready Crawlers" description: "Google-Extended: Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers." canonical: https://agentreadycrawlers.com/glossary/google-extended site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Google-Extended: definition | Agent Ready Crawlers > Google-Extended: Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. Source: https://agentreadycrawlers.com/glossary/google-extended Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Google-Extended In short Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. It does not change Search inclusion or ranking. 16.0 percent of top-10,000 sites refuse it, 12.6 percent by naming it. It has no user-agent string, so it never appears in logs. Example: A Disallow for Google-Extended leaves Googlebot crawling for Search. ## Definition Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. It does not change Search inclusion or ranking. ## Sourced example A Disallow for Google-Extended leaves Googlebot crawling for Search. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ## How it shows up here 16.0 percent of top-10,000 sites refuse it, 12.6 percent by naming it. It has no user-agent string, so it never appears in logs. - [Google-Extended profile](https://agentreadycrawlers.com/crawlers/google-extended/) - [Tokens versus strings](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings/) Previous: [Disallow AI Training](https://agentreadycrawlers.com/glossary/disallow-ai-training/). Next: [HTTP 402 Payment Required](https://agentreadycrawlers.com/glossary/http-402/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "HTTP 402 Payment Required: definition | Agent Ready Crawlers" description: "HTTP 402 Payment Required: An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment." canonical: https://agentreadycrawlers.com/glossary/http-402 site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # HTTP 402 Payment Required: definition | Agent Ready Crawlers > HTTP 402 Payment Required: An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment. Source: https://agentreadycrawlers.com/glossary/http-402 Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # HTTP 402 Payment Required In short An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment. RFC 2068 (January 1997) defined the code and RFC 9110 still lists it as reserved for future use. 28 of 10,565 home pages in the crawl answered 402 to its user agent. ## Definition An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment. ## Sourced example AWS WAF returns a machine-readable 402 under x402 with prices, accepted payment methods and license terms. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. ## How it shows up here RFC 2068 (January 1997) defined the code and RFC 9110 still lists it as reserved for future use. 28 of 10,565 home pages in the crawl answered 402 to its user agent. - [What pay per crawl pays](https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays/) - [AWS WAF Bot Control profile](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/) Previous: [Google-Extended](https://agentreadycrawlers.com/glossary/google-extended/). Next: [HTTP Message Signatures](https://agentreadycrawlers.com/glossary/http-message-signatures/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [RFC 2068, section 10.4.3: 402 Payment Required](https://www.rfc-editor.org/rfc/rfc2068.txt) read 2026-11-05 - [RFC 9110, section 15.5.3: 402 Payment Required](https://www.rfc-editor.org/rfc/rfc9110.html) read 2026-11-05 Last reviewed 2026-11-04 --- --- title: "HTTP Message Signatures: definition | Agent Ready Crawlers" description: "HTTP Message Signatures: RFC 9421, a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers." canonical: https://agentreadycrawlers.com/glossary/http-message-signatures site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # HTTP Message Signatures: definition | Agent Ready Crawlers > HTTP Message Signatures: RFC 9421, a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. Source: https://agentreadycrawlers.com/glossary/http-message-signatures Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # HTTP Message Signatures In short RFC 9421: a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. Web Bot Auth is a profile of it. A verifier library for RFC 9421 is the starting point for the checks in the guide. Example: OpenAI tells site owners to verify ChatGPT Cloud browser requests according to RFC 9421. ## Definition RFC 9421: a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. Web Bot Auth is a profile of it. ## Sourced example OpenAI tells site owners to verify ChatGPT Cloud browser requests according to RFC 9421. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ## How it shows up here A verifier library for RFC 9421 is the starting point for the checks in the guide. - [Verify Web Bot Auth signatures](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) - [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/) Previous: [HTTP 402 Payment Required](https://agentreadycrawlers.com/glossary/http-402/). Next: [IP range JSON](https://agentreadycrawlers.com/glossary/ip-range-json/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "IP range JSON: definition | Agent Ready Crawlers" description: "IP range JSON: A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/ip-range-json site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # IP range JSON: definition | Agent Ready Crawlers > IP range JSON: A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/ip-range-json Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # IP range JSON In short A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. An IETF draft (jafar) proposes a common format. 17 Index rows publish an IP list as JSON, the most common method, and 35 publish no method at all. Example: OpenAI's gptbot.json carried creationTime 2026-09-22 when read on 2026-11-04. ## Definition A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. An IETF draft (jafar) proposes a common format. ## Sourced example OpenAI's gptbot.json carried creationTime 2026-09-22 when read on 2026-11-04. [Source](https://openai.com/gptbot.json), read 2026-11-04. ## How it shows up here 17 Index rows publish an IP list as JSON, the most common method, and 35 publish no method at all. - [Verification Ledger](https://agentreadycrawlers.com/verification/) - [GPTBot profile](https://agentreadycrawlers.com/crawlers/gptbot/) Previous: [HTTP Message Signatures](https://agentreadycrawlers.com/glossary/http-message-signatures/). Next: [Key directory](https://agentreadycrawlers.com/glossary/key-directory/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [openai.com: gptbot (JSON)](https://openai.com/gptbot.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Key directory: definition | Agent Ready Crawlers" description: "Key directory: the JSON file at /.well-known/http-message-signatures-directory that lists a signer's public keys for Web Bot Auth verification." canonical: https://agentreadycrawlers.com/glossary/key-directory site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Key directory: definition | Agent Ready Crawlers > Key directory: the JSON file at /.well-known/http-message-signatures-directory that lists a signer's public keys for Web Bot Auth verification. Source: https://agentreadycrawlers.com/glossary/key-directory Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Key directory In short A JSON document at /.well-known/http-message-signatures-directory listing a signer's public keys for Web Bot Auth, served with the media type application/http-message-signatures-directory+json. 4 operators in the Verification Ledger publish a directory URL. Example: https://chatgpt.com/.well-known/http-message-signatures-directory served one Ed25519 key on 2026-11-04. Related: Verification Ledger and Verify Web Bot Auth signatures. ## Definition A JSON document at /.well-known/http-message-signatures-directory listing a signer's public keys for Web Bot Auth, served with the media type application/http-message-signatures-directory+json. ## Sourced example https://chatgpt.com/.well-known/http-message-signatures-directory served one Ed25519 key on 2026-11-04. [Source](https://chatgpt.com/.well-known/http-message-signatures-directory), read 2026-11-04. ## How it shows up here 4 operators in the Verification Ledger publish a directory URL. - [Verification Ledger](https://agentreadycrawlers.com/verification/) - [Verify Web Bot Auth signatures](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) Previous: [IP range JSON](https://agentreadycrawlers.com/glossary/ip-range-json/). Next: [Know Your Agent (KYA)](https://agentreadycrawlers.com/glossary/know-your-agent/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [chatgpt.com: http message signatures directory](https://chatgpt.com/.well-known/http-message-signatures-directory) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Know Your Agent (KYA): definition | Agent Ready Crawlers" description: "Know Your Agent (KYA): An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token." canonical: https://agentreadycrawlers.com/glossary/know-your-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Know Your Agent (KYA): definition | Agent Ready Crawlers > Know Your Agent (KYA): An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token. Source: https://agentreadycrawlers.com/glossary/know-your-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Know Your Agent (KYA) In short An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token. Skyfire is the vendor profiled here that issues such tokens; the rubric grades it on verification and charging. Example: Skyfire issues kya tokens as ES256-signed JWTs sent in a kyapay-token header. ## Definition An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token. ## Sourced example Skyfire issues kya tokens as ES256-signed JWTs sent in a kyapay-token header. [Source](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md), read 2026-11-04. ## How it shows up here Skyfire is the vendor profiled here that issues such tokens; the rubric grades it on verification and charging. - [Skyfire profile](https://agentreadycrawlers.com/vendors/skyfire/) - [Skyfire vs Stytch IsAgent](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent/) Previous: [Key directory](https://agentreadycrawlers.com/glossary/key-directory/). Next: [KYAPay](https://agentreadycrawlers.com/glossary/kyapay/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "KYAPay: definition | Agent Ready Crawlers" description: "KYAPay: The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step." canonical: https://agentreadycrawlers.com/glossary/kyapay site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # KYAPay: definition | Agent Ready Crawlers > KYAPay: The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step. Source: https://agentreadycrawlers.com/glossary/kyapay Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # KYAPay In short The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step. It is the one scheme profiled here that joins identity and charging in a single token. Example: Skyfire's kya-pay token combines identity and payment. Related: Skyfire profile and Skyfire vs TollBit. ## Definition The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step. ## Sourced example Skyfire's kya-pay token combines identity and payment. [Source](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md), read 2026-11-04. ## How it shows up here It is the one scheme profiled here that joins identity and charging in a single token. - [Skyfire profile](https://agentreadycrawlers.com/vendors/skyfire/) - [Skyfire vs TollBit](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit/) Previous: [Know Your Agent (KYA)](https://agentreadycrawlers.com/glossary/know-your-agent/). Next: [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [docs.skyfire.xyz: introduction to skyfire.md](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Mastercard Agent Pay: definition | Agent Ready Crawlers" description: "Mastercard Agent Pay: Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs." canonical: https://agentreadycrawlers.com/glossary/mastercard-agent-pay site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Mastercard Agent Pay: definition | Agent Ready Crawlers > Mastercard Agent Pay: Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs. Source: https://agentreadycrawlers.com/glossary/mastercard-agent-pay Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Mastercard Agent Pay In short Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs can verify it. The Verification Ledger found no public operator-level registry of agents registered with it either. Example: Cloudflare says Agent Pay and TAP tag requests agent-payer-auth or agent-browser-auth. Related: Let TAP and Agent Pay through and Cloudflare profile. ## Definition Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs can verify it. ## Sourced example Cloudflare says Agent Pay and TAP tag requests agent-payer-auth or agent-browser-auth. [Source](https://blog.cloudflare.com/secure-agentic-commerce/), read 2026-11-04. ## How it shows up here The Verification Ledger found no public operator-level registry of agents registered with it either. - [Let TAP and Agent Pay through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/) - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) Previous: [KYAPay](https://agentreadycrawlers.com/glossary/kyapay/). Next: [Mixed-use crawler](https://agentreadycrawlers.com/glossary/mixed-use-crawler/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Mixed-use crawler: definition | Agent Ready Crawlers" description: "Mixed-use crawler: A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search." canonical: https://agentreadycrawlers.com/glossary/mixed-use-crawler site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Mixed-use crawler: definition | Agent Ready Crawlers > Mixed-use crawler: A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. Source: https://agentreadycrawlers.com/glossary/mixed-use-crawler Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Mixed-use crawler In short A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. Refusal in the crawl differs widely: Googlebot 2.8 percent, Bingbot 3.1 percent, Applebot 14.8 percent of top-10,000 sites, so the cost of refusing a mixed-use crawler is known to most sites. Related: Googlebot profile and Applebot profile. ## Definition A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. ## Sourced example Cloudflare names Applebot, Bingbot and Googlebot as mixed-use crawlers in its 2026-09-15 post. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## How it shows up here Refusal in the crawl differs widely: Googlebot 2.8 percent, Bingbot 3.1 percent, Applebot 14.8 percent of top-10,000 sites, so the cost of refusing a mixed-use crawler is known to most sites. - [Googlebot profile](https://agentreadycrawlers.com/crawlers/googlebot/) - [Applebot profile](https://agentreadycrawlers.com/crawlers/applebot/) - [Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) Previous: [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/). Next: [noarchive](https://agentreadycrawlers.com/glossary/noarchive/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "noarchive: definition | Agent Ready Crawlers" description: "noarchive: A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as \"do not use the page for model training\"." canonical: https://agentreadycrawlers.com/glossary/noarchive site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # noarchive: definition | Agent Ready Crawlers > noarchive: A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as "do not use the page for model training". Source: https://agentreadycrawlers.com/glossary/noarchive Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # noarchive In short A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as "do not use the page for model training". It matters for Bing, which has no robots.txt training token yet; the guide on training opt-outs covers the gap. Example: Amazon says its bots honour noarchive as do not use the page for model. ## Definition A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as "do not use the page for model training". ## Sourced example Amazon says its bots honour noarchive as do not use the page for model training. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. ## How it shows up here It matters for Bing, which has no robots.txt training token yet; the guide on training opt-outs covers the gap. - [Allow search, refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) - [Bingbot profile](https://agentreadycrawlers.com/crawlers/bingbot/) Previous: [Mixed-use crawler](https://agentreadycrawlers.com/glossary/mixed-use-crawler/). Next: [nosnippet](https://agentreadycrawlers.com/glossary/nosnippet/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "nosnippet: definition | Agent Ready Crawlers" description: "nosnippet: A robots meta directive that limits snippets. Apple applies it to opt content out of broad world knowledge answers." canonical: https://agentreadycrawlers.com/glossary/nosnippet site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # nosnippet: definition | Agent Ready Crawlers > nosnippet: A robots meta directive that limits snippets. Apple applies it to opt content out of broad world knowledge answers. Source: https://agentreadycrawlers.com/glossary/nosnippet Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # nosnippet In short A robots meta directive that limits snippets. Apple applies it to opt content out of broad world knowledge answers. It works separately from Applebot-Extended, which governs training, so a page can use one control without the other. Related: Applebot profile and Applebot-Extended profile. ## Definition A robots meta directive that limits snippets. Apple applies it to opt content out of broad world knowledge answers. ## Sourced example Apple says nosnippet opts content out of being used in Siri and Search knowledge answers. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. ## How it shows up here It works separately from Applebot-Extended, which governs training, so a page can use one control without the other. - [Applebot profile](https://agentreadycrawlers.com/crawlers/applebot/) - [Applebot-Extended profile](https://agentreadycrawlers.com/crawlers/applebot-extended/) Previous: [noarchive](https://agentreadycrawlers.com/glossary/noarchive/). Next: [Pay per crawl](https://agentreadycrawlers.com/glossary/pay-per-crawl/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Pay per crawl: definition | Agent Ready Crawlers" description: "Pay per crawl: Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record." canonical: https://agentreadycrawlers.com/glossary/pay-per-crawl site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Pay per crawl: definition | Agent Ready Crawlers > Pay per crawl: Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record. Source: https://agentreadycrawlers.com/glossary/pay-per-crawl Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Pay per crawl In short Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record. The crawl saw 28 home pages answer 402, 0.3 percent of those fetched. A 402 is not proof of pay per crawl, which is in closed beta at Cloudflare. Related: What pay per crawl pays and TollBit vs Cloudflare. ## Definition Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record. ## Sourced example A crawler gets 402 with crawler-price: USD XX.XX and can retry with crawler-exact-price. [Source](https://blog.cloudflare.com/introducing-pay-per-crawl/), read 2026-11-04. ## How it shows up here The crawl saw 28 home pages answer 402, 0.3 percent of those fetched. A 402 is not proof of pay per crawl, which is in closed beta at Cloudflare. - [What pay per crawl pays](https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays/) - [TollBit vs Cloudflare](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/) Previous: [nosnippet](https://agentreadycrawlers.com/glossary/nosnippet/). Next: [Reverse DNS verification](https://agentreadycrawlers.com/glossary/reverse-dns-verification/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Reverse DNS verification: definition | Agent Ready Crawlers" description: "Reverse DNS verification: Checking a client IP by looking up its hostname, confirming the operator's domain. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/reverse-dns-verification site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Reverse DNS verification: definition | Agent Ready Crawlers > Reverse DNS verification: Checking a client IP by looking up its hostname, confirming the operator's domain. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/reverse-dns-verification Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Reverse DNS verification In short Checking a client IP by looking up its hostname, confirming the operator's domain, then looking the hostname up again to confirm it maps back to the same IP (forward-confirmed reverse DNS). 9 Index rows verify by reverse DNS, among them Googlebot, Google-CloudVertexBot, GoogleOther. Example: Google: host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com, which resolves back to 66.249.66.1. ## Definition Checking a client IP by looking up its hostname, confirming the operator's domain, then looking the hostname up again to confirm it maps back to the same IP (forward-confirmed reverse DNS). ## Sourced example Google: host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com, which resolves back to 66.249.66.1. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. ## How it shows up here 9 Index rows verify by reverse DNS, among them Googlebot, Google-CloudVertexBot, GoogleOther. - [Verification Ledger](https://agentreadycrawlers.com/verification/) - [Read agent traffic in logs](https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs/) Previous: [Pay per crawl](https://agentreadycrawlers.com/glossary/pay-per-crawl/). Next: [RFC 9309](https://agentreadycrawlers.com/glossary/rfc-9309/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "RFC 9309: definition | Agent Ready Crawlers" description: "RFC 9309: the IETF Robots Exclusion Protocol standard of September 2022, which defines groups, Allow and Disallow rules and longest-match precedence." canonical: https://agentreadycrawlers.com/glossary/rfc-9309 site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # RFC 9309: definition | Agent Ready Crawlers > RFC 9309: the IETF Robots Exclusion Protocol standard of September 2022, which defines groups, Allow and Disallow rules and longest-match precedence. Source: https://agentreadycrawlers.com/glossary/rfc-9309 Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # RFC 9309 In short The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching. The block rates apply its matching rules, a group naming the token first and the \* group otherwise, and the builder's output is tested against its grammar. Related: Block-rate method and robots.txt builder. ## Definition The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching. ## Sourced example RFC 9309 says crawlers should match the product token in the User-agent line case-insensitively. [Source](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04. ## How it shows up here The block rates apply its matching rules, a group naming the token first and the \* group otherwise, and the builder's output is tested against its grammar. - [Block-rate method](https://agentreadycrawlers.com/method/#block-rate-method) - [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) Previous: [Reverse DNS verification](https://agentreadycrawlers.com/glossary/reverse-dns-verification/). Next: [robots.txt](https://agentreadycrawlers.com/glossary/robots-txt/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "robots.txt: definition | Agent Ready Crawlers" description: "robots.txt: A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token." canonical: https://agentreadycrawlers.com/glossary/robots-txt site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # robots.txt: definition | Agent Ready Crawlers > robots.txt: A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Source: https://agentreadycrawlers.com/glossary/robots-txt Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # robots.txt In short A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Standardised as RFC 9309; it states a preference and enforces nothing. The crawl parsed a robots.txt from 79.2 percent of the 10,886 domains that answered, and 67.4 percent of top-10,000 files carry a Sitemap line. ## Definition A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Standardised as RFC 9309; it states a preference and enforces nothing. ## Sourced example Anthropic documents "User-agent: ClaudeBot" followed by "Disallow: /" to block ClaudeBot site-wide, per subdomain. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. ## How it shows up here The crawl parsed a robots.txt from 79.2 percent of the 10,886 domains that answered, and 67.4 percent of top-10,000 files carry a Sitemap line. - [How block rates are computed](https://agentreadycrawlers.com/method/#block-rate-method) - [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) Previous: [RFC 9309](https://agentreadycrawlers.com/glossary/rfc-9309/). Next: [Robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Robots.txt token: definition | Agent Ready Crawlers" description: "Robots.txt token: The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests." canonical: https://agentreadycrawlers.com/glossary/robots-txt-token site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Robots.txt token: definition | Agent Ready Crawlers > Robots.txt token: The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests. Source: https://agentreadycrawlers.com/glossary/robots-txt-token Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Robots.txt token In short The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests, and some tokens control use rather than crawling. 47 Index rows have a token and 23 have none. 32.1 percent of top-10,000 robots.txt files name at least one AI crawler token. Example: Google-Extended has no HTTP user agent string. ## Definition The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests, and some tokens control use rather than crawling. ## Sourced example Google-Extended has no HTTP user agent string; it is a control token read against Googlebot crawls. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ## How it shows up here 47 Index rows have a token and 23 have none. 32.1 percent of top-10,000 robots.txt files name at least one AI crawler token. - [Tokens versus user-agent strings](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings/) - [Crawler Index](https://agentreadycrawlers.com/crawlers/) Previous: [robots.txt](https://agentreadycrawlers.com/glossary/robots-txt/). Next: [Search-index crawler](https://agentreadycrawlers.com/glossary/search-index-crawler/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Search-index crawler: definition | Agent Ready Crawlers" description: "Search-index crawler: A crawler that builds an index used to surface and cite pages in search or AI answers. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/search-index-crawler site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Search-index crawler: definition | Agent Ready Crawlers > Search-index crawler: A crawler that builds an index used to surface and cite pages in search or AI answers. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/search-index-crawler Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Search-index crawler In short A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training. 15 Index rows are filed under Search index. Among measured tokens the range is 2.8 to 14.8 percent of top-10,000 sites refusing them. Related: All search index rows and OAI-SearchBot profile. ## Definition A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training. ## Sourced example PerplexityBot surfaces sites in Perplexity search results and is not used to crawl content for foundation models. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. ## How it shows up here 15 Index rows are filed under Search index. Among measured tokens the range is 2.8 to 14.8 percent of top-10,000 sites refusing them. - [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [OAI-SearchBot profile](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Previous: [Robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/). Next: [Signature-Agent header](https://agentreadycrawlers.com/glossary/signature-agent-header/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Signature-Agent header: definition | Agent Ready Crawlers" description: "Signature-Agent header: The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys." canonical: https://agentreadycrawlers.com/glossary/signature-agent-header site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Signature-Agent header: definition | Agent Ready Crawlers > Signature-Agent header: The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys. Source: https://agentreadycrawlers.com/glossary/signature-agent-header Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Signature-Agent header In short The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys. Log it on every request: it is the only field that tells you which key directory to fetch. Example: ChatGPT's Cloud browser sends Signature-Agent: "https://chatgpt.com". Related: Verify Web Bot Auth signatures and ChatGPT agent profile. ## Definition The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys. ## Sourced example ChatGPT's Cloud browser sends Signature-Agent: "https://chatgpt.com". [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ## How it shows up here Log it on every request: it is the only field that tells you which key directory to fetch. - [Verify Web Bot Auth signatures](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) - [ChatGPT agent profile](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Previous: [Search-index crawler](https://agentreadycrawlers.com/glossary/search-index-crawler/). Next: [Signed agent](https://agentreadycrawlers.com/glossary/signed-agent/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Signed agent: definition | Agent Ready Crawlers" description: "Signed agent: Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy." canonical: https://agentreadycrawlers.com/glossary/signed-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Signed agent: definition | Agent Ready Crawlers > Signed agent: Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy. Source: https://agentreadycrawlers.com/glossary/signed-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Signed agent In short Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy. Rows marked Cloudflare signed agent in the Index verify this way; the profile of each says what was read. Example: The first signed-agent cohort on 2025-08-28: ChatGPT agent, Goose, Browserbase and Anchor Browser. ## Definition Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy. ## Sourced example The first signed-agent cohort on 2025-08-28: ChatGPT agent, Goose, Browserbase and Anchor Browser. [Source](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. ## How it shows up here Rows marked Cloudflare signed agent in the Index verify this way; the profile of each says what was read. - [Goose profile](https://agentreadycrawlers.com/crawlers/goose/) - [How the verified bot list works](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/) Previous: [Signature-Agent header](https://agentreadycrawlers.com/glossary/signature-agent-header/). Next: [Spoofed crawler](https://agentreadycrawlers.com/glossary/spoofed-crawler/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Spoofed crawler: definition | Agent Ready Crawlers" description: "Spoofed crawler: A request that copies a known crawler's user agent to get treatment meant for the real bot. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/spoofed-crawler site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Spoofed crawler: definition | Agent Ready Crawlers > Spoofed crawler: A request that copies a known crawler's user agent to get treatment meant for the real bot. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/spoofed-crawler Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Spoofed crawler In short A request that copies a known crawler's user agent to get treatment meant for the real bot. The Index pairs every row with the way to check it: 17 by IP list, 9 by reverse DNS, 8 by signature. Example: HUMAN found 5.7 percent of traffic presenting an AI crawler or scraper user agent was spoofed. ## Definition A request that copies a known crawler's user agent to get treatment meant for the real bot. ## Sourced example HUMAN found 5.7 percent of traffic presenting an AI crawler or scraper user agent was spoofed. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04. ## How it shows up here The Index pairs every row with the way to check it: 17 by IP list, 9 by reverse DNS, 8 by signature. - [Verification Ledger](https://agentreadycrawlers.com/verification/) - [Read agent traffic in logs](https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs/) Previous: [Signed agent](https://agentreadycrawlers.com/glossary/signed-agent/). Next: [Training crawler](https://agentreadycrawlers.com/glossary/training-crawler/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Training crawler: definition | Agent Ready Crawlers" description: "Training crawler: A crawler whose fetched content may be used to train AI models. 9 Index rows are filed under Training." canonical: https://agentreadycrawlers.com/glossary/training-crawler site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Training crawler: definition | Agent Ready Crawlers > Training crawler: A crawler whose fetched content may be used to train AI models. 9 Index rows are filed under Training. Source: https://agentreadycrawlers.com/glossary/training-crawler Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Training crawler In short A crawler whose fetched content may be used to train AI models. Most large operators now separate it from their search crawler so sites can refuse one and keep the other. 9 Index rows are filed under Training. The most refused measured token is CCBot at 19.4 percent of top-10,000 sites. Related: All training rows and Allow search, refuse training. ## Definition A crawler whose fetched content may be used to train AI models. Most large operators now separate it from their search crawler so sites can refuse one and keep the other. ## Sourced example GPTBot crawls content that may be used in training OpenAI foundation models. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. ## How it shows up here 9 Index rows are filed under Training. The most refused measured token is CCBot at 19.4 percent of top-10,000 sites. - [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Allow search, refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) Previous: [Spoofed crawler](https://agentreadycrawlers.com/glossary/spoofed-crawler/). Next: [User-agent string](https://agentreadycrawlers.com/glossary/user-agent-string/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "User-agent string: definition | Agent Ready Crawlers" description: "User-agent string: The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures." canonical: https://agentreadycrawlers.com/glossary/user-agent-string site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # User-agent string: definition | Agent Ready Crawlers > User-agent string: The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures. Source: https://agentreadycrawlers.com/glossary/user-agent-string Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # User-agent string In short The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures. Each profile prints the string the operator documents. For the 23 rows with no robots.txt token the field says so. Example: OAI-SearchBot sends a Chrome-like string ending "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot". Related: Tokens versus user-agent strings and GPTBot profile. ## Definition The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures. ## Sourced example OAI-SearchBot sends a Chrome-like string ending "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot". [Source](https://platform.openai.com/docs/bots), read 2026-11-04. ## How it shows up here Each profile prints the string the operator documents. For the 23 rows with no robots.txt token the field says so. - [Tokens versus user-agent strings](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings/) - [GPTBot profile](https://agentreadycrawlers.com/crawlers/gptbot/) Previous: [Training crawler](https://agentreadycrawlers.com/glossary/training-crawler/). Next: [User-triggered fetcher](https://agentreadycrawlers.com/glossary/user-triggered-fetcher/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "User-triggered fetcher: definition | Agent Ready Crawlers" description: "User-triggered fetcher: A fetch made because a person asked an assistant something, such as reading a linked page. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/user-triggered-fetcher site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # User-triggered fetcher: definition | Agent Ready Crawlers > User-triggered fetcher: A fetch made because a person asked an assistant something, such as reading a linked page. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/user-triggered-fetcher Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # User-triggered fetcher In short A fetch made because a person asked an assistant something, such as reading a linked page. Operators often say robots.txt may not apply because a user initiated it. 8 rows are live answer fetchers. Example: Perplexity says Perplexity-User generally ignores robots.txt since a user requested the fetch. Related: All live answer fetchers and ChatGPT-User profile. ## Definition A fetch made because a person asked an assistant something, such as reading a linked page. Operators often say robots.txt may not apply because a user initiated it. ## Sourced example Perplexity says Perplexity-User generally ignores robots.txt since a user requested the fetch. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. ## How it shows up here 8 rows are live answer fetchers. Their operators say robots.txt may not apply, yet 9.7 percent of top-10,000 sites refuse ChatGPT-User, 8.3 percent Perplexity-User and 8.5 percent Meta-ExternalFetcher. - [All live answer fetchers](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [ChatGPT-User profile](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Previous: [User-agent string](https://agentreadycrawlers.com/glossary/user-agent-string/). Next: [Verified bot](https://agentreadycrawlers.com/glossary/verified-bot/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Verified bot: definition | Agent Ready Crawlers" description: "Verified bot: A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/verified-bot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Verified bot: definition | Agent Ready Crawlers > Verified bot: A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/verified-bot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Verified bot In short A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature, and lists in a directory site owners can write rules against. The profile of each row quotes Cloudflare's status where it was read; most rows say it was not checked. Example: Cloudflare Radar lists GPTBot as Verified (read 2026-11-04). ## Definition A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature, and lists in a directory site owners can write rules against. ## Sourced example Cloudflare Radar lists GPTBot as Verified (read 2026-11-04). [Source](https://radar.cloudflare.com/bots/directory/gptbot), read 2026-11-04. ## How it shows up here The profile of each row quotes Cloudflare's status where it was read; most rows say it was not checked. - [How Cloudflare's verified bot list works](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/) - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) Previous: [User-triggered fetcher](https://agentreadycrawlers.com/glossary/user-triggered-fetcher/). Next: [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [radar.cloudflare.com: gptbot](https://radar.cloudflare.com/bots/directory/gptbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Visa Trusted Agent Protocol: definition | Agent Ready Crawlers" description: "Visa Trusted Agent Protocol: Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed." canonical: https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Visa Trusted Agent Protocol: definition | Agent Ready Crawlers > Visa Trusted Agent Protocol: Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed. Source: https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Visa Trusted Agent Protocol In short Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. The Verification Ledger found no public operator-level registry of agents registered with it. Example: Visa publishes a reference implementation with agent-registry and cdn-proxy folders on GitHub. Related: Let TAP and Agent Pay through and Akamai profile. ## Definition Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. ## Sourced example Visa publishes a reference implementation with agent-registry and cdn-proxy folders on GitHub. [Source](https://github.com/visa/trusted-agent-protocol), read 2026-11-04. ## How it shows up here The Verification Ledger found no public operator-level registry of agents registered with it. - [Let TAP and Agent Pay through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/) - [Akamai profile](https://agentreadycrawlers.com/vendors/akamai/) Previous: [Verified bot](https://agentreadycrawlers.com/glossary/verified-bot/). Next: [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Web Bot Auth: definition | Agent Ready Crawlers" description: "Web Bot Auth: An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys. Read 2026-11-04." canonical: https://agentreadycrawlers.com/glossary/web-bot-auth site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Web Bot Auth: definition | Agent Ready Crawlers > Web Bot Auth: An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys. Read 2026-11-04. Source: https://agentreadycrawlers.com/glossary/web-bot-auth Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Web Bot Auth In short An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys, so sites can verify the operator without trusting IPs or user agents. 8 Index rows sign with Web Bot Auth, and 4 operators in the ledger served a key directory on 2026-11-04. ## Definition An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys, so sites can verify the operator without trusting IPs or user agents. ## Sourced example Google signs a subset of Google-Agent requests with Web Bot Auth as https://agent.bot.goog. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth), read 2026-11-04. ## How it shows up here 8 Index rows sign with Web Bot Auth, and 4 operators in the ledger served a key directory on 2026-11-04. - [Verify Web Bot Auth signatures](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) - [Verification Ledger](https://agentreadycrawlers.com/verification/) - [Web Bot Auth working-group draft reaches -01](https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01/) Previous: [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/). Next: [Accountable designation](https://agentreadycrawlers.com/glossary/accountable-designation/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Guides for crawler and agent traffic | Agent Ready Crawlers" description: "14 guides on AI crawler traffic: logs, robots.txt tokens, Web Bot Auth, signed commerce agents, pay per crawl, Content-Signals and testing access." canonical: https://agentreadycrawlers.com/guides site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Guides" --- # Guides for crawler and agent traffic | Agent Ready Crawlers > 14 guides on AI crawler traffic: logs, robots.txt tokens, Web Bot Auth, signed commerce agents, pay per crawl, Content-Signals and testing access. Source: https://agentreadycrawlers.com/guides Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Guides In short 14 guides for the people who own robots.txt and the WAF: reading agent traffic in logs, allowing search while refusing training, verifying Web Bot Auth, letting signed commerce agents through, testing what a 403 does and testing whether agents can reach a site. Each ends with its sources and dates. - [**How to read agent traffic in your logs**](https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs/) A user-agent string is a claim. A log line becomes evidence when the token, the source address and, for signed agents, the Signature-Agent header agree. - [**How to allow search and answer bots while refusing training**](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) Operators have split search from training into separate tokens. Most of the work is naming the training tokens and leaving the search and fetch tokens alone. - [**How to verify Web Bot Auth signatures**](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) Web Bot Auth lets an operator prove a request is its own with a signature. Verification is five steps and the first is reading one header. - [**How to let Visa TAP and Mastercard Agent Pay agents through**](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/) Visa and Mastercard sign their agents' requests, so the question for a merchant is whether the CDN or bot manager already checks those signatures. - [**What pay per crawl pays and who is paying**](https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays/) Three ways to charge a crawler exist today. None publishes payout totals, so what they pay publishers is not yet public. - [**How to test what a 403 does to an AI answer about you**](https://agentreadycrawlers.com/guides/test-what-a-403-does/) A 403 to an agent is invisible in your analytics and decisive for the answer. The test takes an afternoon and needs no tool. - [**The difference between a crawler and a browsing agent**](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent/) A crawler fetches on a schedule under its own name. A browsing agent acts for one person and often borrows that person's browser. - [**How robots.txt tokens differ from user-agent strings**](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings/) The word after User-agent in robots.txt is not what you see in the logs, and some tokens never appear there at all. - [**How Cloudflare's verified bot list works**](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/) Cloudflare's verified list is the closest thing to a shared register of good bots. How a bot gets on it decides what your rules can trust. - [**What Content-Signals changes**](https://agentreadycrawlers.com/guides/what-content-signals-change/) Content-Signal lines add a vocabulary for what a site allows after it has been crawled. They state a preference and enforce nothing. ## Added on 2026-11-05 - [**How to test whether AI agents can reach your site**](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) Read robots.txt, request the home page as each agent, check your logs, then run the scan we recommend. - [**Which AI bots to allow, by type of site**](https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type/) Five kinds of site, the purposes to allow or refuse for each, and a robots.txt generated from the Index for every one. - [**403 to AI agents: what the crawl and Ora show**](https://agentreadycrawlers.com/guides/403-to-ai-agents/) What a 403 to an agent user agent does, the share of home pages that answer one, and how to find yours. - [**The Ora checks that touch bot access**](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/) Seven checks from Ora's catalog that read robots.txt, user agents and the home page, with Ora's own fix advice. ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Forget IPs: using cryptography to verify bot and agent traffic](https://blog.cloudflare.com/web-bot-auth/) read 2026-11-04 - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 - [ora.ai: agentready launch](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=directory) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "403 to AI agents: what the crawl and Ora show | Agent Ready Crawlers" description: "What a 403 to an AI agent does, per Ora's lab, how many top home pages answer 401 or 403 in our crawl, and how to find your own refusals. Read 2026-11-05." canonical: https://agentreadycrawlers.com/guides/403-to-ai-agents site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-05 section: "Guides" --- # 403 to AI agents: what the crawl and Ora show | Agent Ready Crawlers > What a 403 to an AI agent does, per Ora's lab, how many top home pages answer 401 or 403 in our crawl, and how to find your own refusals. Read 2026-11-05. Source: https://agentreadycrawlers.com/guides/403-to-ai-agents Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # 403 to AI agents In short Ora's lab found (2026-10-19) that returning 403 to agent user agents left both test clients at 0 of 5 retrieved answers; that is a vendor finding on one docs site. In our crawl 17.2 percent of top-10,000 home pages answered 401 or 403 to a labelled non-agent user agent, which says how common refusals are, not how many reach agents. ## What Ora's lab found Ora's research lab served one docs site under 19 configurations and asked two clients to pull a planted answer from it: a plain fetch, like most agent tools, and a client that runs JavaScript. It made 190 probes, five topics per client per configuration. Only two kinds of change stopped retrieval (Ora, 19 October 2026; a vendor finding, read 2026-11-05). | Site configuration | Plain fetch | Runs JavaScript | | --- | --- | --- | | Answer only in JavaScript (1 configuration) | 0 of 5 | 5 of 5 | | 403 to agent user agents (2 configurations) | 0 of 5 | 0 of 5 | | Every other configuration (16, including llms.txt, JSON-LD, sitemaps and markdown mirrors) | 5 of 5 | 5 of 5 | The same post reports a second result: across 1,056 business sites, answers written by an agent that could not read the site were 3.7 times as likely to miss every fact the user asked for, and agents clearly recommended agent-ready businesses almost twice as often. Both are Ora's claims about its own sample. Nothing here tests them, and a docs site in a lab is not your site. ## How common a refusal is in the crawl The crawl fetched 10,565 home pages with its own labelled user agent. 12.9 percent answered 401 or 403, 17.2 percent of the 7,545 fetched in the top 10,000 (13.0 percent in ranks 1 to 1,000 and 17.6 percent in ranks 1,001 to 10,000). 644 home pages, 6.1 percent, carried a Cloudflare challenge header. Home pages that answered 401 or 403 to the crawl user agent, crawl of 2026-11-04. Categories with at least 30 home pages fetched; the eight highest shown. | Group | 401 or 403 share | | --- | --- | | All crawled | 12.9% | | Top 10,000 | 17.2% | | News & Intelligence | 25.7% | | Gaming & Media | 12.9% | | CRM | 7.1% | | HR & Recruiting | 5.3% | | Communication & Support | 4.7% | | Education | 4.3% | | Marketing | 4.1% | | Consumer | 4.0% | Read the share as the size of the problem space and no more. The crawl identifies itself as `agent-readiness-crawl`, not as ChatGPT-User or PerplexityBot, so a site that refuses it may admit the agents, and a site that admits it may refuse them. The crawl also skipped 259 domains whose robots.txt disallowed it, and those are in no share above. ## Where the 403 is the refusal that counts For fetchers that a person triggers, operators say robots.txt may not decide whether the request is made. For these 7 rows, whatever your server answers is the whole policy. | Fetcher | Operator | What the operator says | Verification | | --- | --- | --- | --- | | ChatGPT-User | OpenAI | because these actions are initiated by a user, robots.txt rules may not apply. | IP list | | Perplexity-User | Perplexity | since a user requested the fetch, this fetcher generally ignores robots.txt rules. | IP list | | Google-Agent | Google | Google lists it as a user-triggered fetcher; these fetchers generally ignore robots.txt because the fetch was requested by a user. | Web Bot Auth | | Google-GeminiNotebook | Google | generally ignores robots.txt, per Google. | IP list | | Amzn-User | Amazon | because actions can be initiated by a user, Amzn-User may not follow all robots.txt directives. | IP list | | Meta-ExternalFetcher | Meta | may bypass robots.txt because it performs fetches requested by the user. | None published | | FacebookExternalHit | Meta | might bypass robots.txt when performing security or integrity checks. | None published | How each row's robots.txt statement is read is on [the method page](https://agentreadycrawlers.com/method/#index-method). ## Find your own refusals Start from the log, since it shows what real agents received. In the common combined log format the status is the ninth field, so this lists refusals to the agents named in Ora's test: ``` awk '$9 == 403 || $9 == 401' access.log | grep -E 'ChatGPT-User|OAI-SearchBot|Claude-User|PerplexityBot' ``` Then reproduce one with the commands in [step 2 of the testing guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/#step-2). Ora's bot-detection check does the same from its network and counts a 401, 402, 403, 451 or a bot challenge as blocked when a normal browser visit gets in (Ora's methodology, read 2026-11-05). ## What to change Allow by verified identity instead of by string. All four agents in Ora's test, ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, publish an IP list in the Index, so an address rule is possible; Ora's own advice for Cloudflare sites is to allow verified bots, since a user-agent match can be faked. After the change, repeat the test and the scan. [The Ora checks that touch bot access](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/) lists the four checks involved. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). **Limits.** Ora reads public URLs only, and what it records as blocked comes from its own network. It says the same scan run twice converges within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)), and the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)). ## Sources - [Ora's lab: AgentReady launch, 19 Oct 2026 (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora's check catalog (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Block Rate Report, edition 2026-11-04 (this site)](https://agentreadycrawlers.com/block-rates/2026-11-04/) read 2026-11-04 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Allow search, refuse training | Agent Ready Crawlers" description: "How to allow search and answer bots while refusing training: the tokens to name, the control tokens that never crawl, and the Bing gap." canonical: https://agentreadycrawlers.com/guides/allow-search-refuse-training site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # Allow search, refuse training | Agent Ready Crawlers > How to allow search and answer bots while refusing training: the tokens to name, the control tokens that never crawl, and the Bing gap. Source: https://agentreadycrawlers.com/guides/allow-search-refuse-training Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How to allow search and answer bots while refusing training In short Name the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) in Disallow groups and leave search and user fetchers allowed. OpenAI treats OAI-SearchBot and GPTBot as independent settings: a site can allow search while disallowing training. Google-Extended controls Gemini training and grounding and does not affect Search inclusion or ranking. Operators have split search from training into separate tokens. Most of the work is naming the training tokens and leaving the search and fetch tokens alone. ## In practice 1. Name the training tokens in one Disallow group: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot. The robots.txt builder writes this group for you. 2. Leave the search tokens allowed. OpenAI treats OAI-SearchBot and GPTBot as independent settings, so a site can appear in ChatGPT search while refusing training. 3. Know which tokens never crawl. Google-Extended and Applebot-Extended are control tokens: refusing them opts out of training while pages stay in Google Search and Apple search. 4. Split Amazon the same way. Amazonbot may train; Amzn-SearchBot and Amzn-User are not used for training. 5. Plan for the Bing gap. Bing has no robots.txt training token yet; NOARCHIVE is today's opt-out, and Cloudflare says robots.txt support is targeted for early 2027. 6. If you use Cloudflare, its Disallow AI Training setting (2026-09-15) writes the no-training preference into robots.txt and keeps Accountable mixed-use crawlers in search. ### Training tokens in the Index, refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / Generated from the Index: 9 tokens with the purpose Training. Search and fetch tokens are left out on purpose. ## What the sources say - Primary source OpenAI treats OAI-SearchBot and GPTBot as independent settings: a site can allow search while disallowing training. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. - Primary source Google-Extended controls Gemini training and grounding and does not affect Search inclusion or ranking. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. - Primary source Applebot-Extended does not crawl; disallowing it opts out of training while pages stay in Apple search. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. - Primary source Amazon separates Amazonbot (may train) from Amzn-SearchBot and Amzn-User (not used for training). [Source](https://developer.amazon.com/amazonbot), read 2026-11-04. - Primary source Bing has no robots.txt training token yet; NOARCHIVE is today's opt-out and robots.txt support is targeted for early 2027, per Cloudflare. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. - Primary source Cloudflare's Disallow AI Training (2026-09-15) writes the no-training preference into robots.txt and keeps Accountable mixed-use crawlers in search. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## Related - [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) - [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Google-Extended profile](https://agentreadycrawlers.com/crawlers/google-extended/) - [Disallow AI Training](https://agentreadycrawlers.com/glossary/disallow-ai-training/) - [Which AI bots to allow, by type of site](https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type/) - [Training and search policy in robots.txt](https://agentreadycrawlers.com/block-rates/training-and-search/) ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "How Cloudflare's verified bot list works | Agent Ready Crawlers" description: "How Cloudflare's verified bot list works: BotBase submission, Web Bot Auth or IP validation, behaviour classes and the rules you can write on them." canonical: https://agentreadycrawlers.com/guides/cloudflare-verified-bots site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # How Cloudflare's verified bot list works | Agent Ready Crawlers > How Cloudflare's verified bot list works: BotBase submission, Web Bot Auth or IP validation, behaviour classes and the rules you can write on them. Source: https://agentreadycrawlers.com/guides/cloudflare-verified-bots Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How Cloudflare's verified bot list works In short Operators submit bots through BotBase; Cloudflare validates them by Web Bot Auth or IP and classifies their behaviour; site owners write rules on the verified category. Verified bots must validate with Web Bot Auth or IP validation; since 2026-07-01 signed agents sit under the verified bot umbrella. Cloudflare's behaviour taxonomy has 11 entries including Search, Agent, Training and Transact. Cloudflare's verified list is the closest thing to a shared register of good bots. How a bot gets on it decides what your rules can trust. ## In practice 1. A bot operator submits through BotBase. BotBase for Operators (2026-08-28) added submission status, editing and a behaviour model for operators. 2. Cloudflare validates the bot by Web Bot Auth or by IP validation. Since 2026-07-01 signed agents sit under the verified bot umbrella. 3. Cloudflare classifies the bot's behaviour. Its taxonomy has 11 entries, including Search, Agent, Training and Transact. 4. You write rules on the verified category or on a specific bot. OpenAI documents the identifiers for its Cloud browser: bot tag chatgpt-agent and detection ID 129220581. 5. Check Radar for the status of a given bot. Cloudflare Radar lists GPTBot and ChatGPT agent as Verified (read 2026-11-04); the same page for other bots was not confirmed in our reading. ## What the sources say - Primary source Verified bots must validate with Web Bot Auth or IP validation; since 2026-07-01 signed agents sit under the verified bot umbrella. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04. - Primary source BotBase for Operators (2026-08-28) added submission status, editing and a behaviour model for operators. [Source](https://blog.cloudflare.com/botbase-for-operators/), read 2026-11-04. - Primary source Cloudflare's behaviour taxonomy has 11 entries including Search, Agent, Training and Transact. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04. - Primary source OpenAI documents the Cloudflare identifiers for its Cloud browser: bot tag chatgpt-agent and detection ID 129220581. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. - Primary source Cloudflare Radar lists GPTBot and ChatGPT agent as Verified (radar.cloudflare.com/bots/directory, read 2026-11-04). [Source](https://radar.cloudflare.com/bots/directory/gptbot), read 2026-11-04. ## Related - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) - [Verified bot](https://agentreadycrawlers.com/glossary/verified-bot/) - [BotBase](https://agentreadycrawlers.com/glossary/botbase/) - [Signed agent](https://agentreadycrawlers.com/glossary/signed-agent/) ## Sources - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [BotBase for Operators](https://blog.cloudflare.com/botbase-for-operators/) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [radar.cloudflare.com: gptbot](https://radar.cloudflare.com/bots/directory/gptbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Crawler versus browsing agent | Agent Ready Crawlers" description: "The difference between a crawler and a browsing agent: schedule against task, token against none, and what that means for robots.txt." canonical: https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # Crawler versus browsing agent | Agent Ready Crawlers > The difference between a crawler and a browsing agent: schedule against task, token against none, and what that means for robots.txt. Source: https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # The difference between a crawler and a browsing agent In short Crawlers fetch on a schedule under their own token; browsing agents act for one person, often from that person's browser, and many carry no token at all. Cloudflare classifies bots by three controllable behaviours: Search, Training and Agent (user-directed agents visiting for a human). Meta-ExternalFetcher may bypass robots.txt because it fetches links at a user's request. A crawler fetches on a schedule under its own name. A browsing agent acts for one person and often borrows that person's browser. ## In practice 1. Sort by behaviour, as Cloudflare does: Search, Training and Agent, where Agent means user-directed agents visiting for a human. 2. Look for a signature. Cloudflare's signed agents are directed by end users and sign with Web Bot Auth; the first cohort was ChatGPT agent, Goose, Browserbase and Anchor Browser. Google-Agent is used by agents hosted on Google infrastructure, with ranges in user-triggered-agents.json. 3. Expect user-triggered fetchers to differ on robots.txt. Meta-ExternalFetcher may bypass it because it fetches links at a user's request. 4. Expect some agents to leave no mark. Verisoul reports Comet browses from the user's device with Chrome's user agent, no signature and the user's IP. 5. Set policy per behaviour, not per name: allow or refuse the search and agent behaviours at the WAF, because a robots.txt line cannot address an agent with no token. ## What the sources say - Primary source Cloudflare classifies bots by three controllable behaviours: Search, Training and Agent (user-directed agents visiting for a human). [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. - Primary source Cloudflare's signed agents are directed by end users, sign with Web Bot Auth and follow its signed agent policy; the first cohort was ChatGPT agent, Goose, Browserbase and Anchor Browser. [Source](https://blog.cloudflare.com/signed-agents/), read 2026-11-04. - Primary source Google-Agent is used by agents hosted on Google infrastructure to navigate and act on user request, from user-triggered-agents.json ranges. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04. - Primary source Meta-ExternalFetcher may bypass robots.txt because it fetches links at a user's request. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04. - Secondary source Verisoul reports Comet browses from the user's device with Chrome's user agent, no signature and the user's IP. [Source](https://www.verisoul.ai/agents/perplexity-comet), read 2026-11-04. ## Related - [All task agents](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Comet profile](https://agentreadycrawlers.com/crawlers/perplexity-comet/) - [Google-Agent profile](https://agentreadycrawlers.com/crawlers/google-agent/) - [Browsing agent](https://agentreadycrawlers.com/glossary/browsing-agent/) - [User-triggered fetcher](https://agentreadycrawlers.com/glossary/user-triggered-fetcher/) - [Agentic browsers and how they identify](https://agentreadycrawlers.com/crawlers/agentic-browsers/) ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 - [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Letting TAP and Agent Pay agents through | Agent Ready Crawlers" description: "How to let Visa TAP and Mastercard Agent Pay agents through: check whether your CDN verifies Web Bot Auth before you write custom rules." canonical: https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # Letting TAP and Agent Pay agents through | Agent Ready Crawlers > How to let Visa TAP and Mastercard Agent Pay agents through: check whether your CDN verifies Web Bot Auth before you write custom rules. Source: https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How to let Visa TAP and Mastercard Agent Pay agents through In short Both programmes sign agent requests with Web Bot Auth tags; check whether your CDN or bot manager verifies them before writing custom rules. Visa's Trusted Agent Protocol gives agents a cryptographic signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. DataDome lists TAP and KYA trust-authority integrations on its pricing comparison. Visa and Mastercard sign their agents' requests, so the question for a merchant is whether the CDN or bot manager already checks those signatures. ## In practice 1. Ask your CDN or bot manager whether it verifies Web Bot Auth. Cloudflare says TAP and Agent Pay both use it, with the tags agent-browser-auth or agent-payer-auth, and announced managed rulesets for registered agents; the current rule name is not confirmed in its documents. 2. Compare vendors on the question. Akamai and Visa announced a TAP integration with Akamai's edge bot protection on 2025-12-17. DataDome lists TAP and KYA integrations on its pricing comparison, which is a vendor claim. 3. Do not write a user-agent allow rule for these agents. A user-agent string can be copied by anyone, and the point of TAP is a cryptographic signature with timestamps that a merchant verifies. 4. If you need to test, start from Visa's reference implementation on GitHub, which has agent-registry, cdn-proxy, merchant-backend and tap-agent folders. 5. Expect no public list. The Verification Ledger found no public operator-level registry of registered agents for either programme. ## What the sources say - Primary source Visa's Trusted Agent Protocol gives agents a cryptographic signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. [Source](https://github.com/visa/trusted-agent-protocol), read 2026-11-04. - Primary source Cloudflare says TAP and Agent Pay both use Web Bot Auth, with tags agent-browser-auth or agent-payer-auth, and announced managed rulesets for registered agents. [Source](https://blog.cloudflare.com/secure-agentic-commerce/), read 2026-11-04. - Primary source Akamai and Visa announced a TAP integration with Akamai's edge bot and abuse protection on 2025-12-17. [Source](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce), read 2026-11-04. - Vendor claim DataDome lists TAP and KYA trust-authority integrations on its pricing comparison. [Source](https://datadome.co/pricing/), read 2026-11-04. - Primary source Visa publishes a reference implementation on GitHub with agent-registry, cdn-proxy, merchant-backend and tap-agent folders. [Source](https://github.com/visa/trusted-agent-protocol), read 2026-11-04. ## Related - [Cloudflare profile](https://agentreadycrawlers.com/vendors/cloudflare/) - [Akamai profile](https://agentreadycrawlers.com/vendors/akamai/) - [DataDome profile](https://agentreadycrawlers.com/vendors/datadome/) - [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/) - [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/) - [Verified bot programmes compared](https://agentreadycrawlers.com/verification/programmes-compared/) ## Sources - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "The Ora checks that touch bot access | Agent Ready Crawlers" description: "The seven Ora checks that read robots.txt, user agents and the home page, with Ora's fix advice and the matching crawl numbers. Catalog read 2026-11-05." canonical: https://agentreadycrawlers.com/guides/ora-checks-for-bot-access site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-05 section: "Guides" --- # The Ora checks that touch bot access | Agent Ready Crawlers > The seven Ora checks that read robots.txt, user agents and the home page, with Ora's fix advice and the matching crawl numbers. Catalog read 2026-11-05. Source: https://agentreadycrawlers.com/guides/ora-checks-for-bot-access Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The Ora checks that touch bot access In short Seven checks in Ora's catalog test whether crawlers and agents are let in: 2 in the Discovery layer and 5 in Access. Each is described below with Ora's fix advice and, where the crawl of 13,071 domains measured the same tokens, its numbers, from the catalog Ora served on 2026-11-05. The scan we recommend for testing agent access is [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide). Its catalog is public, so what a scan reads can be checked before running one. The catalog served on 2026-11-05 (contract 1.38.0) lists 132 checks in four layers; seven of them test whether crawlers and agents are let in. Other checks read the same files for other reasons, such as the NLWeb feed check (a schemamap line in robots.txt) and the JSON-LD check (the home page's structured data), and are left out. The page gives what each reads, Ora's own advice for failing it and, where the crawl has a matching number, that number. ## The four layers in the catalog Checks listed per layer in GET /api/checks, read 2026-11-05. A layer's points are its weight in Ora's 100-point score, not a count of checks. | Layer | Layer points | Checks listed | Bonus | Draft | | --- | --- | --- | --- | --- | | Discovery | 20 | 16 | 6 | 1 | | Access | 30 | 42 | 12 | 0 | | Usability | 40 | 68 | 37 | 12 | | Payments | 10 | 6 | 6 | 0 | ## The seven checks ### robots.txt AI crawler policy Layer and points Discovery, up to 2 points. Ora shows it as required and verified. What it reads Whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Ora says refusing training crawlers such as GPTBot and ClaudeBot never lowers it. Ora's advice Allow the three AI search crawlers on public pages, keep private and account pages out, and serve robots.txt as plain text with a 200. In the crawl OAI-SearchBot is refused at the root by 7.7 percent of the top-10,000 sites with a parsed robots.txt, Claude-SearchBot by 7.7 percent and PerplexityBot by 11.2 percent. 688 sites, 11.6 percent, refuse at least one of the three. That counts refusals in the file, which is not how Ora scores the check. [Training and search policy](https://agentreadycrawlers.com/block-rates/training-and-search/). ### Homepage not hidden from search Layer and points Discovery, up to 1 point. Ora shows it as recommended and verified. What it reads Whether the home page carries a noindex directive, in a robots meta tag or an X-Robots-Tag header. Ora's advice Remove the directive; Ora lists a staging setting left on after launch and a host that adds the header to preview builds as common causes. ### Not blocked by bot detection Layer and points Access, up to 2 points. Ora shows it as required and verified. What it reads Whether the home page answers ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot as it answers a normal browser. Ora's advice Let those agents through the WAF and bot rules; on Cloudflare, allow verified bots instead of matching user-agent strings, which anyone can fake. In the crawl 17.2 percent of the top-10,000 home pages answered the crawl's own user agent with 401 or 403. That says nothing about the four agents; [the 403 page](https://agentreadycrawlers.com/guides/403-to-ai-agents/) explains the gap. To run the same test by hand, see [step 2 of the testing guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/#step-2). ### robots.txt agent-user policy Layer and points Access, up to 2 points. Ora shows it as required and verified. What it reads The permission robots.txt declares for the home page to ChatGPT-User, Claude-User and Perplexity-User. Ora says it reports declared permission and that a robots.txt change alone does not establish live access. Ora's advice Review the matched rules, and if you mean to permit these requests, narrow the restriction while keeping private paths protected. On this site OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them, so a declared permission can differ from what the fetcher does. [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/), [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) and [Perplexity-User](https://agentreadycrawlers.com/crawlers/perplexity-user/) carry each operator's statement. ### Agent crawler reachability Layer and points Access, up to 2 points. Ora shows it as recommended and verified. What it reads Whether the home page is reachable to the same four agents that act for users and search. Ora's advice Let the four through the WAF, the bot rules and robots.txt. ### Bot-UA markdown serving Layer and points Access, up to 1 point. Ora shows it as emerging. A bonus check. What it reads Whether the server recognises an AI bot by user agent and answers with markdown even when the bot asks for HTML. Ora's advice Optional: detect bot user agents on the server and send a markdown version of the page. ### Content without JavaScript Layer and points Access, up to 3 points. Ora shows it as required and verified. What it reads Whether the home page carries its content in the raw HTML, fetched with JavaScript off. Ora's advice Put at least 500 characters of meaningful home page content in the HTML, with a clear h1, and do the same for inner pages. ## What reaches your server The bot-detection and reachability checks send requests from Ora's network. [OraBot](https://agentreadycrawlers.com/crawlers/orabot/) signs its own requests; the requests that carry other crawlers' user agents are unsigned by design, so a WAF that verifies a claimed GPTBot against OpenAI's addresses will not match them. ## Limits A check result describes what Ora saw from its network on one day. Ora reads public URLs only, says repeated scans of the same site converge within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)), has published no repeat-scan figures on the pages we read, and keeps its fixing product behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)). Ora's methodology says only checks it has verified agents rely on count toward the score; the catalog also lists emerging and draft checks, and the layer table above says how many are drafts. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [Ora's check catalog, GET /api/checks (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora's llms.txt (vendor)](https://ora.ai/llms.txt?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Block Rate Report (this site)](https://agentreadycrawlers.com/block-rates/) read 2026-11-04 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "How to read agent traffic in your logs | Agent Ready Crawlers" description: "How to read AI crawler traffic in your server logs: match the token, check the source address against the operator's list, and log Signature-Agent." canonical: https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # How to read agent traffic in your logs | Agent Ready Crawlers > How to read AI crawler traffic in your server logs: match the token, check the source address against the operator's list, and log Signature-Agent. Source: https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How to read agent traffic in your logs In short Find the token, check the source address against the operator's published list or reverse DNS, and log the Signature-Agent header so signed agents are visible. OpenAI publishes one JSON file of IP prefixes per crawler: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. Google verification is reverse DNS to googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the same IP. A user-agent string is a claim. A log line becomes evidence when the token, the source address and, for signed agents, the Signature-Agent header agree. ## In practice 1. Log the user agent, the request path, the source address and the Signature-Agent header on every request. Log paths because OpenAI and Google both note that a robots.txt fetch can carry a marker or a different string. 2. Match the token in the user agent against the [Crawler Index](https://agentreadycrawlers.com/crawlers/). A match on the string alone is unverified. 3. Check the source address. For OpenAI, compare it with the JSON file for that crawler (gptbot.json, searchbot.json, chatgpt-user.json or adsbot.json). For Google, resolve the address to googlebot.com, google.com or googleusercontent.com, then resolve that name forward and require the same address. 4. Treat copies as likely. Common Crawl warns that other crawlers falsely identify as CCBot and recommends reverse DNS to `*.crawl.commoncrawl.org`. 5. Expect some agents to look like ordinary visitors. Claude in Chrome, Comet and Gemini auto browse send the user's own user agent and IP, so a log cannot single them out. 6. For signed agents such as ChatGPT's Cloud browser, keep all three headers (Signature, Signature-Input, Signature-Agent) through every proxy, or verification fails downstream. ## What the sources say - Primary source OpenAI publishes one JSON file of IP prefixes per crawler: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. - Primary source OpenAI and Google both note that robots.txt fetches can carry a marker or a different string, so log paths alongside user agents. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. - Primary source Google verification is reverse DNS to googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the same IP. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. - Primary source ChatGPT's Cloud browser sends Signature, Signature-Input and Signature-Agent: "https://chatgpt.com"; intermediate proxies must preserve all three headers. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. - Secondary source Agents that run in the user's own browser (Claude in Chrome, Comet, Gemini auto browse) send the user's user agent and IP, so logs show them as ordinary visitors. [Source](https://www.centinelanalytica.com/blog/ai-browser-agents), read 2026-11-04. - Primary source Common Crawl warns that other crawlers falsely identify as CCBot and recommends verifying by reverse DNS to \*.crawl.commoncrawl.org. [Source](https://commoncrawl.org/ccbot), read 2026-11-04. ## Related - [The Verification Ledger](https://agentreadycrawlers.com/verification/) - [GPTBot profile](https://agentreadycrawlers.com/crawlers/gptbot/) - [CCBot profile](https://agentreadycrawlers.com/crawlers/ccbot/) - [ChatGPT agent profile](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) - [Reverse DNS verification](https://agentreadycrawlers.com/glossary/reverse-dns-verification/) - [How to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) - [The names sites write in robots.txt](https://agentreadycrawlers.com/crawlers/named-in-robots-txt/) ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04 - [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "What a 403 does to an AI answer | Agent Ready Crawlers" description: "How to test what a 403 does to an AI answer about you: fetch your pages as each crawler, compare status codes, then ask the assistant." canonical: https://agentreadycrawlers.com/guides/test-what-a-403-does site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # What a 403 does to an AI answer | Agent Ready Crawlers > How to test what a 403 does to an AI answer about you: fetch your pages as each crawler, compare status codes, then ask the assistant. Source: https://agentreadycrawlers.com/guides/test-what-a-403-does Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How to test what a 403 does to an AI answer about you In short Request your own pages with each crawler token and with a plain fetch, compare status codes, then ask the assistant about your site and see whether it quotes you. OpenAI says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. A 403 to an agent is invisible in your analytics and decisive for the answer. The test takes an afternoon and needs no tool. ## In practice 1. Request your own key pages once with a plain fetch and once with each crawler's user-agent string from the Index, and compare the status codes. Ora counts a 401, 402, 403, 451 or bot challenge to an agent, while a normal browser gets in, as blocked. 2. Ask the assistant about your site and see whether it quotes you. Ora's lab found (2026-10-19) that serving 403 to agent user agents left both a plain-fetch and a JavaScript client at 0 of 5 retrieved answers on one docs site; that is a vendor finding. 3. Read the result against what operators say. Sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. Disabling Claude-User prevents retrieval in response to user queries. 4. Remember Perplexity-User generally ignores robots.txt, so a 403 at the WAF is the only refusal it sees. 5. Compare with the field. Cloudflare reports that less than 1 percent of its sites block search bots while 17 percent block training. The crawl behind this site found 12.9 percent of home pages answering 401 or 403 to its labelled user agent. ## What the sources say - Vendor claim Ora's lab found (2026-10-19) that serving 403 to agent user agents left both a plain-fetch client and a JavaScript client at 0 of 5 retrieved answers, across 190 probes of one docs site in 19 configurations. [Source](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide), read 2026-11-04. - Vendor claim Ora's lab found (2026-10-08) that on sites that are not agent-ready, agent runs get blocked by the site 2.1 times as often and 16 percent of answers admit they could not access the information. [Source](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide), read 2026-11-04. - Primary source OpenAI says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. - Primary source Anthropic says disabling Claude-User prevents retrieval in response to user queries. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. - Primary source Perplexity-User generally ignores robots.txt, so a 403 at the WAF is the only refusal it sees. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. - Primary source Cloudflare reports less than 1 percent of its sites block search bots while 17 percent block training. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## Related - [The Block Rate Report](https://agentreadycrawlers.com/block-rates/) - [Perplexity-User profile](https://agentreadycrawlers.com/crawlers/perplexity-user/) - [OAI-SearchBot profile](https://agentreadycrawlers.com/crawlers/oai-searchbot/) - [How Ora's bot-detection check works](https://agentreadycrawlers.com/method/#ora-on-this-layer) - [403 to AI agents: what the crawl and Ora show](https://agentreadycrawlers.com/guides/403-to-ai-agents/) - [How to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) - [401 and 403 by edge and bot-product header](https://agentreadycrawlers.com/block-rates/edge-and-bot-headers/) ## Sources - [ora.ai: agentready launch](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-04 - [ora.ai: ax is the new aeo](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Test whether AI agents can reach your site | Agent Ready Crawlers" description: "How to test whether AI agents can reach your site: robots.txt, home page status per agent user agent, logs and the Ora scan, with curl commands." canonical: https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-05 section: "Guides" --- # Test whether AI agents can reach your site | Agent Ready Crawlers > How to test whether AI agents can reach your site: robots.txt, home page status per agent user agent, logs and the Ora scan, with curl commands. Source: https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # How to test whether AI agents can reach your site In short Four checks show whether AI agents reach a site: read robots.txt by RFC 9309 group matching, request the home page with each agent's user agent and compare the status with a browser's, find real agents in your logs, then run Ora's scan. Ora's catalog lists 42 Access checks and 16 Discovery checks (read 2026-11-05). A site can be open in robots.txt and closed at the WAF, or the other way round. Three questions separate the cases, and each has its own test. Policy What robots.txt permits for each agent's token. Step 1. Answer What the server and the WAF return to a request that carries an agent's user agent. Steps 2 and 3. Outcome Whether an agent that gets in can then finish something. Step 4. ## Step 1: read robots.txt the way a crawler does ``` curl -s https://example.com/robots.txt ``` A crawler follows the group that names its token and falls back to the `*` group (RFC 9309). The longest matching rule wins, and Allow wins a tie. Serve the file as plain text with a 200: Ora's advice for its robots.txt check is that a server error makes crawlers treat the whole site as blocked, and an HTML page gives them no rules. Look for these six tokens first. ChatGPT-User and Perplexity-User are fetchers a person triggers, and OpenAI and Perplexity say robots.txt may not govern them, so a clean file does not prove they get in. Anthropic says disabling Claude-User stops retrieval for user queries. | Token | Purpose | What the operator says | | --- | --- | --- | | ChatGPT-User | Live answer fetch | OpenAI: because these actions are initiated by a user, robots.txt rules may not apply. | | OAI-SearchBot | Search index | OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. | | Claude-User | Live answer fetch | Anthropic: Claude-User allows site owners to control which sites can be accessed through user-initiated requests; disabling it stops retrieval for user queries. | | Claude-SearchBot | Search index | Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. | | PerplexityBot | Search index | Perplexity recommends allowing PerplexityBot in robots.txt to appear in results and says it is not used to crawl content for AI foundation models. | | Perplexity-User | Live answer fetch | Perplexity: since a user requested the fetch, this fetcher generally ignores robots.txt rules. | ## Step 2: request the home page as each agent Send the same request with each agent's user agent and once with a browser's. The commands below take their strings from the Crawler Index; where an operator publishes no full string, the token alone is used. ``` for ua in "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \ "Claude-User" \ "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do printf '%s ' "$ua" curl -s -o /dev/null -w '%{http_code}\n' -A "$ua" https://example.com/ done printf 'browser ' curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0" https://example.com/ ``` Ora counts a 401, 402, 403, 451 or a bot challenge page for an agent as blocked when a normal browser visit gets in: a warning when some agents are blocked, a fail when all are. If the browser request is refused too, or a rate limit, server error, 404 or timeout stops the request, Ora marks that agent "can't verify" and takes no points off (Ora's methodology, read 2026-11-05). Read your own results the same way. A copied string is not a real agent. A WAF that verifies agents by address or signature may refuse the copy and still admit the real one, and Ora's methodology says as much: real agents come from their own networks and may get through. Treat a 200 as good news and a 403 as a prompt to check Step 3, not as proof. ## Step 3: find the real traffic in your logs Of the 70 rows in the Crawler Index, 17 publish an IP list, 9 a reverse DNS pattern, 8 sign with Web Bot Auth and Cloudflare's signed agent list covers 1; 35 publish nothing a site can check. Match what your logs show against the row for each token. [How to read agent traffic in your logs](https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs/) lists the fields to keep, and [the programmes compared](https://agentreadycrawlers.com/verification/programmes-compared/) shows what each method proves. ## Step 4: run the scan we recommend Steps 1 to 3 test your policy and your answers by hand. The check we recommend for the same questions, plus the outcome, is [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide). What it checks 16 Discovery checks and 42 Access checks in Ora's catalog (GET /api/checks, read 2026-11-05, contract 1.38.0). The ones this page tests by hand are **robots.txt AI crawler policy**, **Not blocked by bot detection**, **robots.txt agent-user policy** and **Agent crawler reachability**; [each is described, with Ora's fix advice](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/). How to run it Ora's instructions say to look up a cached score first, then scan: the report for a domain is at `ora.ai/score/yourdomain.com`, and the command line is `npx ax@0.10 audit yourdomain.com`, where `--min-score` fails a CI build under a score. Ora's pricing endpoint lists one plan: free, unlimited score lookups, 10 scans a minute (read 2026-11-05). What it sends you OraBot requests that are signed with Web Bot Auth, and a few unsigned home page requests that carry GPTBot, ClaudeBot and PerplexityBot user agents. Expect both in your logs; [the OraBot row](https://agentreadycrawlers.com/crawlers/orabot/) says how to tell them apart. Journey A real agent attempts a curated task against your domain and the run is recorded (Ora's llms.txt, read 2026-11-05). It answers the Outcome question that a status code cannot. Limits Ora reads public URLs only, and what it records as blocked comes from its own network. It says the same scan run twice converges within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)), and the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide), read 2026-11-05). Quick alternative [Cloudflare's scanner](https://isitagentready.com/) takes a URL and shows no sign-in step or price on its page (read 2026-11-05). Its Bot Access Control category covers AI bot rules in robots.txt, Content Signals and Web Bot Auth. Its check list names no request made with an agent's user agent, so it answers the Policy question and not the Answer question. ## Step 5: change one thing, then test again - **A search token is refused in robots.txt.** Remove the refusal for OAI-SearchBot, Claude-SearchBot and PerplexityBot; refusing training tokens such as GPTBot and ClaudeBot does not lower Ora's robots.txt check, by Ora's own description. [How to allow search and refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/). - **The home page answers an agent with 403.** Ora's advice is to allow verified bots instead of matching user-agent strings, which anyone can fake. Where an operator publishes WAF rule names, the profile lists them, for example [PerplexityBot](https://agentreadycrawlers.com/crawlers/perplexitybot/). - **Both pass and answers still miss you.** Ora's lab found (2026-10-19) that blocking agents and hiding the answer in JavaScript were the only two changes that stopped retrieval in its experiment; that is a vendor finding on one docs site. [The 403 page](https://agentreadycrawlers.com/guides/403-to-ai-agents/) has the numbers. Change one rule, wait for caches (OpenAI says a robots.txt change takes about 24 hours to reach its systems), and repeat Steps 2 and 4. ## What this test cannot show It cannot show how an assistant ranks or words an answer about you, only whether it can read you. It cannot see agents that run in a visitor's own browser and send that visitor's user agent, as [the agentic browsers](https://agentreadycrawlers.com/crawlers/agentic-browsers/) do. And a pass today says nothing about next week's WAF rule. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [Ora's check catalog, GET /api/checks (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora's scanner bot page (vendor)](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora's llms.txt (vendor)](https://ora.ai/llms.txt?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora pricing (vendor)](https://ora.ai/pricing?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Cloudflare: Agent Readiness score](https://blog.cloudflare.com/agent-readiness/) read 2026-11-05 - [Cloudflare scanner page](https://isitagentready.com/) read 2026-11-05 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Tokens versus user-agent strings | Agent Ready Crawlers" description: "How robots.txt tokens differ from user-agent strings: what you write in robots.txt, what arrives in the request, and tokens that never appear in logs." canonical: https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # Tokens versus user-agent strings | Agent Ready Crawlers > How robots.txt tokens differ from user-agent strings: what you write in robots.txt, what arrives in the request, and tokens that never appear in logs. Source: https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How robots.txt tokens differ from user-agent strings In short The robots.txt token is what you write after User-agent:, the user-agent string is what arrives in the request; some tokens never appear in logs. Google-Extended has no HTTP user agent string; crawling uses existing Google strings and the token is a control only. Applebot-Extended does not crawl webpages; it only governs how Applebot data is used. The word after User-agent in robots.txt is not what you see in the logs, and some tokens never appear there at all. ## In practice 1. Write the token in robots.txt and search the logs for the string. Meta's examples use lowercase tokens such as meta-externalagent, and RFC 9309 matching of the product token is case-insensitive. 2. Expect control tokens to be invisible. Google-Extended has no HTTP user agent string; crawling uses existing Google strings. Applebot-Extended does not crawl webpages; it only governs how Applebot data is used. 3. Do not assume the token is in the header. Firecrawl matches the token FirecrawlAgent in robots.txt but tells site owners not to assume it is the User-Agent header on the fetch. 4. Remember some crawlers have more than one token. Google says a rule needs to match only one of them. 5. Use the Index's two columns: the token column is for robots.txt, the profile's user-agent string is for your logs. ## What the sources say - Primary source Google-Extended has no HTTP user agent string; crawling uses existing Google strings and the token is a control only. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. - Primary source Applebot-Extended does not crawl webpages; it only governs how Applebot data is used. [Source](https://support.apple.com/en-us/119829), read 2026-11-04. - Primary source Firecrawl matches the token FirecrawlAgent in robots.txt but tells site owners not to assume it is the User-Agent header on the fetch. [Source](https://docs.firecrawl.dev/advanced-scraping-guide), read 2026-11-04. - Primary source Meta's robots.txt examples use lowercase tokens such as meta-externalagent; RFC 9309 matching of the product token is case-insensitive. [Source](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04. - Primary source Google says some crawlers have more than one token and a rule needs to match only one of them. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ## Related - [Google-Extended profile](https://agentreadycrawlers.com/crawlers/google-extended/) - [Applebot-Extended profile](https://agentreadycrawlers.com/crawlers/applebot-extended/) - [FirecrawlAgent profile](https://agentreadycrawlers.com/crawlers/firecrawl/) - [robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/) - [User-agent string](https://agentreadycrawlers.com/glossary/user-agent-string/) ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [docs.firecrawl.dev: advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "How to verify Web Bot Auth signatures | Agent Ready Crawlers" description: "How to verify Web Bot Auth signatures: read Signature-Agent, fetch the key directory, check the RFC 9421 signature and keep IP checks as a fallback." canonical: https://agentreadycrawlers.com/guides/verify-web-bot-auth site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # How to verify Web Bot Auth signatures | Agent Ready Crawlers > How to verify Web Bot Auth signatures: read Signature-Agent, fetch the key directory, check the RFC 9421 signature and keep IP checks as a fallback. Source: https://agentreadycrawlers.com/guides/verify-web-bot-auth Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How to verify Web Bot Auth signatures In short Read Signature-Agent, fetch the key directory at /.well-known/http-message-signatures-directory, verify the RFC 9421 signature, and fall back to IP checks for unsigned traffic. Web Bot Auth builds on HTTP Message Signatures (RFC 9421); the signer publishes Ed25519 public keys in a directory named by the Signature-Agent header. The IETF webbotauth working group adopted draft-ietf-webbotauth-httpsig-protocol; revision -01 is dated 2026-10-29. Web Bot Auth lets an operator prove a request is its own with a signature. Verification is five steps and the first is reading one header. ## In practice 1. Read the Signature-Agent header, along with Signature and Signature-Input. Compare Signature-Agent exactly with the value the operator documents. OpenAI says to confirm it matches "https://chatgpt.com", quotation marks included. 2. Fetch the key directory from the origin that header names, at `/.well-known/http-message-signatures-directory`. It holds the signer's Ed25519 public keys. 3. Verify the RFC 9421 HTTP Message Signature against those keys. Cloudflare does no nonce validation today and relies on short expires values, so reject stale signatures. 4. Keep IP and reverse DNS checks for everything unsigned. Google signs only a subset of Google-Agent requests, as https://agent.bot.goog, and says to keep them. 5. Decide policy separately. A valid signature proves who signed, not whether the request is authorised; that stays your origin's decision. ## What the sources say - Primary source Web Bot Auth builds on HTTP Message Signatures (RFC 9421); the signer publishes Ed25519 public keys in a directory named by the Signature-Agent header. [Source](https://blog.cloudflare.com/web-bot-auth/), read 2026-11-04. - Primary source The IETF webbotauth working group adopted draft-ietf-webbotauth-httpsig-protocol; revision -01 is dated 2026-10-29. [Source](https://datatracker.ietf.org/wg/webbotauth/documents/), read 2026-11-04. - Primary source OpenAI tells site owners to confirm Signature-Agent exactly matches "https://chatgpt.com", including quotation marks, before verifying. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. - Primary source Google signs only a subset of Google-Agent requests as https://agent.bot.goog and says to keep IP and reverse-DNS checks. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth), read 2026-11-04. - Primary source Cloudflare accepts Ed25519 keys and does no nonce validation today, relying on short expires values. [Source](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/), read 2026-11-04. - Secondary source A valid signature proves who signed, not whether the request is authorised; that remains origin policy, per the draft as quoted by Centinel Analytica. [Source](https://www.centinelanalytica.com/blog/ai-browser-agents), read 2026-11-04. ## Related - [Which operators publish key directories](https://agentreadycrawlers.com/verification/) - [ChatGPT agent profile](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) - [Google-Agent profile](https://agentreadycrawlers.com/crawlers/google-agent/) - [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/) - [Key directory](https://agentreadycrawlers.com/glossary/key-directory/) - [Verified bot programmes compared](https://agentreadycrawlers.com/verification/programmes-compared/) ## Sources - [Forget IPs: using cryptography to verify bot and agent traffic](https://blog.cloudflare.com/web-bot-auth/) read 2026-11-04 - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) read 2026-11-04 - [developers.cloudflare.com: web bot auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) read 2026-11-04 - [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "What Content-Signals changes | Agent Ready Crawlers" description: "What Content-Signals changes: the search, ai-input and ai-train lines in robots.txt, Cloudflare's defaults and why they enforce nothing." canonical: https://agentreadycrawlers.com/guides/what-content-signals-change site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # What Content-Signals changes | Agent Ready Crawlers > What Content-Signals changes: the search, ai-input and ai-train lines in robots.txt, Cloudflare's defaults and why they enforce nothing. Source: https://agentreadycrawlers.com/guides/what-content-signals-change Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # What Content-Signals changes In short Content-Signal lines in robots.txt state search, ai-input and ai-train preferences; they express a preference and enforce nothing. The Content Signals Policy defines three signals: search, ai-input and ai-train, written as comma-delimited yes or no values. Example from Cloudflare: Content-Signal: search=yes, ai-train=no under User-Agent: \*. Google intends to support the AIPREF training signal written as user-agent: \_\_usage\_ai-training in early 2027. Content-Signal lines add a vocabulary for what a site allows after it has been crawled. They state a preference and enforce nothing. ## In practice 1. Know the three signals: search, ai-input and ai-train, written as comma-delimited yes or no values. Cloudflare's example is `Content-Signal: search=yes, ai-train=no` under `User-Agent: *`. 2. Expect defaults to differ. Cloudflare updated managed robots.txt for over 3.8 million domains to serve search=yes and ai-train=no, and serves no ai-input value for them. 3. Do not treat the line as a block. Cloudflare says a robots.txt directive cannot identify who is crawling or stop a crawler that ignores it. 4. Watch the next vocabulary. Google intends to support the AIPREF training signal, written as `user-agent: __usage_ai-training`, in early 2027. 5. Add the line with the [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) if you want it; it sets the three values from your choices. ## What the sources say - Primary source The Content Signals Policy defines three signals: search, ai-input and ai-train, written as comma-delimited yes or no values. [Source](https://blog.cloudflare.com/content-signals-policy/), read 2026-11-04. - Primary source Example from Cloudflare: Content-Signal: search=yes, ai-train=no under User-Agent: \*. [Source](https://blog.cloudflare.com/content-signals-policy/), read 2026-11-04. - Primary source Cloudflare updated managed robots.txt for over 3.8 million domains to serve search=yes, ai-train=no, and serves no ai-input value for them. [Source](https://blog.cloudflare.com/content-signals-policy/), read 2026-11-04. - Primary source Cloudflare says a robots.txt directive cannot identify who is crawling or stop a crawler that ignores it. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. - Primary source Google intends to support the AIPREF training signal written as user-agent: \_\_usage\_ai-training in early 2027. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ## Related - [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) - [Content-Signal](https://agentreadycrawlers.com/glossary/content-signal/) - [ai-input](https://agentreadycrawlers.com/glossary/ai-input/) - [AIPREF training signal](https://agentreadycrawlers.com/glossary/aipref-training-signal/) ## Sources - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "What pay per crawl pays and who is paying | Agent Ready Crawlers" description: "What pay per crawl pays and who is paying: Cloudflare, AWS WAF and TollBit compared, and why no payout totals are public yet." canonical: https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # What pay per crawl pays and who is paying | Agent Ready Crawlers > What pay per crawl pays and who is paying: Cloudflare, AWS WAF and TollBit compared, and why no payout totals are public yet. Source: https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # What pay per crawl pays and who is paying In short Three charging paths exist today: Cloudflare pay per crawl (closed beta), AWS WAF x402 monetization (CloudFront), and partner paywalls such as TollBit; none publishes payout totals. Cloudflare pay per crawl returns HTTP 402 with a crawler-price header; crawlers retry with crawler-exact-price or send crawler-max-price up front. Three ways to charge a crawler exist today. None publishes payout totals, so what they pay publishers is not yet public. ## In practice 1. Cloudflare pay per crawl answers with HTTP 402 and a crawler-price header; a crawler retries with crawler-exact-price or sends crawler-max-price up front. Cloudflare acts as Merchant of Record, and the feature is in closed beta per documents updated 2026-07-28. 2. AWS WAF AI traffic monetization returns an x402 HTTP 402 and settles through Coinbase in stablecoins. Stripe payouts were listed as coming soon. 3. TollBit sells a Bot Paywall that publishers price by usage. TollBit reports more than 2.6 billion AI bots directed to its paywall in the first half of 2026, which is its own count. 4. Skyfire pay tokens commit the buyer's funds, so a seller that charges a valid token is guaranteed payment up to the committed amount. 5. Read the vendor pages before choosing. What each does not do is on its profile. ## What the sources say - Primary source Cloudflare pay per crawl returns HTTP 402 with a crawler-price header; crawlers retry with crawler-exact-price or send crawler-max-price up front. [Source](https://blog.cloudflare.com/introducing-pay-per-crawl/), read 2026-11-04. - Primary source Cloudflare acts as Merchant of Record; pay per crawl is in closed beta per docs updated 2026-07-28. [Source](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/), read 2026-11-04. - Primary source AWS WAF AI traffic monetization returns an x402 HTTP 402 and settles through Coinbase in stablecoins; Stripe payouts were listed as coming soon. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. - Vendor claim TollBit reports more than 2.6 billion AI bots directed to its Bot Paywall in H1 2026. [Source](https://tollbit.com/), read 2026-11-04. - Primary source Skyfire pay tokens commit buyer funds, so a seller charging a valid token is guaranteed payment up to the committed amount. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04. ## Related - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) - [AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/) - [TollBit](https://agentreadycrawlers.com/vendors/tollbit/) - [Skyfire](https://agentreadycrawlers.com/vendors/skyfire/) - [TollBit vs Cloudflare](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/) - [Pay per crawl](https://agentreadycrawlers.com/glossary/pay-per-crawl/) ## Sources - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Which AI bots to allow, by type of site | Agent Ready Crawlers" description: "Which AI bots to allow for a publisher, a docs site, a store, a logged-in app or a closed site, with a robots.txt generated from the Crawler Index." canonical: https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Guides" --- # Which AI bots to allow, by type of site | Agent Ready Crawlers > Which AI bots to allow for a publisher, a docs site, a store, a logged-in app or a closed site, with a robots.txt generated from the Crawler Index. Source: https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Which AI bots to allow, by type of site In short Five kinds of site need different answers on four purposes: training, search, live fetches and task agents. Each gets a policy table and a robots.txt generated from the Crawler Index by the same builder as the Builder page. In the crawl, 11.1 percent of top-10,000 sites refuse training and keep AI search open, and 77.3 percent refuse neither. Every crawler in the Crawler Index has one of six purposes, and operators now let a site treat them separately. The choice that matters is which purposes pay you back. This page is our reading of that trade for five kinds of site; the numbers and the operator statements are sourced, the recommendation is a judgement and says so. ## The policy for each type | Site type | Training | Search index | Live answer fetch | Task agent | Shopping agent | | --- | --- | --- | --- | --- | --- | | Publisher selling ads or subscriptions | Refuse | Allow | Allow | Allow | Allow | | Documentation or developer platform | Allow | Allow | Allow | Allow | Allow | | Online store | Refuse | Allow | Allow | Allow | Allow | | Logged-in app or regulated service | Refuse | Allow | Allow | Refuse | Refuse | | Site that wants no AI use | Refuse | Refuse | Refuse | Refuse | Refuse | ## Why each type gets its choices Publisher selling ads or subscriptions Refuse training and keep search and fetches open. OpenAI treats OAI-SearchBot and GPTBot as independent settings, and says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can still appear as navigational links. Cloudflare reports (2026-09-15) that fewer than 1 percent of its sites block search bots while 17 percent block training. The crawl agrees on the shape: 660 of 5,938 top-10,000 sites, 11.1 percent, refuse at least one of six training tokens and none of three AI search tokens. Documentation or developer platform Allow everything, because agents read docs. Ora's lab (vendor claim, 2026-10-19) found agents reached docs pages in 83 percent of traced runs and the home page in 69 percent. Refusing training is the only choice with a cost here: a model that never saw your docs cannot recall them. That is an inference, not a measurement, and a licensing concern can outweigh it. Online store The same file as a publisher, with the work moved to the WAF. 3 shopping rows exist in the Index and 0 publish a token, so decide signed commerce agents by signature: [how to let Visa TAP and Mastercard Agent Pay agents through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/). Logged-in app or regulated service Refuse training, task agents and shopping agents, and keep every private path behind authentication, since robots.txt states a preference and enforces nothing. The Index has 20 task-agent rows and 1 of them publishes a token, so the refusal in the file can address only that one. [The agentic browsers](https://agentreadycrawlers.com/crawlers/agentic-browsers/) table shows how the rest identify. Site that wants no AI use Refuse every AI purpose and accept the cost: you leave the search products too. 292 top-10,000 sites, 4.9 percent, refuse all nine training and AI search tokens the crawl measured, and 168 of those name none of the nine, so a blanket `Disallow: /` in the \* group is doing the work. ## The files Each file is the output of the [robots.txt builder](https://agentreadycrawlers.com/robots-builder/) for the choices above, with Allow lines for every other purpose. Types that share a file share a block. The files name only agents that publish a token, so they cannot address browsing agents that run in a visitor's own browser. **Publisher selling ads or subscriptions and Online store**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` **Documentation or developer platform**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: allowed User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Allow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` **Logged-in app or regulated service**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: refused User-agent: Claude-Agent Disallow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` **Site that wants no AI use**: 68 lines ``` # robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: refused User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Disallow: / # Live answer fetch: refused User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Disallow: / # Task agents: refused User-agent: Claude-Agent Disallow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: / ``` ## What no file can do A robots.txt line is a request. OpenAI and Perplexity say the fetchers a person triggers may ignore it, and a crawler that does not identify itself never reads the group you wrote for it. Pair the file with a check of what your server answers: [how to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Ora's lab: AgentReady launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Training and search policy (this site)](https://agentreadycrawlers.com/block-rates/training-and-search/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Method: index, block rates, vendors | Agent Ready Crawlers" description: "Method: how the Crawler Index, the Block Rate Report and the vendor rubric are built, what the crawl does, and why we recommend Ora as the access check." canonical: https://agentreadycrawlers.com/method site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Method" --- # Method: index, block rates, vendors | Agent Ready Crawlers > Method: how the Crawler Index, the Block Rate Report and the vendor rubric are built, what the crawl does, and why we recommend Ora as the access check. Source: https://agentreadycrawlers.com/method Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Method In short How the three assets are built: Crawler Index rows come from each operator's own pages, block rates from a crawl of 13,071 domains on 4 November 2026 parsed to RFC 9309, and vendors are graded on five documented questions. It also says why we recommend Ora as the access check. - [Index method](#index-method) - [Block-rate method](#block-rate-method) - [What our own crawl does](#own-crawl) - [Vendor rubric](#vendor-rubric) - [The check we recommend: Ora](#ora-on-this-layer) - [API and MCP server](#api) ## Index method A row is a crawler or agent that identifies itself to websites, or that an operator or a detection vendor documents well enough to describe. Every field is read from the operator's own page where one exists, with the URL and the date read. Where the operator publishes nothing, the row says "operator publishes no bot documentation" with the date checked, and any third-party source is named as such. The user-agent string and the robots.txt token are different things. The string is what a request carries; the token is what a User-agent line names, matched case-insensitively under RFC 9309. Some tokens, such as Google-Extended and Applebot-Extended, are control tokens that no request ever carries. Legacy tokens that operators no longer list stay in the Index because sites still write them. Training Collects content to train or tune a model. Refusing it is the training opt-out. Search index Builds a search index that sends visitors back. Refusing it removes pages from that product. Live answer fetch Fetches a page because a person asked an assistant about it, one request at a time. Task agent A browsing agent that clicks, fills forms and finishes a job for a person. Shopping agent An agent that compares, selects or buys goods for a person. Other Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. ### Respects robots.txt One phrase per row, from the operator's own statement, which the profile quotes with its link. "Yes, per operator" means the operator says the bot honours robots.txt. "May not apply" means the operator says a user-triggered fetch can ignore it. - Partly, because one pass skipped the check - Yes, per operator - May not apply (user-triggered) - Depends on a setting - No statement - No documentation - Not on current operator page - Reported by a third party ### Verification The method the operator publishes for the row: a published IP list, reverse DNS, a Web Bot Auth signature, Cloudflare signed-agent status, or nothing. The [Verification Ledger](https://agentreadycrawlers.com/verification/) lists every URL by operator. ## Block-rate method The block rate is computed from one crawl, read on 2026-11-04. The universe is Tranco list 746WX (the top 10,000 domains), the public Ora leaderboard's sites and the vendors and platforms this site names: 13,071 domains, of which 10,886 answered. In the main pass the crawl asks every domain for /robots.txt and fetches other paths only where the group for its own token or the \* group allows them. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains without a per-path robots.txt check; the block rates and the 401 and 403 share use only the main pass's robots.txt and home page requests. A domain counts in a rate when its robots.txt returned HTTP 200 as plain text and was parsed: 8,624 domains, 5,938 of them in the top 10,000. 767 had no robots.txt and 1,495 answered with an error, a block page or HTML; neither is in any denominator. For each token the crawl reads the file the way RFC 9309 says a crawler should: a group that names the token wins, otherwise the \* group applies; the longest matching rule wins and Allow wins a tie. A token is **refused** when the path / is disallowed. **Named** is the part of that where a group names the token, so the site chose to address it. **Via \*** is the rest: the site refuses it only because its wildcard group refuses everything. The **401 and 403 share** is the share of fetched home pages that answered 401 or 403 to the crawl's labelled user agent. Categories are the Ora public leaderboard's; a category is published when at least 30 sites have a parsed robots.txt. Rank bands use the Tranco rank. The **header signals** are response headers the crawl records, grouped by what they can tell us: some show a request was challenged, some show a bot product's header or cookie, and most show only that a CDN or host is present. Limits: one crawl from one network position, so a site that treats real crawlers differently from ours is measured as it treated us. robots.txt states a preference and enforces nothing. A 403 to our user agent does not show a 403 to every agent. The script that computes every number is `scripts/slice-crawl.mjs` in this site's repository, and a test recomputes the GPTBot rate from the raw crawl file and compares it with the published one. Row count: 13,071, read 2026-11-04. Crawl date: 2026-11-04. Next edition: after the next crawl. ## What our own crawl does The crawl's row in the Index is [agent-readiness-crawl](https://agentreadycrawlers.com/crawlers/agent-readiness-crawl/). Its user agent is `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`. The main pass read /robots.txt first and fetched each of about 18 paths per domain only where the group for agent-readiness-crawl, or the \* group, allowed it. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains with the same user agent but without a per-path robots.txt check; the crawl now checks every path for future runs. Main pass: about 18 plain GET requests per domain (robots.txt, the home page twice with one asking for text/markdown, one made-up missing path, llms.txt, llms-full.txt and 12 discovery paths), up to 100 domains at once, a 10 second timeout, no JavaScript, one request at a time per domain. Follow-up pass: ten more well-known paths on 10,508 reachable domains. Run on 2026-11-04 from a single machine. The user-agent string carries no contact URL and the crawl publishes no IP list, so it fails its own verification test: a site owner cannot confirm a request is ours except by this documentation. The follow-up pass skipped the per-path robots.txt check that the main pass applied, which is now fixed in the crawl for future runs. It stores only extracted fields and the first 200 words of a home page. Its own token is in the report as a control: its bar shows the share of sites whose wildcard group refuses everything. The ora-scanner row is a second control and is not Ora's user agent; Ora's scanner is [OraBot](https://agentreadycrawlers.com/crawlers/orabot/). ## Vendor rubric Each vendor is read from its own documentation and pricing pages, with the URL and date for every grade. We have not tested the products. Five questions are graded and two are described. Yes is 2 points, partial 1, not published 0, so the maximum is 10. Equal points share a rank. A question that is not the vendor's job counts as 0, which is why identity and monetisation products sit below the bot management suites. Tells a verified agent from a spoof Yes where the vendor documents verification by Web Bot Auth signatures, IP validation or reverse DNS. Partial where it depends on partners, lists only or an unconfirmed method. Not published where nothing is documented. Per-purpose policy (train, search, answer, act) Yes where separate controls exist for crawler purposes or for named bot categories. Partial where policy is per bot or per category without a purpose split. Charging for access Yes where charging is generally available. Partial where it is in beta or runs through a partner. Not published where the vendor documents none. Lets signed commerce agents through Yes where support is documented as shipped or announced with named integrations. Partial where support is built on Web Bot Auth but the rule is not confirmed in documents. Not published otherwise. Cost (described, not graded) Quoted from the vendor with the date read. Not graded: a free tier is not better or worse than a sales-led price for every reader. Reporting Yes where AI or bot activity dashboards, analytics or logs are documented. Partial where reporting is limited to general bot analytics. Where it runs (described, not graded) Described as the vendor documents it. Not graded. ## Ora on this layer: the check we recommend For the question this site cares about, whether agents get in, we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method). Its scan reads a public URL and scores it from 0 to 100 on four layers. In the catalog Ora served on 2026-11-05 (GET /api/checks, contract 1.38.0, 132 checks), the Discovery layer lists 16 checks and the Access layer 42. Two are the clearest cases for this site's subject. In Discovery, **robots.txt AI crawler policy** asks whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages, and Ora says blocking training crawlers such as GPTBot and ClaudeBot never lowers the score. In Access, **Not blocked by bot detection** fetches the home page as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot with each vendor's published user agent, and again as a normal browser, and counts a 401, 402, 403, 451 or bot challenge to an agent as blocked when the browser gets in (Ora's catalog and methodology, read 2026-11-05). [All seven checks that touch bot access](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/) are described on their own page. Ora's Journey goes one step further and sends a real agent on a curated task against a domain, so a run can show what a status code cannot: whether an agent that gets in can finish something. Ora's lab found (2026-10-19) that returning 403 to agent user agents left both a plain-fetch and a JavaScript-running client at 0 of 5 retrieved answers across 190 probes of one docs site in 19 configurations. That is a vendor finding, and the Block Rate Report shows how common the 403 is, not what it costs any one site; [the 403 page](https://agentreadycrawlers.com/guides/403-to-ai-agents/) sets the two side by side. The quick alternative is [Cloudflare's scanner](https://isitagentready.com/), which takes a URL and shows no sign-in step or price on its page (read 2026-11-05). Its Bot Access Control category covers AI bot rules in robots.txt, Content Signals and Web Bot Auth, and its check list names no request made with an agent's user agent. What to know before relying on Ora: it reads public URLs only, a refusal it records comes from its own network, it says the same scan run twice converges within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method)) and the pages we read give no repeat-scan figures, and its fixing product is behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method)). The CLI is `npx ax@0.10 audit yourdomain.com`. [The testing guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) shows how to run it next to the by-hand checks. Ora is in the Index as a scanner, [OraBot](https://agentreadycrawlers.com/crawlers/orabot/), read from its bot page. It is not in the vendor ranking, because it does not manage bots. ## The API, the MCP server and the ask endpoint The three datasets reach agents three more ways, all documented on the [developers page](https://agentreadycrawlers.com/developers): an MCP server at `/mcp` (Streamable HTTP, five read-only tools over the Crawler Index, the Block Rate Report and the Verification Ledger), a REST API under `/api/v1/` described by [openapi.json](https://agentreadycrawlers.com/openapi.json), and an NLWeb `/ask` endpoint that searches the rows and pages and returns schema.org objects with a source URL each. Each surface reads the published files in `/data/` at request time, so a tool answer and the page it cites come from the same edition. Nothing writes and nothing needs a credential. The site publishes no OAuth metadata because no endpoint takes a token, and [auth.md](https://agentreadycrawlers.com/auth.md) says so. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora's check catalog (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora's llms.txt (vendor)](https://ora.ai/llms.txt?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Cloudflare scanner page](https://isitagentready.com/) read 2026-11-05 - [Tranco list 746WX, top 10,000](https://tranco-list.eu/list/746WX/10000) read 2026-11-04 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "News: crawler and agent changes | Agent Ready Crawlers" description: "15 dated news items on AI crawlers, agents and bot management, from 2026-08-19 to 2026-11-02, each tied to the Crawler Index rows it changes. Read 2026-11-04." canonical: https://agentreadycrawlers.com/news site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-02 section: "News" --- # News: crawler and agent changes | Agent Ready Crawlers > 15 dated news items on AI crawlers, agents and bot management, from 2026-08-19 to 2026-11-02, each tied to the Crawler Index rows it changes. Read 2026-11-04. Source: https://agentreadycrawlers.com/news Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # News In short 15 dated changes to crawlers, agents and the vendors at the door, from 2026-08-19 to 2026-11-02, newest first. Each item names the rows in the Crawler Index it affects and links its operator or vendor source, read 2026-11-04. The latest, from 2026-11-02, is Signed-Agent-Metadata header proposed for Web Bot Auth. - 2026-11-02[**Signed-Agent-Metadata header proposed for Web Bot Auth**](https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-auth/) draft-vilgalys-webbotauth-signed-agent-metadata-00 proposes an HTTP header that carries signed metadata about the agent alongside a Web Bot Auth signature. - 2026-10-29[**Vercel splits AI crawlers from AI assistants**](https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistants/) Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans. - 2026-10-29[**Web Bot Auth working-group draft reaches -01**](https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01/) draft-ietf-webbotauth-httpsig-protocol-01, the adopted working-group draft that replaced the individual Meunier drafts, was posted on 2026-10-29. - 2026-10-27[**OpenAI updates the ChatGPT-User IP list**](https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list/) openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. - 2026-10-22[**IETF drafts propose a common IP-range format and crawler best practices**](https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-bes/) New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft. - 2026-10-21[**Anthropic's Claude Duet starts browsing with a Claude-Agent token**](https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent/) Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. - 2026-10-20[**Anthropic refreshes its crawler IP list**](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) claude.com/crawling/bots.json, the list Anthropic links for verifying ClaudeBot, Claude-User and Claude-SearchBot, carried creationTime 2026-10-20 when read, starting with 216.73.216.0/22. - 2026-10-15[**DataDome Q3 report: 25 billion AI agent requests**](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) DataDome counted nearly 25 billion AI agent requests in Q3 2026, up 41.5 percent on the quarter, with Meta-ExternalAgent and Meta-WebIndexer at 49.7 percent. - 2026-10-01[**DataDome joins Experian Agent Trust**](https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trust/) Experian announced DataDome joined its Agent Trust ecosystem, using verified Know Your Agent signals for continuous intent checks during a session. - 2026-09-17[**Arcjet launches Agent Runtime Security**](https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-security/) Arcjet added runtime controls for AI agents that observe activity through OpenTelemetry, enforce policies on tool calls, APIs and databases, and keep an audit trail. - 2026-09-15[**Cloudflare adds Disallow AI Training and the Accountable label**](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. - 2026-09-08[**Meta launches Muse, a personal agent that does not identify itself**](https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i/) Meta released Muse in the US on iOS, Android, the web and WhatsApp. - 2026-09-02[**HUMAN publishes Agent Trust Levels**](https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levels/) HUMAN's AgenticTrust now grades observed agents High, Medium or Low, with High reserved for agents verifiable by a cryptographic key on each request. - 2026-08-28[**Cloudflare opens BotBase for Operators**](https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators/) Bot operators get a dashboard home for submissions under Application Security, with status (waiting, accepted, rejected with reasons), editing of existing entries, and a behaviour model to declare how a bot uses content. - 2026-08-19[**Google retires the Google-NotebookLM user agent**](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/) Google's user-triggered fetcher list now names Google-GeminiNotebook for URLs users add as sources and marks the former Google-NotebookLM string as supported until August 2026. ## Sources - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [openai.com: chatgpt user (JSON)](https://openai.com/chatgpt-user.json) read 2026-11-04 - [verisoul.ai: claude duet](https://www.verisoul.ai/agents/claude-duet) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 - [helpnetsecurity.com: datadome ai agent traffic report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report) read 2026-11-04 - [experianplc.com: datadome joins experian agent trust ecosystem to add continuous](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous) read 2026-11-04 - [arcjet.com: introducing agent runtime security observe enforce audit](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [verisoul.ai: meta muse](https://www.verisoul.ai/agents/meta-muse) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [BotBase for Operators](https://blog.cloudflare.com/botbase-for-operators/) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Google retires Google-NotebookLM string | Agent Ready Crawlers" description: "Google retires the Google-NotebookLM user agent. The page, last updated 2026-08-19, also lists Google-Agent and its experimental Web Bot Auth identity." canonical: https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-08-19 section: "News" --- # Google retires Google-NotebookLM string | Agent Ready Crawlers > Google retires the Google-NotebookLM user agent. The page, last updated 2026-08-19, also lists Google-Agent and its experimental Web Bot Auth identity. Source: https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-08-19 # Google retires the Google-NotebookLM user agent In short 19 August 2026: Google retires the Google-NotebookLM user agent. The page, last updated 2026-08-19, also lists Google-Agent and its experimental Web Bot Auth identity. Google-GeminiNotebook is Google's live answer fetcher. Google-Agent is Google's browsing agent. Rows affected in this site: Google-GeminiNotebook and Google-Agent. Google's user-triggered fetcher list now names Google-GeminiNotebook for URLs users add as sources and marks the former Google-NotebookLM string as supported until August 2026. The page, last updated 2026-08-19, also lists Google-Agent and its experimental Web Bot Auth identity. Primary source [developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04. ## Where it stands in the Index [Google-GeminiNotebook](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Google, live answer fetch. No robots.txt token. Verification: ip list. [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Google, task agent. No robots.txt token. Verification: web bot auth. [All news](https://agentreadycrawlers.com/news/). ## Sources - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare opens BotBase for Operators | Agent Ready Crawlers" description: "Cloudflare opens BotBase for Operators. Cloudflare says accurate entries decide whether a bot earns and keeps Verified status." canonical: https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-08-28 section: "News" --- # Cloudflare opens BotBase for Operators | Agent Ready Crawlers > Cloudflare opens BotBase for Operators. Cloudflare says accurate entries decide whether a bot earns and keeps Verified status. Source: https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-08-28 # Cloudflare opens BotBase for Operators In short 28 August 2026: Cloudflare opens BotBase for Operators. Bot operators get a dashboard home for submissions under Application Security, with status (waiting, accepted, rejected with reasons), editing of existing entries, and a behaviour model to declare how a bot uses content. Cloudflare says accurate entries decide whether a bot earns and keeps Verified status. Rows affected in this site: Cloudflare. Bot operators get a dashboard home for submissions under Application Security, with status (waiting, accepted, rejected with reasons), editing of existing entries, and a behaviour model to declare how a bot uses content. Cloudflare says accurate entries decide whether a bot earns and keeps Verified status. Primary source [blog.cloudflare.com/botbase-for-operators/](https://blog.cloudflare.com/botbase-for-operators/), read 2026-11-04. ## Where it stands in the Index [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) Vendor. CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta. [All news](https://agentreadycrawlers.com/news/). ## Sources - [BotBase for Operators](https://blog.cloudflare.com/botbase-for-operators/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "HUMAN publishes Agent Trust Levels | Agent Ready Crawlers" description: "HUMAN publishes Agent Trust Levels. Customers set permissions per action such as account creation and checkout. Dated 2026-09-02." canonical: https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levels site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-09-02 section: "News" --- # HUMAN publishes Agent Trust Levels | Agent Ready Crawlers > HUMAN publishes Agent Trust Levels. Customers set permissions per action such as account creation and checkout. Dated 2026-09-02. Source: https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levels Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-09-02 # HUMAN publishes Agent Trust Levels In short 2 September 2026: HUMAN publishes Agent Trust Levels. HUMAN's AgenticTrust now grades observed agents High, Medium or Low, with High reserved for agents verifiable by a cryptographic key on each request. Customers set permissions per action such as account creation and checkout. HUMAN reports 5.7 percent of AI crawler user agents it saw were spoofed. Affects HUMAN Security. HUMAN's AgenticTrust now grades observed agents High, Medium or Low, with High reserved for agents verifiable by a cryptographic key on each request. Customers set permissions per action such as account creation and checkout. HUMAN reports 5.7 percent of AI crawler user agents it saw were spoofed. Primary source [humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04. ## Where it stands in the Index [HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/) Vendor. Bot and fraud defense vendor whose AgenticTrust module rates AI agents High, Medium or Low trust and lets sites allow or deny agent actions such as browsing, login, account creation and checkout. [All news](https://agentreadycrawlers.com/news/). ## Sources - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Meta's Muse agent does not identify itself | Agent Ready Crawlers" description: "Meta launches Muse, a personal agent that does not identify itself. Meta released Muse in the US on iOS, Android, the web and WhatsApp." canonical: https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-09-08 section: "News" --- # Meta's Muse agent does not identify itself | Agent Ready Crawlers > Meta launches Muse, a personal agent that does not identify itself. Meta released Muse in the US on iOS, Android, the web and WhatsApp. Source: https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-09-08 # Meta launches Muse, a personal agent that does not identify itself In short 8 September 2026: Meta launches Muse, a personal agent that does not identify itself. Meta released Muse in the US on iOS, Android, the web and WhatsApp. Detection vendor Verisoul reports Muse sends no agent token or signature and was 68 percent of named personal-agent sessions on its customers' sites between 29 October and 1 November 2026. Affects Muse. Meta released Muse in the US on iOS, Android, the web and WhatsApp. Detection vendor Verisoul reports Muse sends no agent token or signature and was 68 percent of named personal-agent sessions on its customers' sites between 29 October and 1 November 2026. Secondary source [verisoul.ai/agents/meta-muse](https://www.verisoul.ai/agents/meta-muse), read 2026-11-04. ## Where it stands in the Index [Muse](https://agentreadycrawlers.com/crawlers/meta-muse/) Meta, shopping agent. No robots.txt token. Verification: none published. [All news](https://agentreadycrawlers.com/news/). ## Sources - [verisoul.ai: meta muse](https://www.verisoul.ai/agents/meta-muse) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cloudflare adds Disallow AI Training | Agent Ready Crawlers" description: "Cloudflare adds Disallow AI Training and the Accountable label: a no-training line in robots.txt that keeps mixed-use crawlers in search. Dated 2026-09-15." canonical: https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-09-15 section: "News" --- # Cloudflare adds Disallow AI Training | Agent Ready Crawlers > Cloudflare adds Disallow AI Training and the Accountable label: a no-training line in robots.txt that keeps mixed-use crawlers in search. Dated 2026-09-15. Source: https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-09-15 # Cloudflare adds Disallow AI Training and the Accountable label In short 15 September 2026: Cloudflare adds Disallow AI Training and the Accountable label. Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. Block now applies to mixed-use crawlers too, Block AI Bots is deprecated, and Managed Robots.txt becomes Bot Preference Sync. Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. Block now applies to mixed-use crawlers too, Block AI Bots is deprecated, and Managed Robots.txt becomes Bot Preference Sync. Primary source [blog.cloudflare.com/accountable-mixed-use-ai-crawlers/](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## Where it stands in the Index [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) Vendor. CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta. [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) Google, search index. Token `Googlebot`. Verification: reverse dns. Refused by 2.8% of top-10,000 sites. [Applebot](https://agentreadycrawlers.com/crawlers/applebot/) Apple, search index. Token `Applebot`. Verification: reverse dns. Refused by 14.8% of top-10,000 sites. [Bingbot](https://agentreadycrawlers.com/crawlers/bingbot/) Microsoft, search index. Token `bingbot`. Verification: reverse dns. Refused by 3.1% of top-10,000 sites. [All news](https://agentreadycrawlers.com/news/). ## Sources - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Arcjet launches Agent Runtime Security | Agent Ready Crawlers" description: "Arcjet launches Agent Runtime Security. Its inbound bot protection still verifies known bots by IP and reverse DNS. Dated 2026-09-17." canonical: https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-security site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-09-17 section: "News" --- # Arcjet launches Agent Runtime Security | Agent Ready Crawlers > Arcjet launches Agent Runtime Security. Its inbound bot protection still verifies known bots by IP and reverse DNS. Dated 2026-09-17. Source: https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-security Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-09-17 # Arcjet launches Agent Runtime Security In short 17 September 2026: Arcjet launches Agent Runtime Security. Arcjet added runtime controls for AI agents that observe activity through OpenTelemetry, enforce policies on tool calls, APIs and databases, and keep an audit trail. Its inbound bot protection still verifies known bots by IP and reverse DNS. Arcjet's price: from USD 25/month plus USD 5 per 1M web requests. Affects Arcjet. Arcjet added runtime controls for AI agents that observe activity through OpenTelemetry, enforce policies on tool calls, APIs and databases, and keep an audit trail. Its inbound bot protection still verifies known bots by IP and reverse DNS. Primary source [arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit), read 2026-11-04. ## Where it stands in the Index [Arcjet](https://agentreadycrawlers.com/vendors/arcjet/) Vendor. Developer security SDK that runs bot detection, rate limits and WAF rules inside application code; it verifies known bots by IP and reverse DNS and in late 2026 added runtime security for AI agents. [All news](https://agentreadycrawlers.com/news/). ## Sources - [arcjet.com: introducing agent runtime security observe enforce audit](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DataDome joins Experian Agent Trust | Agent Ready Crawlers" description: "DataDome joins Experian Agent Trust. DataDome already lists Web Bot Auth, KYA tokens, IP lists and reverse DNS as identification methods." canonical: https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trust site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-01 section: "News" --- # DataDome joins Experian Agent Trust | Agent Ready Crawlers > DataDome joins Experian Agent Trust. DataDome already lists Web Bot Auth, KYA tokens, IP lists and reverse DNS as identification methods. Source: https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trust Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-01 # DataDome joins Experian Agent Trust In short 1 October 2026: DataDome joins Experian Agent Trust. Experian announced DataDome joined its Agent Trust ecosystem, using verified Know Your Agent signals for continuous intent checks during a session. DataDome already lists Web Bot Auth, KYA tokens, IP lists and reverse DNS as identification methods. DataDome's price: from USD 3,830/month. Rows affected in this site: DataDome. Experian announced DataDome joined its Agent Trust ecosystem, using verified Know Your Agent signals for continuous intent checks during a session. DataDome already lists Web Bot Auth, KYA tokens, IP lists and reverse DNS as identification methods. Primary source [experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous), read 2026-11-04. ## Where it stands in the Index [DataDome](https://agentreadycrawlers.com/vendors/datadome/) Vendor. Bot and fraud protection vendor whose Agent Trust layer identifies AI agents, scores them per customer, applies per-agent policies and connects to monetization partners, with prices published from USD 3,830 per month. [All news](https://agentreadycrawlers.com/news/). ## Sources - [experianplc.com: datadome joins experian agent trust ecosystem to add continuous](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "DataDome Q3: 25 billion agent requests | Agent Ready Crawlers" description: "DataDome Q3 report: 25 billion AI agent requests. It counted 29.1 million Muse requests in under two weeks without self-identification." canonical: https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-15 section: "News" --- # DataDome Q3: 25 billion agent requests | Agent Ready Crawlers > DataDome Q3 report: 25 billion AI agent requests. It counted 29.1 million Muse requests in under two weeks without self-identification. Source: https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-15 # DataDome Q3 report: 25 billion AI agent requests In short 15 October 2026: DataDome Q3 report: 25 billion AI agent requests. It counted 29.1 million Muse requests in under two weeks without self-identification. Figures via Help Net Security. DataDome's price: from USD 3,830/month. Meta-ExternalAgent is Meta's training crawler. Rows affected in this site: DataDome, Meta-ExternalAgent, Meta-WebIndexer and Muse. DataDome counted nearly 25 billion AI agent requests in Q3 2026, up 41.5 percent on the quarter, with Meta-ExternalAgent and Meta-WebIndexer at 49.7 percent. It counted 29.1 million Muse requests in under two weeks without self-identification. Figures via Help Net Security. Secondary source [helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report), read 2026-11-04. ## Where it stands in the Index [DataDome](https://agentreadycrawlers.com/vendors/datadome/) Vendor. Bot and fraud protection vendor whose Agent Trust layer identifies AI agents, scores them per customer, applies per-agent policies and connects to monetization partners, with prices published from USD 3,830 per month. [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) Meta, training. Token `meta-externalagent`. Verification: none published. Refused by 16.0% of top-10,000 sites. [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) Meta, search index. Token `meta-webindexer`. Verification: none published. [Muse](https://agentreadycrawlers.com/crawlers/meta-muse/) Meta, shopping agent. No robots.txt token. Verification: none published. [All news](https://agentreadycrawlers.com/news/). ## Sources - [helpnetsecurity.com: datadome ai agent traffic report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Anthropic refreshes its crawler IP list | Agent Ready Crawlers" description: "Anthropic refreshes its crawler IP list. Anthropic warns that IP blocking may stop its bots reading robots.txt. Dated 2026-10-20." canonical: https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-20 section: "News" --- # Anthropic refreshes its crawler IP list | Agent Ready Crawlers > Anthropic refreshes its crawler IP list. Anthropic warns that IP blocking may stop its bots reading robots.txt. Dated 2026-10-20. Source: https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-20 # Anthropic refreshes its crawler IP list In short 20 October 2026: Anthropic refreshes its crawler IP list. claude.com/crawling/bots.json, the list Anthropic links for verifying ClaudeBot, Claude-User and Claude-SearchBot, carried creationTime 2026-10-20 when read, starting with 216.73.216.0/22. Anthropic warns that IP blocking may stop its bots reading robots.txt. Claude-User is Anthropic's live answer fetcher. Rows affected in this site: ClaudeBot, Claude-User and Claude-SearchBot. claude.com/crawling/bots.json, the list Anthropic links for verifying ClaudeBot, Claude-User and Claude-SearchBot, carried creationTime 2026-10-20 when read, starting with 216.73.216.0/22. Anthropic warns that IP blocking may stop its bots reading robots.txt. Primary source [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json), read 2026-11-04. ## Also read - [Anthropic support: does Anthropic crawl data from the web, and how can site owners block the crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). Anthropic says blocking the IP addresses its bots operate from may not work correctly or persistently guarantee an opt-out, because it impedes its ability to read your robots.txt. Read 2026-11-04. ## Where it stands in the Index [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Anthropic, training. Token `ClaudeBot`. Verification: ip list. Refused by 17.6% of top-10,000 sites. [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Anthropic, live answer fetch. Token `Claude-User`. Verification: ip list. Refused by 8.1% of top-10,000 sites. [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Anthropic, search index. Token `Claude-SearchBot`. Verification: ip list. Refused by 7.7% of top-10,000 sites. [All news](https://agentreadycrawlers.com/news/). ## Sources - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 - [Anthropic support: does Anthropic crawl data from the web, and how can site owners block the crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Claude Duet browses as Claude-Agent | Agent Ready Crawlers" description: "Anthropic's Claude Duet starts browsing with a Claude-Agent token. Claude Duet launched as a waitlist research preview for US Pro and Max subscribers." canonical: https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-21 section: "News" --- # Claude Duet browses as Claude-Agent | Agent Ready Crawlers > Anthropic's Claude Duet starts browsing with a Claude-Agent token. Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. Source: https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-21 # Anthropic's Claude Duet starts browsing with a Claude-Agent token In short 21 October 2026: Anthropic's Claude Duet starts browsing with a Claude-Agent token. Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. Verisoul reports its cloud browser sends a Linux Chrome string with "compatible; Claude-Agent/1.0"; Anthropic's crawler page does not yet list it and no signature is documented. Affects Claude Duet cloud browser. Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. Verisoul reports its cloud browser sends a Linux Chrome string with "compatible; Claude-Agent/1.0"; Anthropic's crawler page does not yet list it and no signature is documented. Secondary source [verisoul.ai/agents/claude-duet](https://www.verisoul.ai/agents/claude-duet), read 2026-11-04. ## Where it stands in the Index [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Anthropic, task agent. Token `Claude-Agent`. Verification: none published. [All news](https://agentreadycrawlers.com/news/). ## Sources - [verisoul.ai: claude duet](https://www.verisoul.ai/agents/claude-duet) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "IETF proposes common IP-range format | Agent Ready Crawlers" description: "IETF drafts propose a common IP-range format and crawler best practices: one for publishing IP ranges, one for crawler conduct. Dated 2026-10-22." canonical: https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-bes site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-22 section: "News" --- # IETF proposes common IP-range format | Agent Ready Crawlers > IETF drafts propose a common IP-range format and crawler best practices: one for publishing IP ranges, one for crawler conduct. Dated 2026-10-22. Source: https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-bes Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-22 # IETF drafts propose a common IP-range format and crawler best practices In short 22 October 2026: IETF drafts propose a common IP-range format and crawler best practices. New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft. No Index row or vendor profile is tied to it; read the source before changing any rule. New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft. Primary source [datatracker.ietf.org/wg/webbotauth/documents/](https://datatracker.ietf.org/wg/webbotauth/documents/), read 2026-11-04. [All news](https://agentreadycrawlers.com/news/). ## Sources - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "OpenAI updates the ChatGPT-User IP list | Agent Ready Crawlers" description: "OpenAI updates the ChatGPT-User IP list. openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04." canonical: https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-27 section: "News" --- # OpenAI updates the ChatGPT-User IP list | Agent Ready Crawlers > OpenAI updates the ChatGPT-User IP list. openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. Source: https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-27 # OpenAI updates the ChatGPT-User IP list In short 27 October 2026: OpenAI updates the ChatGPT-User IP list. openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. WAF allowlists built from earlier copies may miss new prefixes; OpenAI recommends allowing its published ranges. ChatGPT-User is OpenAI's live answer fetcher. Rows affected in this site: ChatGPT-User. openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. WAF allowlists built from earlier copies may miss new prefixes; OpenAI recommends allowing its published ranges. Primary source [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json), read 2026-11-04. ## Where it stands in the Index [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) OpenAI, live answer fetch. Token `ChatGPT-User`. Verification: ip list. Refused by 9.7% of top-10,000 sites. [All news](https://agentreadycrawlers.com/news/). ## Sources - [openai.com: chatgpt user (JSON)](https://openai.com/chatgpt-user.json) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Vercel splits AI crawlers from AI assistants | Agent Ready Crawlers" description: "Vercel splits AI crawlers from AI assistants. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Dated 2026-10-29." canonical: https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistants site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-29 section: "News" --- # Vercel splits AI crawlers from AI assistants | Agent Ready Crawlers > Vercel splits AI crawlers from AI assistants. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Dated 2026-10-29. Source: https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistants Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-29 # Vercel splits AI crawlers from AI assistants In short 29 October 2026: Vercel splits AI crawlers from AI assistants. Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Vercel's price: basic free; Deep Analysis USD 1 per 1,000 checks on Pro. Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Primary source [vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. ## Where it stands in the Index [Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/) Vendor. Hosting platform with an invisible CAPTCHA (BotID, Deep Analysis by Kasada), a verified bot directory checked by IP, reverse DNS and Web Bot Auth, and a firewall ruleset that treats AI crawlers and AI assistants separately. [All news](https://agentreadycrawlers.com/news/). ## Sources - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Web Bot Auth working-group draft reaches -01 | Agent Ready Crawlers" description: "Web Bot Auth working-group draft reaches -01. A registry and Signature Agent card draft (-04) followed on 2026-10-27. Dated 2026-10-29." canonical: https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01 site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-10-29 section: "News" --- # Web Bot Auth working-group draft reaches -01 | Agent Ready Crawlers > Web Bot Auth working-group draft reaches -01. A registry and Signature Agent card draft (-04) followed on 2026-10-27. Dated 2026-10-29. Source: https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01 Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-10-29 # Web Bot Auth working-group draft reaches -01 In short 29 October 2026: Web Bot Auth working-group draft reaches -01. draft-ietf-webbotauth-httpsig-protocol-01, the adopted working-group draft that replaced the individual Meunier drafts, was posted on 2026-10-29. A registry and Signature Agent card draft (-04) followed on 2026-10-27. For a site owner nothing changes yet: a working-group draft is not a finished standard, so keep your current verification in place. draft-ietf-webbotauth-httpsig-protocol-01, the adopted working-group draft that replaced the individual Meunier drafts, was posted on 2026-10-29. A registry and Signature Agent card draft (-04) followed on 2026-10-27. Primary source [datatracker.ietf.org/wg/webbotauth/documents/](https://datatracker.ietf.org/wg/webbotauth/documents/), read 2026-11-04. [All news](https://agentreadycrawlers.com/news/). ## Sources - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Signed-Agent-Metadata header proposed | Agent Ready Crawlers" description: "Signed-Agent-Metadata header proposed for Web Bot Auth. A new draft would carry signed metadata about an agent beside its signature. Dated 2026-11-02." canonical: https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-auth site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-02 section: "News" --- # Signed-Agent-Metadata header proposed | Agent Ready Crawlers > Signed-Agent-Metadata header proposed for Web Bot Auth. A new draft would carry signed metadata about an agent beside its signature. Dated 2026-11-02. Source: https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-auth Site: Agent Ready Crawlers Last reviewed: 2026-11-04 News from 2026-11-02 # Signed-Agent-Metadata header proposed for Web Bot Auth In short 2 November 2026: Signed-Agent-Metadata header proposed for Web Bot Auth. draft-vilgalys-webbotauth-signed-agent-metadata-00 proposes an HTTP header that carries signed metadata about the agent alongside a Web Bot Auth signature. For a site owner nothing changes yet: a proposed header is not something any operator sends today, so no rule needs to change. draft-vilgalys-webbotauth-signed-agent-metadata-00 proposes an HTTP header that carries signed metadata about the agent alongside a Web Bot Auth signature. Primary source [datatracker.ietf.org/wg/webbotauth/documents/](https://datatracker.ietf.org/wg/webbotauth/documents/), read 2026-11-04. [All news](https://agentreadycrawlers.com/news/). ## Sources - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Privacy | Agent Ready Crawlers" description: "Privacy on Agent Ready Crawlers: the site sets no cookies, runs no third-party scripts and keeps what you type in the robots.txt builder in your browser." canonical: https://agentreadycrawlers.com/privacy site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Publication" --- # Privacy | Agent Ready Crawlers > Privacy on Agent Ready Crawlers: the site sets no cookies, runs no third-party scripts and keeps what you type in the robots.txt builder in your browser. Source: https://agentreadycrawlers.com/privacy Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Privacy This site's pages set no cookies and load no third-party scripts, fonts or images. Two small scripts run in your browser: the filter on the full index and the robots.txt builder. What you type into them stays in your browser and is not sent to us. Visiting a page makes a request to the host, which may count page views in aggregate and keep server logs under its own terms. We do not build profiles of visitors and have no accounts. If you email the editor mailbox, we use your message only to answer it and to correct the page it concerns. The crawl behind the Block Rate Report fetched public robots.txt files, home pages and well-known discovery paths. It kept the first 200 words of a home page and a set of extracted fields, not page bodies; see [what the crawl does](https://agentreadycrawlers.com/method/#own-crawl). ## Sources - [Terms (this site)](https://agentreadycrawlers.com/terms/) read 2026-11-04 - [robots.txt builder (this site)](https://agentreadycrawlers.com/robots-builder/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "robots.txt builder for AI crawlers | Agent Ready Crawlers" description: "Build a robots.txt from the Crawler Index: allow or refuse training, search, answer and agent tokens, then copy the file." canonical: https://agentreadycrawlers.com/robots-builder site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Tools" --- # robots.txt builder for AI crawlers | Agent Ready Crawlers > Build a robots.txt from the Crawler Index: allow or refuse training, search, answer and agent tokens, then copy the file. Source: https://agentreadycrawlers.com/robots-builder Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # robots.txt builder In short Choose a policy for each purpose and copy a robots.txt that names 47 tokens from the Crawler Index, one group per purpose. 23 rows in the Index have no token, so no robots.txt can address them. The search group includes Googlebot and Bingbot, so refusing it removes pages from those search results. The default below refuses training, allows everything else and adds no sitemap or Content-Signal line. Change a purpose to Allow or Refuse and the file updates on the page. Without JavaScript the default file is still complete in the box. Your robots.txt \# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: \* Allow: / ## What this file cannot do - It cannot address the 23 rows with no token, such as browsing agents that run in a visitor's own browser. See [crawler versus browsing agent](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent/). - It states a preference. Operators of user-triggered fetchers say robots.txt may not apply to them; see [live answer fetchers](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/), and enforce at the WAF. - Google-Extended and Applebot-Extended are control tokens: refusing them opts out of training while search crawling continues. [How to allow search and refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/). - The Content-Signal line is an extension from the Content Signals Policy, not part of RFC 9309. Crawlers that do not read it ignore it. [What Content-Signals changes](https://agentreadycrawlers.com/guides/what-content-signals-change/). ## Tokens the builder writes Training GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, MistralAI-Training Search index OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Storebot-Google, Applebot, bingbot, Amzn-SearchBot, meta-webindexer, DuckDuckBot, MistralAI-Index, YouBot, Timpibot, PetalBot, ExaSearchBot Live answer fetch ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, meta-externalfetcher, DuckAssistBot, MistralAI-User Task agent Claude-Agent Shopping agent None with a token Other OAI-AdsBot, Google-CloudVertexBot, GoogleOther, MicrosoftPreview, meta-externalads, facebookexternalhit, FacebookBot, Bytespider, cohere-ai, Diffbot, omgili, ImagesiftBot, FirecrawlAgent, OraBot, agent-readiness-crawl The search group holds OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot and 11 more. ## How the file is tested A test in this site's repository builds the file for several combinations of choices and checks every line against the grammar of RFC 9309 section 2: product tokens, groups that start with a user-agent line, rules that begin with a slash, and an absolute sitemap URL. The two extension lines, Sitemap and Content-Signal, are checked separately. The test runs on every build. ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Crawler Index (this site)](https://agentreadycrawlers.com/crawlers/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Terms | Agent Ready Crawlers" description: "Terms of use for Agent Ready Crawlers: how to cite the pages and datasets, what the data covers and how it is dated, and how to reach the editors about changes." canonical: https://agentreadycrawlers.com/terms site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Publication" --- # Terms | Agent Ready Crawlers > Terms of use for Agent Ready Crawlers: how to cite the pages and datasets, what the data covers and how it is dated, and how to reach the editors about changes. Source: https://agentreadycrawlers.com/terms Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Terms You may read, quote and link to any page. The Crawler Index and block-rate datasets on [the data page](https://agentreadycrawlers.com/data/) are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); credit them with the cite-this line shown with each dataset: the masthead, the title, the edition date and the URL. The information is published as read on the dates shown on each page. Operators and vendors change their documents, so check the linked source before you rely on a row. Quoted operator statements belong to their authors and keep their source link. Do not present this site's grades or rankings as the vendors' own. Links to other sites are references, and we do not control those sites. We may change a page or a dataset to correct it or to bring it up to date; corrections are logged on [the corrections page](https://agentreadycrawlers.com/corrections/). ## Sources - [Data downloads (this site)](https://agentreadycrawlers.com/data/) read 2026-11-04 - [Editorial policy (this site)](https://agentreadycrawlers.com/editorial-policy/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Vendors: operators, bot management, licensing | Agent Ready Crawlers" description: "The full vendor list: 73 crawler operators, bot management, verification and licensing vendors with standards, dated prices and sources. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-05 section: "Vendors" --- # Vendors: operators, bot management, licensing | Agent Ready Crawlers > The full vendor list: 73 crawler operators, bot management, verification and licensing vendors with standards, dated prices and sources. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors Site: Agent Ready Crawlers Last reviewed: 2026-11-05 73 companies in four types # Vendors In short 73 companies are profiled in four types: 31 crawler operators, 23 bot management vendors, 11 agent verification providers and 8 licensing intermediaries. Each profile holds sourced facts, a standards table, dated prices where a vendor publishes them and what the vendor does not do. Read 2026-11-05: 671 facts from 331 sources. This list profiles 73 companies that sit between a crawler and a page: the operators that run crawlers and agents, the vendors that decide at the door whether they get in, the providers that let a site check who an agent is, and the intermediaries that turn access into a licence or a price. Crawler operators are every company behind a row of the Crawler Index (27 of them), plus 4 browser-agent and search-API companies whose own pages say how their traffic identifies itself. 15 of the vendors keep the ids and rubric pages they had before; their profiles now carry the same sections as the rest. A company is listed only when its own pages state what it does for this layer. Where a vendor's site refused the crawl's user agent, the profile cites a wire reprint, trade press or the vendor's developer documentation and says so; 12 profiles carry at least one third-party source. 671 facts come from 331 pages read on 2026-11-05, each with its source in the download and on its profile. A sentence that repeats one printed on another profile, or in a graded vendor's rubric text, appears once; 68 such sentences are kept in the download only. 50 vendors carry a dated pricing entry, and 34 of them state an amount. 32 profiles carry our own probes of key directories and IP files. Every homepage was fetched once on 2026-11-05 with its status recorded: 68 answered 200; Amazon answered 202, OpenAI answered 403, xAI answered 403, Akamai answered 403 and DataDome answered 403. 15 of these vendors are also graded on a five-question rubric. [The grades and the rubric](https://agentreadycrawlers.com/vendors/ranking/) are on their own page. - [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) - [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/) - [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) - [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Standards are counted when the vendor's own pages state support, and listed by id: `rfc9309` is [RFC 9309 (robots.txt)](https://agentreadycrawlers.com/glossary/rfc-9309/); `web-bot-auth` is [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/); `content-signals` is [Content Signals](https://agentreadycrawlers.com/glossary/content-signal/); `ip-list` is [Published IP list](https://agentreadycrawlers.com/glossary/ip-range-json/); `reverse-dns` is [Reverse DNS check](https://agentreadycrawlers.com/glossary/reverse-dns-verification/); `cloudflare-verified-bots` is [Cloudflare verified bots](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/); `visa-tap` is [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/); `mastercard-agent-pay` is [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/). Rows are alphabetical inside each type. ## Crawler operators The companies behind the crawlers and agents in the Crawler Index, plus browser-agent and search-API companies whose own pages say how their traffic identifies itself. [The 31 crawler operators on a page of their own](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/). | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Amazon | Crawler operator | Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 202, 4 probes 2026-11-05 | 2026-11-05 | | Anchor Browser | Crawler operator | Cloud browser infrastructure for computer-use agents that can sign sessions with Cloudflare Web Bot Auth on its upper plans, and also sells stealth fingerprinting, proxies, CAPTCHA bypass and a pay-on-success web unlocker. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Anthropic | Crawler operator | Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Apify | Crawler operator | Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. | 0; partial: rfc9309, web-bot-auth, cloudflare-verified-bots | platform usage; subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Apple | Crawler operator | Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Block | Crawler operator | Company that built goose, an open-source local AI agent Cloudflare classed as a signed agent in 2025; Block has since donated goose to the Linux Foundation's Agentic AI Foundation and publishes no key directory for it. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Browser Use | Crawler operator | Maker of the open-source browser-use agent library and a hosted cloud browser for agents that ships with stealth fingerprinting, residential proxies and CAPTCHA solving on by default, and no published identifier for site owners. | 0 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Browserbase | Crawler operator | Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned. | 2 web-bot-auth, cloudflare-verified-bots; partial: rfc9309 | subscription plus usage | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Cohere | Crawler operator | Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page. | 1 rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Common Crawl | Crawler operator | Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Diffbot | Crawler operator | Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. | 1 rfc9309 | subscription with credits; credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | DuckDuckGo | Crawler operator | DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Exa | Crawler operator | AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities. | 2 rfc9309, web-bot-auth | pay as you go | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Firecrawl | Crawler operator | Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends. | 1 rfc9309 | credit subscription | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Google | Crawler operator | Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic. | 2 ip-list, reverse-dns; partial: rfc9309, web-bot-auth | No price recorded | Homepage HTTP 200, 5 probes 2026-11-05 | 2026-11-05 | | Hive | Crawler operator | AI model company, renamed from Hive to Deploy in October 2026, whose ImageSift unit runs ImagesiftBot to collect public images and page text for a reverse image search index. | 1 rfc9309 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Huawei | Crawler operator | Huawei runs PetalBot for the Petal search engine, whose index also feeds Huawei Assistant and AI Search; PetalBot follows robots.txt and is verified by reverse DNS under aspiegel.com or an IP lookup tool. | 2 rfc9309, reverse-dns | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Hyperbrowser | Crawler operator | Cloud browser platform for AI agents whose sessions can sign requests with Web Bot Auth under the fixed Signature-Agent www.hyperbrowser.ai, on request per team, alongside stealth, proxies and CAPTCHA solving. | 1 web-bot-auth | credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Kernel | Crawler operator | Cloud browser company for AI agents that runs two signed identities, Kernel Agent for user-directed browsing and KernelSearchBot for search crawling, each with its own Web Bot Auth key directory so sites can treat them separately. | 3 rfc9309, web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Manus | Crawler operator | General-purpose AI agent, owned by Meta since December 2025, that browses either in an isolated cloud browser or through an extension inside the user's own browser, and publishes no identifier, robots.txt policy or signing keys for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Meta | Crawler operator | Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt. | 0; partial: rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Microsoft | Crawler operator | Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Mistral AI | Crawler operator | Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two. | 1 rfc9309; partial: ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Notte | Crawler operator | Browser platform for AI agents (sessions, agents and serverless browser functions) that says it is registered as a verified bot with Cloudflare and signs session requests with Web Bot Auth when a customer turns it on. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | OpenAI | Crawler operator | OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. | 2 web-bot-auth, ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 403, 5 probes 2026-11-05 | 2026-11-05 | | Parallel Web Systems | Crawler operator | Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only. | 2 rfc9309, web-bot-auth; partial: ip-list | pay per request | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Perplexity | Crawler operator | Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Timpi | Crawler operator | Operator of an independent search index built by a decentralised network of contributor nodes; its crawler appears in robots.txt files as Timpibot, but Timpi publishes no crawler documentation for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Webz.io | Crawler operator | Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training. | 1 rfc9309 | pay as you go | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | xAI | Crawler operator | xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic. | 0 | No price recorded | Homepage HTTP 403, 1 probe 2026-11-05 | 2026-11-05 | | You.com | Crawler operator | You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | ## Bot management vendors CDN, WAF, CAPTCHA and bot-detection vendors that decide at the door whether a crawler or agent gets in, and whether it pays. [The 23 bot management vendors on a page of their own](https://agentreadycrawlers.com/vendors/by-type/bot-management/). | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Akamai | Bot management | CDN and security company whose Bot & Agent Control solutions judge AI agents at the edge, take identity from Visa's Trusted Agent Protocol and Know Your Agent tokens, and send AI traffic to TollBit or Skyfire for paid access. | 2 web-bot-auth, visa-tap | sales-led | Homepage HTTP 403 2026-11-05 | 2026-11-05 | | AWS WAF Bot Control | Bot management | Cloud provider whose WAF Bot Control labels and verifies bots, checks Web Bot Auth signatures and can charge AI bots with HTTP 402, and whose AgentCore Browser signs its own agent traffic with Web Bot Auth. | 2 web-bot-auth, ip-list | per web ACL plus per request | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Arcjet | Bot management | Developer security SDK for JavaScript, Python and Go that detects bots inside application code, lets developers allow or deny named bots or categories such as CATEGORY:AI, and adds runtime guards for agent frameworks. | 2 ip-list, reverse-dns | monthly plan plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Castle | Bot management | Bot and account abuse API whose risk responses include an agent object that names AI agents and browser automation, identified by a verified Web Bot Auth signature, a browser fingerprint or a user-agent string. | 1 web-bot-auth | usage-based plans | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Centinel Analytica | Bot management | Bot and AI crawler protection vendor whose edge integration verifies declared search and AI crawlers against operators' published IP ranges and detects scrapers that run real browsers behind residential proxies. | 1 ip-list | quoted per deployment | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Cequence Security | Bot management | Application and API protection vendor whose Bot Management includes Agent Trust for personal AI shopping agents, and whose AI Gateway governs enterprise agents' tool calls behind OAuth 2.1 authentication. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Cloudflare | Bot management | Connectivity cloud whose AI Crawl Control, Bot Management and verified bot program sort crawlers by purpose, verify them by IP list or Web Bot Auth signature, and let site owners charge crawlers per request in a private beta. | 5 rfc9309, web-bot-auth, content-signals, ip-list, cloudflare-verified-bots | included in every plan; site owner sets one per-request price; 1 more | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | DataDome | Bot management | Bot and fraud protection vendor whose Agentic Trust layer names each AI agent, rates how strong that identification is, applies per-agent and per-endpoint policies, and hands chosen agents to TollBit for paid access. | 3 web-bot-auth, ip-list, reverse-dns | monthly, by request volume | Homepage HTTP 403 2026-11-05 | 2026-11-05 | | F5 | Bot management | Application delivery and security company whose Distributed Cloud Bot Defense puts humans, trusted AI agents and malicious bots under one policy, accepts Web Bot Auth and Skyfire's Know Your Agent tokens, and runs on F5's platform, BIG-IP or public cloud. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Fastly | Bot management | Edge cloud vendor whose Bot Management adds edge detection (ContentGuard), TLS fingerprinting, client challenges and a verified bots signal, with separate AI crawler and AI fetcher signals for training bots and user-triggered fetchers. | 0 | contact sales | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Fingerprint | Bot management | Device intelligence vendor whose Bot Detection and Authorized AI Agent Detection signals separate good bots, bad bots and AI agents, and verify signed agents from providers such as OpenAI, AWS AgentCore and Browserbase. | 1 web-bot-auth | monthly plan plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Google Cloud Fraud Defense | Bot management | Google Cloud's bot and fraud service, the evolution of reCAPTCHA, which verifies AI agents by Web Bot Auth signature or by user agent matched to published IP ranges or reverse DNS, and reports verified and suspected agent sessions. | 3 web-bot-auth, ip-list, reverse-dns | per assessment | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | hCaptcha | Bot management | Bot detection and CAPTCHA vendor whose Enterprise Agent Controls identify AI agents, verify partner platforms that sign with Web Bot Auth, and let sites allow browsing while requiring verification before an agent reserves inventory. | 1 web-bot-auth | contact sales; monthly plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | HUMAN Security | Bot management | Bot and fraud defense vendor whose AgenticTrust module in Sightline names AI agents, grades them High, Medium or Low trust, verifies signed agents, and lets sites set what each agent may do from browsing to checkout. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Imperva | Bot management | Application security vendor owned by Thales whose Advanced Bot Protection sorts human, good bot and bad bot traffic across web, mobile and API; verified-agent controls so far ship in a controlled-availability Google Cloud offering. | 0 | sales-led, free trial | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Kasada | Bot management | Bot defense vendor founded in Sydney whose AI Agent Trust adds a verified agent directory with Web Bot Auth support and endpoint-level access rules, enforced at the edge before automated traffic reaches the application. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Known Agents | Bot management | Agent analytics and access-control service, formerly Dark Visitors, that identifies AI agents and crawlers from server-side data, keeps an automatic robots.txt current, and blocks bots that ignore it or spoof trusted identities. | 1 rfc9309 | monthly by events | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Link11 | Bot management | German security provider whose WAAP platform joins WAF, API security, Bot Management and Web DDoS protection, with a setting to monitor or block AI crawlers; the reblaze.com domain now redirects to Link11. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Netacea | Bot management | Server-side bot protection vendor that integrates at the CDN or load balancer without client code, classifies humans, good bots, bad bots and AI agents, and enforces access by intent rather than declared identity. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Netlify | Bot management | Web hosting platform that blocks chosen AI crawlers by user-agent string through an Edge Function extension and offers IP, geography and header-based traffic rules and rate limiting; it does not verify bot identity. | 0 | plan-based | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Radware | Bot management | Application security vendor whose Bot Manager and Agent Trust Management identify Web Bot Auth-compliant and non-compliant agents, capture an agent's declared intent in conversation, and restrict actions such as login or checkout. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Vercel | Bot management | Hosting platform whose firewall verifies known bots by IP, reverse DNS and Web Bot Auth, offers an AI Bots managed ruleset that treats AI crawlers and AI assistants separately, and sells BotID checks for high-value routes. | 3 web-bot-auth, ip-list, reverse-dns | included; per call | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Verisoul | Bot management | Fake account and fraud detection platform whose Agent & Bot Detection names AI agents by signature, self-declared user agent or inference, and whose public Agent Atlas documents how each agent shows up on a site. | 1 web-bot-auth | sales-led, 30-day trial | Homepage HTTP 200 2026-11-05 | 2026-11-05 | ## Verification providers Web Bot Auth implementers, card-network agent programmes and agent identity products that let a site check who an agent is and who sent it. [The 11 verification providers on a page of their own](https://agentreadycrawlers.com/vendors/by-type/verification-provider/). | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | American Express | Verification provider | Card network whose ACE developer kit plans an agent registration service so only verified AI agents get tokenized Amex credentials, with Agent Purchase Protection for registered agent purchases; the registration spec is still in development. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Auth0 for AI Agents | Verification provider | Okta's customer identity platform; its AI Agents add-on gives agents a company builds a user-bound identity, scoped tokens, a Token Vault for third-party APIs and Cross App Access, and it verifies no inbound crawlers. | 0 | add-on, 50% of the base plan price | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Descope | Verification provider | Customer and agent identity platform whose Agentic Identity Hub gives AI agents and MCP clients OAuth 2.1 access, short-lived scoped credentials and an agent directory with policies, for agents calling a company's APIs rather than crawling its site. | 0 | free tier, then monthly plans | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Mastercard | Verification provider | Card network whose Agent Pay registers AI agents and has them sign requests with Web Bot Auth under the tag Agent-pay-auth, so a merchant or its CDN can check the agent's key in a network directory before letting it buy. | 2 web-bot-auth, mastercard-agent-pay; partial: visa-tap | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Microsoft Entra Agent ID | Verification provider | Microsoft's identity service for AI agents inside an Entra tenant: each agent gets its own service principal created from a blueprint, with tokens, a human sponsor and audit, for agents an organisation runs rather than agents visiting a website. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Prove | Verification provider | Identity verification company whose Verified Agent product binds an AI agent to a verified person or business with identity-bound tokens, checks participants against a shared trust registry and logs co-signed consent for agent purchases. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Stytch IsAgent | Verification provider | Identity platform, part of Twilio since November 2025, that verifies Web Bot Auth signatures through its IsAgent SDK and Device Fingerprinting, and issues scoped OAuth tokens to AI agents and MCP clients through Connected Apps. | 1 web-bot-auth | free tier, then contact sales; per fingerprint above the included volume | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Trulioo | Verification provider | Identity verification company whose Know Your Agent programme, in alpha, runs business and AML checks on the company behind an AI agent, binds the agent's fingerprint to it in a signed Digital Agent Profile and lets counterparties check it by API or MCP. | 0 | alpha, access on request | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Visa | Verification provider | Card network whose Trusted Agent Protocol lets a merchant or its CDN verify, from RFC 9421 signatures tagged agent-browser-auth or agent-payer-auth, that a visiting agent is approved under Visa Intelligent Commerce and acting for a known consumer. | 2 web-bot-auth, visa-tap | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Vouched | Verification provider | Identity verification company whose Know Your Agent suite detects AI agents on a site by behaviour, enforces allow, challenge, block or redirect rules, and ties each agent action to a verified person through signed delegations. | 0 | free; monthly fee plus per transaction | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | World | Verification provider | Proof-of-human network whose AgentKit lets a person with a verified World ID register an agent's wallet, so x402-protected sites can tell a human-backed agent from a script and limit access per unique person. | 0 | developer preview | Homepage HTTP 200 2026-11-05 | 2026-11-05 | ## Licensing intermediaries Marketplaces, standards bodies and paywalls that turn crawler access into a licence or a price. [The 8 licensing intermediaries on a page of their own](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/). | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Dappier | Licensing intermediary | Content marketplace where publishers list their content as priced RAG models and AI agents buy query access through an agent storefront that takes x402, MPP or card payments without signup. | 0; partial: rfc9309 | publisher-set price per query | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | FetchRight | Licensing intermediary | Licensing service behind the open Peek-Then-Pay protocol: a site publishes /.well-known/peek.json, AI agents get a priced HTTP 203 preview, buy a JWT license per intent and present it to an edge enforcer. | 0 | open specification; sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | IAB Tech Lab | Licensing intermediary | Ad-industry standards body whose Content Monetization Protocols (CoMP) 1.0 define how a licensed AI crawler asks a publisher for content and gets a token, alongside a paid Spiders and Bots List and a free Agent Registry. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | open specification; annual subscription; 1 more | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | ProRata | Licensing intermediary | AI attribution company that splits an AI answer into claims and measures each source's share as a basis for licensing and payment, and runs Gist Answers on a network of licensed publisher titles. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | RSL Collective | Licensing intermediary | Nonprofit collective rights organisation and co-developer of the RSL standard, a machine-readable license file a site links from robots.txt so crawlers can see, buy and prove a license for training, AI input, search or crawling. | 1 rfc9309 | open standard; free to join for publishers | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Skyfire | Licensing intermediary | Agent identity and payment network whose signed KYA and pay tokens let a website, API or bot manager admit a verified agent and charge it per request, with Skyfire vetting the agent platform and the person or business behind the agent. | 0 | subscription for verification (Individual or Organization) | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Supertab | Licensing intermediary | Managed RSL licensing service: Supertab Connect hosts a publisher's license.xml and RSL license server, issues license tokens to crawlers, checks them at the CDN with its SDK and bills the use, priced from USD 99 a month. | 1 rfc9309 | monthly subscription by visitors and licensees | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | TollBit | Licensing intermediary | Licensing marketplace that sends AI bots, by user agent at the publisher's CDN, to an Agent Site on a tollbit subdomain, where bots holding a short-lived TollBit JWT get licensed content at publisher-set rates per 1,000 pages. | 0; partial: rfc9309 | free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee | Homepage HTTP 200 2026-11-05 | 2026-11-05 | ## Not listed, and why The research pass looked at these companies and did not profile them. A name taken from a roundup with no page of its own is left out the same way. - **ByteDance.** No ByteDance page states a robots.txt policy, an IP list or a verification method for Bytespider, so the Bytespider row stays in the Crawler Index without a vendor profile. - **Steel, Browserless, Skyvern, Airtop, Bright Data, Zyte, Oxylabs and ScrapingBee.** Their documentation covers stealth, proxies and CAPTCHA solving for customers' traffic but states no user agent token, robots.txt policy or Web Bot Auth signing. - **Microsoft Azure WAF bot protection.** Microsoft Learn describes good-bot and bad-bot rules and says nothing about AI crawlers, agents or Web Bot Auth. - **Gcore.** Its robots.txt disallows the crawl user agent on the bot protection page. - **Friendly Captcha.** No page states how agents are detected. - **Peakhour, Myra Security and Barracuda.** The pages read state no AI crawler or agent feature. - **Sucuri, Wordfence, Prophaze and Indusface.** Not reached in this edition. - **GoDaddy and Persona.** Their sites answered 403 to the crawl user agent. - **Ping Identity and Crossmint.** No agent product page was found. - **Spawning.** Its site showed a maintenance page on 2026-11-05. - **ScalePost.** It now sells AI visibility measurement, which is not this site's subject. - **Copyright Clearance Center and Created by Humans.** Their licences cover internal AI use or book rights and say nothing about crawlers or site access. - **Coinbase x402.** Coinbase documents paid APIs and agent tool calls, not charging crawlers. - **Human Native.** Cloudflare announced its acquisition on 15 Jan 2026, so it is covered in the Cloudflare profile. ## Download the list One row per vendor: type, operator, headquarters, products with their prices, the standards assessed, our probes, what the vendor does not do, and every sourced fact in the JSON file. Edition 2026-11-05. - [vendors.csv](https://agentreadycrawlers.com/vendors/vendors.csv) text/csv, about 55 KB - [vendors.json](https://agentreadycrawlers.com/vendors/vendors.json) application/json, about 412 KB **Cite this.** Agent Ready Crawlers, "Vendors", edition 2026-11-05, https://agentreadycrawlers.com/vendors/. Free to use with this line; see [the terms](https://agentreadycrawlers.com/terms/). ## Sources - [Method: index, block rates, vendors (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Amazon: amazon.com](https://www.amazon.com) read 2026-11-05 - [Anchor Browser: anchorbrowser.io](https://anchorbrowser.io) read 2026-11-05 - [Anthropic: anthropic.com](https://www.anthropic.com) read 2026-11-05 - [Apify: apify.com](https://apify.com) read 2026-11-05 - [Apple: apple.com](https://www.apple.com) read 2026-11-05 - [Block: block.xyz](https://block.xyz) read 2026-11-05 - [Browser Use: browser-use.com](https://browser-use.com) read 2026-11-05 - [Browserbase: browserbase.com](https://www.browserbase.com) read 2026-11-05 - [Cohere: cohere.com](https://cohere.com) read 2026-11-05 - [Common Crawl: commoncrawl.org](https://commoncrawl.org) read 2026-11-05 - [Diffbot: diffbot.com](https://www.diffbot.com) read 2026-11-05 - [DuckDuckGo: duckduckgo.com](https://duckduckgo.com) read 2026-11-05 - [Exa: exa.ai](https://exa.ai) read 2026-11-05 - [Firecrawl: firecrawl.dev](https://www.firecrawl.dev) read 2026-11-05 - [Google: google.com](https://www.google.com) read 2026-11-05 - [Hive: deploy.com](https://deploy.com) read 2026-11-05 - [Huawei: huawei.com](https://www.huawei.com) read 2026-11-05 - [Hyperbrowser: hyperbrowser.ai](https://www.hyperbrowser.ai) read 2026-11-05 - [Kernel: kernel.sh](https://www.kernel.sh) read 2026-11-05 - [Manus: manus.im](https://manus.im) read 2026-11-05 - [Meta: meta.com](https://www.meta.com) read 2026-11-05 - [Microsoft: microsoft.com](https://www.microsoft.com) read 2026-11-05 - [Mistral AI: mistral.ai](https://mistral.ai) read 2026-11-05 - [Notte: notte.cc](https://www.notte.cc) read 2026-11-05 - [OpenAI: openai.com](https://openai.com) read 2026-11-05 - [Parallel Web Systems: parallel.ai](https://parallel.ai) read 2026-11-05 - [Perplexity: perplexity.ai](https://www.perplexity.ai) read 2026-11-05 - [Timpi: timpi.io](https://timpi.io) read 2026-11-05 - [Webz.io: webz.io](https://webz.io) read 2026-11-05 - [xAI: x.ai](https://x.ai) read 2026-11-05 - [You.com: you.com](https://you.com) read 2026-11-05 - [Akamai: akamai.com](https://www.akamai.com) read 2026-11-05 - [AWS WAF Bot Control: aws.amazon.com](https://aws.amazon.com/waf/features/bot-control/) read 2026-11-05 - [Arcjet: arcjet.com](https://arcjet.com) read 2026-11-05 - [Castle: castle.io](https://castle.io) read 2026-11-05 - [Centinel Analytica: centinelanalytica.com](https://www.centinelanalytica.com) read 2026-11-05 - [Cequence Security: cequence.ai](https://www.cequence.ai) read 2026-11-05 - [Cloudflare: cloudflare.com](https://www.cloudflare.com) read 2026-11-05 - [DataDome: datadome.co](https://datadome.co) read 2026-11-05 - [F5: f5.com](https://www.f5.com) read 2026-11-05 - [Fastly: fastly.com](https://www.fastly.com) read 2026-11-05 - [Fingerprint: fingerprint.com](https://fingerprint.com) read 2026-11-05 - [Google Cloud Fraud Defense: cloud.google.com](https://cloud.google.com/security/products/recaptcha) read 2026-11-05 - [hCaptcha: hcaptcha.com](https://www.hcaptcha.com) read 2026-11-05 - [HUMAN Security: humansecurity.com](https://www.humansecurity.com) read 2026-11-05 - [Imperva: imperva.com](https://www.imperva.com) read 2026-11-05 - [Kasada: kasada.io](https://www.kasada.io) read 2026-11-05 - [Known Agents: knownagents.com](https://knownagents.com) read 2026-11-05 - [Link11: link11.com](https://www.link11.com) read 2026-11-05 - [Netacea: netacea.com](https://netacea.com) read 2026-11-05 - [Netlify: netlify.com](https://www.netlify.com) read 2026-11-05 - [Radware: radware.com](https://www.radware.com) read 2026-11-05 - [Vercel: vercel.com](https://vercel.com) read 2026-11-05 - [Verisoul: verisoul.ai](https://www.verisoul.ai) read 2026-11-05 - [American Express: americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/) read 2026-11-05 - [Auth0 for AI Agents: auth0.com](https://auth0.com/ai) read 2026-11-05 - [Descope: descope.com](https://www.descope.com/use-cases/ai) read 2026-11-05 - [Mastercard: developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/) read 2026-11-05 - [Microsoft Entra Agent ID: learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities) read 2026-11-05 - [Prove: prove.com](https://www.prove.com/solutions/agentic-suite) read 2026-11-05 - [Stytch IsAgent: stytch.com](https://stytch.com) read 2026-11-05 - [Trulioo: trulioo.com](https://www.trulioo.com/products/kya-verification) read 2026-11-05 - [Visa: developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol) read 2026-11-05 - [Vouched: vouched.id](https://www.vouched.id/know-your-agent) read 2026-11-05 - [World: world.org](https://world.org) read 2026-11-05 - [Dappier: dappier.com](https://dappier.com) read 2026-11-05 - [FetchRight: fetchright.ai](https://fetchright.ai) read 2026-11-05 - [IAB Tech Lab: iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/) read 2026-11-05 - [ProRata: prorata.ai](https://prorata.ai) read 2026-11-05 - [RSL Collective: rslcollective.org](https://rslcollective.org) read 2026-11-05 - [Skyfire: skyfire.xyz](https://skyfire.xyz) read 2026-11-05 - [Supertab: supertab.co](https://www.supertab.co) read 2026-11-05 - [TollBit: tollbit.com](https://tollbit.com) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Akamai: bot management profile | Agent Ready Crawlers" description: "Akamai: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15." canonical: https://agentreadycrawlers.com/vendors/akamai site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Akamai: bot management profile | Agent Ready Crawlers > Akamai: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. Source: https://agentreadycrawlers.com/vendors/akamai Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 1 (tied) of 15 # Akamai In short Akamai is a CDN and security company whose bot controls verify agents at the edge and take Visa's Trusted Agent Protocol signals, but paid access runs through TollBit and Skyfire, so charging and per-purpose policy grade only partial, for 8 of 10 points. No price is public; Bot Manager is sold through sales. CDN and security vendor whose Bot & Agent Control products verify Web Bot Auth signatures at the edge, plug in Visa TAP and Skyfire KYA identity, and hand AI bot monetization to TollBit and Skyfire. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Akamai](https://agentreadycrawlers.com/vendors/akamai/alternatives/) · [akamai.com](https://www.akamai.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [akamai.com](https://www.akamai.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | Web Bot Auth (RFC 9421) at the edge, verified bot list incl. AI bots; Visa TAP and Skyfire/Experian KYA via the agentic framework | Blog states Web Bot Auth support in bot and abuse solutions; framework release names TAP and KYA. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | allow, challenge, delay or block per verified bot or category; no published search versus training switch (checked 2026-11-04) | Policy actions on Web Bot Auth verified traffic. Source, read 2026-11-04. | | Charging for access | Partial | through TollBit and Skyfire partners, no native billing | Partnership release with TollBit and Skyfire. Source, read 2026-11-04. | | Lets signed commerce agents through | Yes | Visa Trusted Agent Protocol integration, Skyfire and Experian KYA | Visa release Dec 2025 and framework release Jun 2026. Source, read 2026-11-04. | | Cost | Not graded | no public price; sales-led | No price on framework release or product pages read. Source, read 2026-11-04. | | Reporting | Yes | TrafficPeak and web security analytics | Framework release names TrafficPeak for observability. Source, read 2026-11-04. | | Where it runs | Not graded | edge: verification and enforcement on Akamai's edge network | Edge verification described in Web Bot Auth blog. Source, read 2026-11-04. | ## Cost no public price; sales-led (checked 2026-11-04) [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04. ## What Akamai does not do - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). - Akamai publishes no price for Bot Manager or Bot & Agent Control; the June 2026 release names partners and pillars only. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Akamai says Web Bot Auth is already supported by its bot and abuse solutions, including bot visibility and mitigation in App & API Protector. [Source](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication), read 2026-11-04, dated 2025-11-10. - Primary source For Web Bot Auth the Akamai edge fetches the bot's signature directory and validates the signature under RFC 9421 rules before the request reaches origin. [Source](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication), read 2026-11-04, dated 2025-11-10. - Primary source Traffic verified with Web Bot Auth can be allowed or mitigated with challenge, delay or block actions. [Source](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication), read 2026-11-04, dated 2025-11-10. - Primary source Bot Manager treats AI bots from platforms such as OpenAI, Perplexity and Gemini as part of its good or verified bot category, alongside search, social and SEO bots. [Source](https://www.akamai.com/blog/security/bot-management-agentic-era), read 2026-11-04, dated 2025-11-20. - Primary source On 15 Jun 2026 Akamai announced a unified agentic framework for its Bot & Agent Control solutions that joins identity, observability, trust and edge security into one real-time decision layer. [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04, dated 2026-06-15. - Primary source The framework release names integrations with Visa's Trusted Agent Protocol, Skyfire and Experian Know Your Agent, Auth0, Ping Identity and TollBit, with TrafficPeak for observability. [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04, dated 2026-06-15. - Primary source Akamai and Visa announced a collaboration integrating Visa's Trusted Agent Protocol with Akamai's edge behavioral intelligence, user recognition and bot and abuse protection. [Source](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce), read 2026-11-04, dated 2025-12-17. - Primary source Akamai partners with TollBit and Skyfire so publishers can redirect AI bot operators to register and pay rather than only block them. [Source](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue), read 2026-11-04. - Primary source AWS's AgentCore Browser documentation lists Akamai among the bot control vendors that recognize its Web Bot Auth signatures. [Source](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html), read 2026-11-04. - Vendor claim Akamai's own blog says adoption of Web Bot Auth, KYAPay and TAP will take time and investment from all parties. [Source](https://www.akamai.com/blog/security/bot-management-agentic-era), read 2026-11-04, dated 2025-11-20. - Primary source On 27 Oct 2026 Akamai launched Akamai Inference and Akamai Agent Sandbox, an isolated execution environment for AI agents on Akamai Cloud. [Source](https://www.akamai.com/newsroom/press-release/akamai-launches-ai-inference-service-and-agent-execution-environment), read 2026-11-04, dated 2026-10-27. ## Dated news - 2026-10-27**Akamai launches Akamai Inference and Agent Sandbox**. Akamai introduced managed model serving with OpenAI-compatible APIs and Akamai Agent Sandbox, an isolated environment where AI agents can run code, spanning core regions and edge locations. This is on the agent-hosting side rather than bot management. [Source](https://www.akamai.com/newsroom/press-release/akamai-launches-ai-inference-service-and-agent-execution-environment). ## Compared with - [Akamai vs Cloudflare](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare/): Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). - [DataDome vs Akamai](https://agentreadycrawlers.com/compare/datadome-vs-akamai/): DataDome publishes plan prices and lists AI monetization from USD 8,670 a month; Akamai publishes no prices and names Visa TAP, Skyfire and Experian KYA, Auth0 and TollBit in its June 2026 agentic framework. - [AWS WAF Bot Control vs Akamai](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai/): AWS WAF publishes Bot Control prices (USD 10 a month per web ACL, USD 1 per million common requests after 10 million) and charges bots natively over x402; Akamai is sales-led and charges through partners. - [Akamai vs HUMAN Security](https://agentreadycrawlers.com/compare/akamai-vs-human-security/): Akamai is listed by OpenAI as recognising ChatGPT's Cloud browser under the bot name "ChatGPT Agent"; HUMAN lists the same agent in Known Bots & Crawlers and AgenticTrust permissions. - [Fastly vs Akamai](https://agentreadycrawlers.com/compare/fastly-vs-akamai/): Both route AI bot charging to TollBit, and both partner with Skyfire (Fastly 2026-06-10, Akamai 2026-06-15); Akamai documents Web Bot Auth verification and Visa TAP, Fastly docs read did not. - [Kasada vs Akamai](https://agentreadycrawlers.com/compare/kasada-vs-akamai/): Kasada says it can be deployed behind an incumbent bot manager; Akamai's Bot & Agent Control runs on Akamai's own edge with partner integrations for identity and charging. ## What Akamai does for this layer ### Claims the vendor makes about itself Akamai said, as reported by Digital Commerce 360, that nine of the world's ten largest retailers rely on it, and that its 2025 Digital Fraud and Abuse Report found AI-powered bot traffic up 300% in a year. ([digitalcommerce360.com](https://www.digitalcommerce360.com/2025/12/17/akamai-visa-partner-fraud-prevention-agentic-commerce/), dated 2025-12-17) Akamai's figures in the same report put the commerce sector at more than 25 billion AI bot requests during a two-month period. ([digitalcommerce360.com](https://www.digitalcommerce360.com/2025/12/17/akamai-visa-partner-fraud-prevention-agentic-commerce/), dated 2025-12-17) ### Reported by a third party TollBit's chief executive is quoted in Akamai's June 2026 release describing the flow: Akamai identifies agentic traffic at the edge and redirects it to TollBit's Agent Site, where the site owner sets access rules and charges. ([stocktitan.net](https://www.stocktitan.net/news/AKAM/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-kkg1mkunpvj5.html), dated 2026-06-15) Akamai names TrafficPeak, its log analytics product, as where customers separate human users, beneficial AI agents and malicious bots over time. ([stocktitan.net](https://www.stocktitan.net/news/AKAM/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-kkg1mkunpvj5.html), dated 2026-06-15) ### What our crawl saw www.akamai.com answered the crawl user agent with HTTP 403 on 2026-11-05, robots.txt included, so the facts in this profile come from a wire reprint, trade press and one Akamai blog post read with a different fetch tool. ([akamai.com](https://www.akamai.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot Manager / Bot & Agent Control | Edge bot and agent classification with Web Bot Auth verification, Trusted Agent Protocol and KYA identity, and partner monetization. | sales-led The June 2026 release gives no prices or availability dates. | Not published | 2026-11-05 | | TrafficPeak | Log analytics used to observe how humans, agents and bots use a site. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Akamai's post of 10 Nov 2025 says its bot and abuse solutions already support Web Bot Auth and that the Akamai edge validates the signature under RFC 9421 rules. (akamai.com) | | Content Signals | No statement | No Content Signals statement in the sources read on 2026-11-05. | | Visa Trusted Agent Protocol | Supports | Akamai and Visa integrated the Trusted Agent Protocol with Akamai's edge protection (Dec 2025). (digitalcommerce360.com) | | Mastercard Agent Pay | No statement | Neither the June 2026 release nor the December 2025 coverage names Mastercard Agent Pay. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Akamai (Bot & Agent Control)](https://agentreadycrawlers.com/vendors/akamai/alternatives/) graded on the rubric ## Sources - [akamai.com: akamai.com](https://www.akamai.com) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [akamai.com: redefine trust web bot authentication](https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [akamai.com: akamai and visa join forces to secure the next era of agentic commerce](https://www.akamai.com/newsroom/press-release/akamai-and-visa-join-forces-to-secure-the-next-era-of-agentic-commerce) read 2026-11-04 - [akamai.com: bot management agentic era](https://www.akamai.com/blog/security/bot-management-agentic-era) read 2026-11-04 - [docs.aws.amazon.com: browser web bot auth](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html) read 2026-11-04 - [akamai.com: akamai launches ai inference service and agent execution environment](https://www.akamai.com/newsroom/press-release/akamai-launches-ai-inference-service-and-agent-execution-environment) read 2026-11-04 - [Akamai launches agentic AI security framework for commerce | AKAM Stock News](https://www.stocktitan.net/news/AKAM/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-kkg1mkunpvj5.html) read 2026-11-05 - [Akamai, Visa partner on fraud prevention for agentic commerce](https://www.digitalcommerce360.com/2025/12/17/akamai-visa-partner-fraud-prevention-agentic-commerce/) read 2026-11-05 - [Akamai (homepage, HTTP 403 to the crawl user agent)](https://www.akamai.com/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Akamai | Agent Ready Crawlers" description: "Alternatives to Akamai: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/akamai/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Akamai | Agent Ready Crawlers > Alternatives to Akamai: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/akamai/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Akamai In short Akamai ties with Cloudflare, DataDome and AWS WAF Bot Control at the top of the rubric, each on 8 of 10 points, and head-to-head pages cover six of its rivals. The points hide a gap: Akamai sends paid access through TollBit or Skyfire rather than billing crawlers itself, and it publishes no price. ## Head-to-head, with the sourced difference - [**Akamai vs Cloudflare**](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare/). Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). [Source](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue), read 2026-11-04. - [**Akamai vs DataDome**](https://agentreadycrawlers.com/compare/datadome-vs-akamai/). DataDome publishes plan prices and lists AI monetization from USD 8,670 a month; Akamai publishes no prices and names Visa TAP, Skyfire and Experian KYA, Auth0 and TollBit in its June 2026 agentic framework. [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04. - [**Akamai vs AWS WAF Bot Control**](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai/). AWS WAF publishes Bot Control prices (USD 10 a month per web ACL, USD 1 per million common requests after 10 million) and charges bots natively over x402; Akamai is sales-led and charges through partners. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - [**Akamai vs HUMAN Security**](https://agentreadycrawlers.com/compare/akamai-vs-human-security/). Akamai is listed by OpenAI as recognising ChatGPT's Cloud browser under the bot name "ChatGPT Agent"; HUMAN lists the same agent in Known Bots & Crawlers and AgenticTrust permissions. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. - [**Akamai vs Fastly**](https://agentreadycrawlers.com/compare/fastly-vs-akamai/). Both route AI bot charging to TollBit, and both partner with Skyfire (Fastly 2026-06-10, Akamai 2026-06-15); Akamai documents Web Bot Auth verification and Visa TAP, Fastly docs read did not. [Source](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce), read 2026-11-04. - [**Akamai vs Kasada**](https://agentreadycrawlers.com/compare/kasada-vs-akamai/). Kasada says it can be deployed behind an incumbent bot manager; Akamai's Bot & Agent Control runs on Akamai's own edge with partner integrations for identity and charging. [Source](https://www.kasada.io/), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Akamai's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Akamai on | Weaker on | Cost | | --- | --- | --- | --- | --- | | AWS WAF Bot Control | 8 | per-purpose policy, charging | commerce agents | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | per-purpose policy | commerce agents | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | charging | commerce agents | From USD 3,830/month | | HUMAN Security | 7 | per-purpose policy | commerce agents | No public price; sales-led | | Skyfire | 6 | charging | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | charging | verifies agents, commerce agents | Free for publishers to start | | Vercel | 6 | per-purpose policy | charging, commerce agents | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | nothing | charging, commerce agents | No public price; sales-led | | Fastly | 4 | nothing | verifies agents, commerce agents, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, charging, commerce agents, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | nothing | verifies agents, charging, commerce agents, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy, charging, commerce agents | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, charging, commerce agents, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, charging, commerce agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Akamai is not the fit - No native per-crawl billing; monetization runs through TollBit or Skyfire (Akamai blog and partnership release). - The June 2026 framework release gives no availability dates or prices (press release read 2026-11-04). - No published control that separates search crawling from model training as purposes (pages read 2026-11-04). ## Sources - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [investors.fastly.com: fastly and skyfire partner enable trusted agentic commerce](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Amazon: crawler operator profile | Agent Ready Crawlers" description: "Amazon: crawler operator profile covering Amazonbot, Amzn-SearchBot, Amzn-User and 1 more. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/amazon site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Amazon: crawler operator profile | Agent Ready Crawlers > Amazon: crawler operator profile covering Amazonbot, Amzn-SearchBot, Amzn-User and 1 more. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/amazon Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Amazon In short Amazon publishes three crawler identities with separate robots.txt settings: Amazonbot may feed model training, while Amzn-SearchBot and Amzn-User do not. None of them supports crawl-delay, and robots.txt copies can be 30 days old. Buy for Me, in beta since April 2025, buys from brand sites for shoppers but has no documented identifier. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [amazon.com](https://www.amazon.com) In one line Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent. Last read 2026-11-05 ## What Amazon does for this layer ### From the vendor's own pages Amazon documents three user agents, Amazonbot, Amzn-SearchBot and Amzn-User, says each setting is independent, and says changes may take about 24 hours to reach its systems. ([developer.amazon.com](https://developer.amazon.com/docs/amazonbot/amazonbot.html)) Amazon says Amazonbot content may be used to train Amazon AI models, while Amzn-SearchBot and Amzn-User do not crawl content for generative AI model training. ([developer.amazon.com](https://developer.amazon.com/docs/amazonbot/amazonbot.html)) Amazon says that when robots.txt does not mention Amzn-SearchBot but allows other search bots, Amzn-SearchBot follows the directives given to those bots. ([developer.amazon.com](https://developer.amazon.com/docs/amazonbot/amazonbot.html)) Amazon's crawlers fetch robots.txt per host or use a cached copy up to 30 days old, treat an unfetchable file as absent, and do not support crawl-delay. ([developer.amazon.com](https://developer.amazon.com/docs/amazonbot/amazonbot.html)) Amazon's crawlers honour rel=nofollow and the robots meta values noarchive, noindex and none, and Amazon reads noarchive as do not use the page for model training. ([developer.amazon.com](https://developer.amazon.com/docs/amazonbot/amazonbot.html)) Amazon launched Buy for Me in beta on 2025-04-03 to buy items from other brands' websites on a customer's behalf, and says the feature runs on Amazon Bedrock with Anthropic's Claude models. ([aboutamazon.com](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands), dated 2025-04-03) Amazon's Shop Direct Merchant Workspace, announced 2026-11-04, lets eligible merchants register their stores for Shop Direct, whose products may be bought through Buy for Me; Amazon says there are no fees and merchants can opt out at any time. ([aboutamazon.com](https://www.aboutamazon.com/news/retail/amazon-shop-direct-merchant-workspace), dated 2026-11-04) ### Claims the vendor makes about itself Amazon's crawler page says sites that allow Amazonbot may qualify for Amazon Content Partners, which offers a "+1% affiliate commission boost", hosting credits and AI traffic management tools. ([developer.amazon.com](https://developer.amazon.com/docs/amazonbot/amazonbot.html)) ### What our crawl saw Amazon's Amzn-User range page embedded a JSON object with 1,023 IPv4 prefixes and a creationTime of 2025-11-04 when fetched on 2026-11-05. ([developer.amazon.com](https://developer.amazon.com/amazonbot/live-ip-addresses/), dated 2025-11-04) www.amazon.com answered HTTP 404 at the Web Bot Auth key directory path on 2026-11-05. ([amazon.com](https://www.amazon.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Amazonbot | Crawler that improves Amazon products and may train Amazon AI models. | No price recorded | | Amzn-SearchBot | Crawler for search experiences such as Alexa; not used for generative model training. | No price recorded | | Amzn-User | Fetcher for live answers to Alexa queries; may not follow all robots.txt directives. | No price recorded | | Buy for Me | Agent in the Amazon Shopping app that completes purchases on brand websites for the customer. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | Automated crawling honours the Robots Exclusion Protocol; Amzn-User may not follow all directives when a user starts the action. (developer.amazon.com) | | Web Bot Auth | No statement | Not on the crawler page; key directory path 404 on www.amazon.com (observed 2026-11-05). | | Published IP list | Supports | Three range pages, one per user agent, each embedding a JSON object. (developer.amazon.com) | | Reverse DNS check | No statement | No reverse DNS pattern on the crawler page (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 202 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | Amazonbot range page | 1,292 IPv4 prefixes, creationTime 2026-09-08 | GET and count prefixes in the embedded JSON | 2026-11-05 | | Amzn-SearchBot range page | 816 IPv4 prefixes, creationTime 2026-09-08 | GET and count prefixes in the embedded JSON | 2026-11-05 | | Amzn-User range page | 1,023 IPv4 prefixes, creationTime 2025-11-04 | GET and count prefixes in the embedded JSON | 2026-11-05 | ## What Amazon does not do - Amazon's crawler page documents no identifier for Buy for Me or Rufus traffic (read 2026-11-05). - Amazon names no robots.txt token for Buy for Me; the opt-out it describes is for merchants enrolled in Shop Direct. - Amazon's crawlers ignore crawl-delay, so rate limits need server-side controls. ## Reading notes Amazon's cloud products, AWS WAF and AgentCore Browser, are profiled under [AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Amazon in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/amazon/) 5 rows - [Amazonbot](https://agentreadycrawlers.com/crawlers/amazonbot/) token Amazonbot - [Amzn-SearchBot](https://agentreadycrawlers.com/crawlers/amzn-searchbot/) token Amzn-SearchBot - [Amzn-User](https://agentreadycrawlers.com/crawlers/amzn-user/) token Amzn-User - [Buy for Me (shopping agent)](https://agentreadycrawlers.com/crawlers/amazon-buy-for-me/) no token - [Rufus (shopping assistant)](https://agentreadycrawlers.com/crawlers/amazon-rufus/) no token ## Sources - [amazon.com](https://www.amazon.com) read 2026-11-05 - [Amazon Web Search crawlers (Amazonbot)](https://developer.amazon.com/docs/amazonbot/amazonbot.html) read 2026-11-05 - [Amazonbot Live Crawl IP addresses (Amzn-User)](https://developer.amazon.com/amazonbot/live-ip-addresses/) read 2026-11-05 - [Amazonbot IP addresses](https://developer.amazon.com/amazonbot/ip-addresses/) read 2026-11-05 - [Amazon's new Buy for Me feature helps customers find and buy products from other brands' sites](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands) read 2026-11-05 - [Amazon launches Shop Direct Merchant Workspace for merchants](https://www.aboutamazon.com/news/retail/amazon-shop-direct-merchant-workspace) read 2026-11-05 - [www.amazon.com Web Bot Auth key directory path (GET probe)](https://www.amazon.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [Amazon Searchbot IP addresses](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "American Express: vendor profile | Agent Ready Crawlers" description: "American Express: agent verification provider profile covering ACE developer kit and Amex Agent Purchase Protection. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/american-express site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # American Express: vendor profile | Agent Ready Crawlers > American Express: agent verification provider profile covering ACE developer kit and Amex Agent Purchase Protection. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/american-express Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # American Express In short American Express offers developers five services under ACE, spanning registering agents, enabling accounts, reading intent, issuing credentials and adding cart context. As of 14 April 2026 the kit was half built: registration and cart context were still in development, and the list of registered agents said Coming soon. Cardholders will manage agent spending from the Amex App. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/) In one line Card network whose ACE developer kit plans an agent registration service so only verified AI agents get tokenized Amex credentials, with Agent Purchase Protection for registered agent purchases; the registration spec is still in development. Last read 2026-11-05 ## What American Express does for this layer ### From the vendor's own pages American Express describes its Agentic Commerce Experiences (ACE) developer kit as five services: Identify for agent registration, Activate for account enablement, Contextualize for intent intelligence, Fulfill for payment credentials and Enhance for cart context. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) American Express says the Account Enablement, Intent Intelligence and Payment Credentials specifications were available as of 14 Apr 2026, while Agent Registration and Cart Context were still under development. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/), dated 2026-04-14) American Express says its Identify service will ensure that only verified AI agents are authorized to transact on the American Express network. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) American Express says it will issue tokenized agentic payment credentials only to verified agents and will verify the Card Member's approval of each purchase intent. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) Amex Agent Purchase Protection is described as protection for eligible customers from charges caused by AI agent error when a registered agent sends American Express the customer's authenticated purchase intent. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) American Express says a full list of registered agents will be published "Coming soon", and no list was on the page on 2026-11-05. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) American Express says Card Members will manage spend controls and active purchase intents for agent transactions in the American Express App. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) ### Claims the vendor makes about itself The ACE page shows Adyen, Cloudflare, Crossmint, Fiserv, Forter, Global Payments, Google, Microsoft, Nekuda, OpenAI, PayPal, Stripe and VGS among the companies supporting the programme. ([americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/)) ### Reported by a third party Mastercard's merchant playbook lists the American Express ACE Developer Kit with Agent Purchase Protection as one of three network frameworks merchants should plan for. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | ACE developer kit | Specifications for agent registration, account enablement, purchase intent, tokenized credentials and cart context. | No price recorded | | Amex Agent Purchase Protection | Card Member protection from charges caused by registered agent error, tied to authenticated purchase intent. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The ACE page shows Cloudflare's logo but describes no request-signing scheme (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol mention on the ACE page (checked 2026-11-05). | | Mastercard Agent Pay | No statement | No Agent Pay mention on the ACE page (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What American Express does not do - American Express has not published the Agent Registration specification, so a site cannot yet verify an Amex-registered agent at the door (page read 2026-11-05). - Amex Agent Purchase Protection covers Card Members' charges; it gives merchants no crawler or bot policy. ## Reading notes Amex's newsroom release and its developer overview are JavaScript pages that a plain fetch cannot read, so the facts come from the company's agentic commerce page. ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management ## Sources - [Agentic Commerce Experiences (American Express)](https://www.americanexpress.com/en-us/company/agentic-commerce/) read 2026-11-05 - [Developer Playbook on Preparing for Agentic Commerce (Mastercard Developers)](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Anchor Browser: crawler operator profile | Agent Ready Crawlers" description: "Anchor Browser: crawler operator profile covering Anchor cloud browsers, Web Unlocker and OmniConnect. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/anchor site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Anchor Browser: crawler operator profile | Agent Ready Crawlers > Anchor Browser: crawler operator profile covering Anchor cloud browsers, Web Unlocker and OmniConnect. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/anchor Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Anchor Browser In short Anchor Browser leases managed Chromium sessions to computer-use agents, from USD 50 a month for the Starter tier that adds CAPTCHA bypass. A session carries a Web Bot Auth signature only if the customer switches it on and holds the USD 2,000 Growth plan or Enterprise. Anchor was among the launch partners of Cloudflare's signed-agent programme in August 2025. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Operator Anchor Homepage [anchorbrowser.io](https://anchorbrowser.io) In one line Cloud browser infrastructure for computer-use agents that can sign sessions with Cloudflare Web Bot Auth on its upper plans, and also sells stealth fingerprinting, proxies, CAPTCHA bypass and a pay-on-success web unlocker. Last read 2026-11-05 ## What Anchor Browser does for this layer ### From the vendor's own pages Anchor Browser signs all HTTP requests in a session as Anchor Browser when the session setting web\_bot\_auth.active is true; the setting defaults to false. ([docs.anchorbrowser.io](https://docs.anchorbrowser.io/advanced/cloudflare-web-bot-auth.md)) Anchor's pricing table lists "Cloudflare verified browser agents" only on the Growth plan at USD 2,000 a month and on Enterprise, not on Free or Starter (USD 50 a month). ([docs.anchorbrowser.io](https://docs.anchorbrowser.io/pricing.md)) Anchor's Free plan gives USD 5 in credits a month with up to 5 concurrent browsers; automated captcha bypass and geolocation proxies start on Starter. ([docs.anchorbrowser.io](https://docs.anchorbrowser.io/pricing.md)) Anchor's docs index lists Extra Stealth Mode for anti-detection and fingerprint protection, bring-your-own proxy, dedicated sticky IPs, Anchor VPN and a Web Unlocker for CAPTCHA- and bot-protected pages that is "billed only on success". ([docs.anchorbrowser.io](https://docs.anchorbrowser.io/llms.txt)) Anchor's homepage cost calculator prices browser time at USD 0.09 an hour, proxy traffic at USD 8.00 per GB and AI steps at USD 0.01 each. ([anchorbrowser.io](https://anchorbrowser.io/)) ### Claims the vendor makes about itself Anchor describes its product as "fully managed, humanized chromium instances that can assume any identity and access any website". ([anchorbrowser.io](https://anchorbrowser.io/)) ### Reported by a third party Cloudflare named Anchor Browser in its first signed-agent cohort on 2025-08-28, quoting Anchor's CEO that the partnership lets developers "ship verified browser agents". ([blog.cloudflare.com](https://blog.cloudflare.com/signed-agents/), dated 2025-08-28) ### What our crawl saw A GET for anchorbrowser.io/.well-known/http-message-signatures-directory returned HTTP 404 on 2026-11-05, and the Web Bot Auth doc names no Signature-Agent host. ([anchorbrowser.io](https://anchorbrowser.io/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Anchor cloud browsers | Managed Chromium sessions for agents with stealth, proxies, identities with MFA and optional Web Bot Auth. | subscription plus credits Starter a month; Web Bot Auth signing needs Growth at USD 2,000 a month or Enterprise. | USD 50 | 2026-11-05 | | Web Unlocker | Sessionless fetch of rendered HTML from CAPTCHA- and bot-protected pages, billed on success. | No price recorded | | OmniConnect | Authentication for computer-use agents, announced on the homepage. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | No robots.txt page in the docs index (2026-11-05). | | Web Bot Auth | Supports | Opt-in web_bot_auth.active session flag, default false. (docs.anchorbrowser.io) | | Cloudflare verified bots | Supports | Pricing lists Cloudflare verified browser agents on Growth and Enterprise. (docs.anchorbrowser.io) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://anchorbrowser.io/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Anchor Browser does not do - Anchor does not sign sessions unless the customer turns web\_bot\_auth on, and signing is limited to the Growth and Enterprise plans. - Anchor publishes no robots.txt policy or token for its sessions (docs index checked 2026-11-05). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Anchor Browser](https://agentreadycrawlers.com/crawlers/anchor-browser/) no token ## Sources - [anchorbrowser.io](https://anchorbrowser.io) read 2026-11-05 - [Anchor - Secure Infrastructure for Computer Use Agents](https://anchorbrowser.io/) read 2026-11-05 - [Cloudflare Web Bot Auth | Anchor Browser docs](https://docs.anchorbrowser.io/advanced/cloudflare-web-bot-auth.md) read 2026-11-05 - [Pricing | Anchor Browser docs](https://docs.anchorbrowser.io/pricing.md) read 2026-11-05 - [Anchor Browser documentation index (llms.txt)](https://docs.anchorbrowser.io/llms.txt) read 2026-11-05 - [The age of agents: cryptographically recognizing agent traffic | Cloudflare Blog](https://blog.cloudflare.com/signed-agents/) read 2026-11-05 - [Anchor key directory probe](https://anchorbrowser.io/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Anthropic: crawler operator profile | Agent Ready Crawlers" description: "Anthropic: crawler operator profile covering ClaudeBot, Claude-SearchBot, Claude-User and 1 more. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/anthropic site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Anthropic: crawler operator profile | Agent Ready Crawlers > Anthropic: crawler operator profile covering ClaudeBot, Claude-SearchBot, Claude-User and 1 more. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/anthropic Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Anthropic In short Anthropic sorts its crawling into three robots, ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests, plus a hosted browser that adds a Claude-Agent token to the browser's own user agent. Since 30 October 2026 it also reads an IETF draft no-training preference. It publishes no signing keys. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [anthropic.com](https://www.anthropic.com) In one line Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. Last read 2026-11-05 ## What Anthropic does for this layer ### From the vendor's own pages Anthropic's help article lists three robots, ClaudeBot, Claude-User and Claude-SearchBot, and says restricting ClaudeBot signals that a site's future materials should be left out of AI training datasets. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic supports the non-standard Crawl-delay extension to robots.txt and gives "Crawl-delay: 1" under "User-agent: ClaudeBot" as its example. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs on the sites they crawl. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic warns that blocking its IP addresses may not work as an opt-out, because it stops the bots from reading robots.txt, and asks for a robots.txt rule on every subdomain. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic began recognising a machine-readable "no AI training" preference on 2026-10-30, based on the IETF aipref draft draft-kelley-aipref-training-preference-signal, and is rolling it out across its services over time. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), dated 2026-10-30) Anthropic treats a robots.txt rule that names ClaudeBot as more specific than the general training preference, so the ClaudeBot rule wins for that path. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic-hosted browsers, used by Claude Duet, send the browser's own user agent with the added token Claude-Agent, and Anthropic advises matching that token rather than the full string. ([support.claude.com](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf)) Anthropic publishes the egress addresses for Claude-Agent at claude.com/.well-known/sail in the draft-kelley-webbotauth-sail-00 format and recommends updating rules from that file automatically. ([support.claude.com](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf)) ### What our crawl saw Anthropic's SAIL file listed nine IPv4 addresses in an agentic-browser-use prefix set marked exclusive, modified 2026-10-22, with expires\_after set to 3600 seconds, when fetched on 2026-11-05. ([claude.com](https://claude.com/.well-known/sail), dated 2026-10-22) Anthropic's bots.json, the range file for its three robots, listed 14 IPv4 prefixes with a creationTime of 2026-10-20 when fetched on 2026-11-05. ([claude.com](https://claude.com/crawling/bots.json), dated 2026-10-20) Anthropic served no Web Bot Auth key directory on 2026-11-05: the well-known path returned HTTP 404 on claude.com, claude.ai and www.anthropic.com. ([claude.com](https://claude.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | ClaudeBot | Crawler that collects web content that may contribute to model training. | No price recorded | | Claude-SearchBot | Crawler that indexes content to improve search results for Claude users. | No price recorded | | Claude-User | Fetcher that retrieves pages when a Claude user asks a question. | No price recorded | | Claude-Agent hosted browser | Browser on Anthropic infrastructure that Claude Duet drives for users; identified by the Claude-Agent token and the SAIL IP file. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Anthropic's bots honour robots.txt directives and Crawl-delay; Anthropic names robots.txt as the opt-out. (support.claude.com) | | Web Bot Auth | No statement | No key directory on claude.com, claude.ai or anthropic.com (observed 404, 2026-11-05) and no mention in either help article. | | Content Signals | No statement | Anthropic cites the IETF aipref training signal, not Content-Signals (read 2026-11-05). | | Published IP list | Supports | bots.json for the three robots and a SAIL file for Claude-Agent. (support.claude.com) | | Reverse DNS check | No statement | Neither help article gives a reverse DNS pattern (read 2026-11-05). | | Cloudflare verified bots | No statement | Not mentioned in Anthropic's help articles (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 on claude.com, claude.ai and www.anthropic.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | bots.json | 14 IPv4 prefixes, creationTime 2026-10-20 | GET and count prefixes | 2026-11-05 | | SAIL file | 9 IPv4 addresses for Claude-Agent, modified 2026-10-22, expires_after 3600 | GET and read the TOML | 2026-11-05 | ## What Anthropic does not do - Anthropic publishes no signing keys for Claude-Agent, so the token and the SAIL address file are the only checks a site can run. - Anthropic's hosted-browser article covers Claude Duet's hosted browser only and gives no identifier for Claude in Chrome (read 2026-11-05). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Anthropic in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/anthropic/) 7 rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) no token ## Sources - [anthropic.com](https://www.anthropic.com) read 2026-11-05 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-05 - [How do Anthropic-hosted browsers identify themselves when they browse the web on a user's behalf?](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf) read 2026-11-05 - [Anthropic SAIL file (Claude-Agent egress IPs)](https://claude.com/.well-known/sail) read 2026-11-05 - [Anthropic bots.json IP ranges](https://claude.com/crawling/bots.json) read 2026-11-05 - [claude.com Web Bot Auth key directory path (GET probe)](https://claude.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Apify: crawler operator profile | Agent Ready Crawlers" description: "Apify: crawler operator profile covering Website Content Crawler and Apify platform and Store. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/apify site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Apify: crawler operator profile | Agent Ready Crawlers > Apify: crawler operator profile covering Website Content Crawler and Apify platform and Store. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/apify Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Apify In short Apify runs a scraping platform where customers' Actors do the crawling. A run signs its traffic with Web Bot Auth only when its input asks for that, and an API caller who keeps the defaults also skips robots.txt. The signing key sits on api.apify.com rather than the main site, and a free tier covers light use. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Based in Czech Republic Homepage [apify.com](https://apify.com) In one line Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. Last read 2026-11-05 ## What Apify does for this layer ### From the vendor's own pages Apify's Website Content Crawler has an experimental signHttpRequests input that signs all HTTP requests with the crawler's Web Bot Auth private key, which the input schema says is needed to use the crawler as a Cloudflare Signed Agent. ([apify.com](https://apify.com/apify/website-content-crawler/input-schema)) Website Content Crawler's respectRobotsTxtFile option consults robots.txt before each page, but the input schema says the crawler "does not use any specific user agent identifier" and does not yet support crawl-delay. ([apify.com](https://apify.com/apify/website-content-crawler/input-schema)) Apify's blog says the Apify Console prefills respectRobotsTxtFile while the API does not, and that the measured run in the post used the API default, which is off. ([blog.apify.com](https://blog.apify.com/ai-agent-infrastructure/)) Website Content Crawler also offers a useLlmsTxt input that crawls /llms.txt and Markdown files. ([apify.com](https://apify.com/apify/website-content-crawler/input-schema)) Apify's blog says that since June 2026 an agent can pay for Actor calls through x402 by buying a prepaid USDC token with a wallet, limited to Actors with pay-per-event pricing. ([blog.apify.com](https://blog.apify.com/ai-agent-infrastructure/)) Apify's pricing page lists Free at USD 0 with USD 5 of monthly usage at USD 0.2 per compute unit, Starter at USD 19, Scale at USD 199 (USD 0.16 per compute unit) and Business at USD 999 a month (USD 0.13 per compute unit). ([apify.com](https://apify.com/pricing)) ### Claims the vendor makes about itself Apify says it was launched in 2015 from the Y Combinator Fellowship in Mountain View and moved to the Czech Republic in 2016; its About page counts 110,737 ready-made Actors in Apify Store. ([apify.com](https://apify.com/about)) ### What our crawl saw Apify's key directory at api.apify.com/.well-known/http-message-signatures-directory returned HTTP 200 with content type application/http-message-signatures-directory+json, one Ed25519 key and a "purpose" value of "rag" on 2026-11-05. ([api.apify.com](https://api.apify.com/.well-known/http-message-signatures-directory)) The same path on apify.com returned HTTP 404 on 2026-11-05, so Apify's signing keys live on the api.apify.com host only. ([apify.com](https://apify.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Website Content Crawler | Actor that crawls sites into text or Markdown for RAG; optional robots.txt adherence and Web Bot Auth signing. | platform usage Per compute unit on Free and Starter; lower per-unit rates on Scale and Business. | USD 0.2 | 2026-11-05 | | Apify platform and Store | Runs Actors (scrapers and agents) by API, MCP or schedule. | subscription plus usage Free USD 0 with USD 5 usage; Starter USD 19, Scale USD 199, Business USD 999 a month. | USD 0 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | respectRobotsTxtFile is an option, off by API default, with no own token and no crawl-delay support. (apify.com) | | Web Bot Auth | Partial | Experimental signHttpRequests input on Website Content Crawler; key directory served on api.apify.com. (apify.com) | | Published IP list | No statement | No egress IP list for Actor traffic found (2026-11-05). | | Cloudflare verified bots | Partial | The input schema says signing is required to run as a Cloudflare Signed Agent. (apify.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key, purpose rag | GET https://api.apify.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404 | GET https://apify.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Apify does not do - Apify's Website Content Crawler names no robots.txt token of its own, so a site can address it only through the \* group (input schema read 2026-11-05). - Apify does not sign requests unless the run turns signHttpRequests on, and robots.txt is ignored when the API caller leaves respectRobotsTxtFile at its default. ## Reading notes The Crawler Index row for the [Website Content Crawler](https://agentreadycrawlers.com/crawlers/apify/) recorded a 404 for the key directory at apify.com on 4 November 2026; the directory is served from api.apify.com, as the checks above show. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Apify Website Content Crawler](https://agentreadycrawlers.com/crawlers/apify/) no token ## Sources - [apify.com](https://apify.com) read 2026-11-05 - [Input · Website Content Crawler · Apify](https://apify.com/apify/website-content-crawler/input-schema) read 2026-11-05 - [Apify key directory (api.apify.com)](https://api.apify.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [Apify key directory probe (apify.com)](https://apify.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [AI agent infrastructure stack: from web data to retrieval | Apify Blog](https://blog.apify.com/ai-agent-infrastructure/) read 2026-11-05 - [Apify pricing - flexible plan + pay as you go · Apify](https://apify.com/pricing) read 2026-11-05 - [About · Apify](https://apify.com/about) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Apple: crawler operator profile | Agent Ready Crawlers" description: "Apple: crawler operator profile covering Applebot, Applebot-Extended and iTMS. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/apple site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Apple: crawler operator profile | Agent Ready Crawlers > Apple: crawler operator profile covering Applebot, Applebot-Extended and iTMS. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/apple Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Apple In short Applebot gathers pages for Spotlight, Siri and Safari search, and the same crawl can be used when Apple trains its foundation models. Publishers who want out of training aim a Disallow at the Applebot-Extended token, which leaves ordinary search crawling alone. Genuine visits resolve through reverse DNS to applebot.apple.com or fall inside 24 published CIDR prefixes. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [apple.com](https://www.apple.com) In one line Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file. Last read 2026-11-05 ## What Apple does for this layer ### From the vendor's own pages Apple says Applebot data powers search in Spotlight, Siri and Safari, and may also help train Apple foundation models behind Apple Intelligence, Services and Developer Tools. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple's training opt-out is a robots.txt Disallow for Applebot-Extended, which leaves normal Applebot crawling for search in place. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple says publishers can keep content out of AI-generated broad world knowledge answers in Siri and Search by applying the nosnippet meta tag to that content. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple says Applebot traffic resolves by reverse DNS into the applebot.apple.com domain and can also be matched against the CIDR prefixes in applebot.json. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple says that when robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple states Applebot does not follow crawl-delay and instead slows its crawl rate automatically when a site slows down or returns errors. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple's iTMS user agent, which crawls URLs tied to registered Apple Podcasts content from applebot.apple.com hosts, does not follow robots.txt. ([support.apple.com](https://support.apple.com/en-us/119829)) Apple warns that Applebot may render pages in a browser, so blocking JavaScript, CSS or XHR resources in robots.txt can stop it rendering content properly. ([support.apple.com](https://support.apple.com/en-us/119829)) ### What our crawl saw Apple's applebot.json listed 24 IPv4 prefixes with a creationTime of 2026-09-15 when fetched on 2026-11-05. ([search.developer.apple.com](https://search.developer.apple.com/applebot.json), dated 2026-09-15) www.apple.com returned HTTP 404 for the Web Bot Auth key directory path on 2026-11-05. ([apple.com](https://www.apple.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Applebot | Crawler and renderer for Spotlight, Siri and Safari search, whose data may also train Apple foundation models. | No price recorded | | Applebot-Extended | Robots.txt token whose Disallow opts content out of Apple's generative model training. | No price recorded | | iTMS | Special crawler for URLs registered with Apple Podcasts; does not follow robots.txt. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Applebot respects standard robots.txt directives targeted at it in general search crawls; iTMS is the exception. (support.apple.com) | | Web Bot Auth | No statement | Not on the Applebot page; key directory path 404 on www.apple.com (observed 2026-11-05). | | Content Signals | No statement | Not mentioned on the Applebot page (read 2026-11-05). | | Published IP list | Supports | applebot.json lists the CIDR prefixes. (support.apple.com) | | Reverse DNS check | Supports | Hostnames in *.applebot.apple.com with a forward lookup check. (support.apple.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | applebot.json | 24 IPv4 prefixes, creationTime 2026-09-15 | GET and count prefixes | 2026-11-05 | ## What Apple does not do - Disallowing Applebot-Extended does not stop Applebot from crawling; Apple says the content stays discoverable in Spotlight, Siri and Safari. - Applebot ignores crawl-delay, so a site that needs a slower crawl cannot ask for one in robots.txt. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Apple in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/apple/) 2 rows - [Applebot](https://agentreadycrawlers.com/crawlers/applebot/) token Applebot - [Applebot-Extended (control token)](https://agentreadycrawlers.com/crawlers/applebot-extended/) token Applebot-Extended ## Sources - [apple.com](https://www.apple.com) read 2026-11-05 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-05 - [Applebot IP CIDRs (applebot.json)](https://search.developer.apple.com/applebot.json) read 2026-11-05 - [www.apple.com Web Bot Auth key directory path (GET probe)](https://www.apple.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Arcjet: bot management profile | Agent Ready Crawlers" description: "Arcjet: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 11 of 15." canonical: https://agentreadycrawlers.com/vendors/arcjet site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Arcjet: bot management profile | Agent Ready Crawlers > Arcjet: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 11 of 15. Source: https://agentreadycrawlers.com/vendors/arcjet Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 11 (tied) of 15 # Arcjet In short Arcjet is a developer SDK that runs bot checks inside application code, which earns it 3 of 10 points and a shared 11th place. Verification, per-purpose policy and reporting grade partial; charging and commerce agents are not published, and paid plans begin at USD 25 a month. Developer security SDK that runs bot detection, rate limits and WAF rules inside application code; it verifies known bots by IP and reverse DNS and in late 2026 added runtime security for AI agents. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Arcjet](https://agentreadycrawlers.com/vendors/arcjet/alternatives/) · [arcjet.com](https://arcjet.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [arcjet.com](https://arcjet.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Partial | IP data and reverse DNS for known bots; Web Bot Auth not mentioned in docs (checked 2026-11-04) | Bot protection concepts page. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | allow or deny by bot name or category (e.g. CATEGORY:AI) per route | Reference page allow/deny lists. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | No charging feature in bot protection docs. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned in docs. Source, read 2026-11-04. | | Cost | Not graded | published: from USD 25/month plus USD 5 per 1M web requests | Pricing page. Source, read 2026-11-04. | | Reporting | Partial | request decision logs with 1 to 24 hours retention on self-serve plans | Pricing page log retention. Source, read 2026-11-04. | | Where it runs | Not graded | app: SDK calls in application code | Concepts page: configured in your application code. Source, read 2026-11-04. | ## Cost Individual USD 25/month, Startup USD 299/month, Enterprise custom; usage USD 5 per 1M web requests and USD 50 per 1M agent requests; free plan 10,000 requests/month (hard stop) after a 15-day trial. [Source](https://arcjet.com/pricing), read 2026-11-04. ## What Arcjet does not do - No Web Bot Auth or cryptographic agent verification in bot protection docs (checked 2026-11-04). - Does not run at the CDN edge; requests reach the app before Arcjet decides (concepts page). - No AI crawler monetization or pay-per-crawl (docs read 2026-11-04). - Arcjet does not decide at a CDN; the request reaches the application before the SDK rule runs. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Arcjet bot protection detects more than 600 bots grouped into 23 categories such as CATEGORY:AI. [Source](https://docs.arcjet.com/bot-protection/concepts), read 2026-11-04. - Primary source Arcjet checks that a bot is who it claims to be using IP data and reverse DNS lookups. [Source](https://docs.arcjet.com/bot-protection/concepts), read 2026-11-04. - Primary source Bot rules are configured inside the application code through the Arcjet SDK rather than at a CDN. [Source](https://docs.arcjet.com/bot-protection/concepts), read 2026-11-04. - Primary source A rule takes exactly one of an allow or deny list; an empty allow list blocks every detected bot. [Source](https://docs.arcjet.com/bot-protection/reference), read 2026-11-04. - Primary source Allow and deny lists draw on Arcjet's published bot list and bot categories, and docs advise checking verification status after an allow decision. [Source](https://docs.arcjet.com/bot-protection/reference), read 2026-11-04. - Primary source Paid plans cost USD 25/month (Individual) or USD 299/month (Startup), plus USD 5 per 1M web requests and USD 50 per 1M agent requests. [Source](https://arcjet.com/pricing), read 2026-11-04. - Primary source After a 15-day trial the free plan allows 10,000 requests a month as a hard stop rather than metered overage. [Source](https://arcjet.com/pricing), read 2026-11-04. - Primary source Log retention is 1 hour on Individual and 24 hours on Startup. [Source](https://arcjet.com/pricing), read 2026-11-04. - Primary source On 17 Sep 2026 Arcjet launched Agent Runtime Security, which observes agent activity via OpenTelemetry, enforces policies on tool calls, APIs and databases, and records an audit trail. [Source](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit), read 2026-11-04, dated 2026-09-17. - Primary source On 8 Oct 2026 Arcjet added security policies for coding agents including Claude Code, Codex, Cursor, Muse Code and Copilot. [Source](https://arcjet.com/blog/introducing-arcjet-for-coding-agents), read 2026-11-04, dated 2026-10-08. ## Dated news - 2026-09-17**Arcjet launches Agent Runtime Security**. Arcjet introduced a product for teams building AI agents: it observes agent activity through OpenTelemetry without code changes, enforces deterministic policies on tool calls, APIs and databases before and after they run, and records inputs and decisions for audit. [Source](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit). - 2026-10-08**Arcjet for coding agents**. Arcjet extended its policies to coding agents, covering Claude Code, Codex, Cursor, Muse Code and Copilot, so teams can control what those agents may do in their environments. [Source](https://arcjet.com/blog/introducing-arcjet-for-coding-agents). - 2026-10-14**Arcjet details agent behavior drift analysis**. Arcjet described how it monitors AI coding agent sessions in real time to spot behavior drift, using Cloudflare Clef. [Source](https://arcjet.com/blog/how-we-built-ai-agent-behavior-drift-analysis-with-cloudflare-clef). ## Compared with - [Arcjet vs Vercel](https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid/): Arcjet verifies known bots by IP and reverse DNS inside application code through its SDK; Vercel verifies by IP, reverse DNS and Web Bot Auth at its edge and splits AI crawlers from AI assistants in a managed ruleset since 2026-10-29. - [Arcjet vs Cloudflare](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare/): Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. ## What Arcjet does for this layer ### From the vendor's own pages Arcjet's detectBot rule accepts individual bot names such as GOOGLE\_CRAWLER and BING\_CRAWLER or category names, and runs in LIVE mode to block or DRY\_RUN mode to log only. ([docs.arcjet.com](https://docs.arcjet.com/bot-protection/identifying-bots)) Arcjet defines CATEGORY:AI as bots that scrape data for AI and LLM purposes, beside categories such as CATEGORY:SEARCH\_ENGINE, CATEGORY:PREVIEW and CATEGORY:ARCHIVE. ([docs.arcjet.com](https://docs.arcjet.com/bot-protection/identifying-bots)) Arcjet exposes the same bot identifiers in TypeScript, Python and Go; Python gets a BotCategory enum and Go gets BotCategory constants. ([docs.arcjet.com](https://docs.arcjet.com/bot-protection/identifying-bots)) Arcjet's isVerifiedBot helper (is\_verified\_bot in Python) lets application code allow verified search engine bots while ignoring other signals. ([docs.arcjet.com](https://docs.arcjet.com/bot-protection/reference)) Arcjet's product menu lists Shield WAF, bot protection, rate limiting, signup form protection, email validation and sensitive information detection. ([arcjet.com](https://arcjet.com/pricing)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot protection (detectBot) | In-app bot detection with allow or deny lists of named bots and 23 categories. | monthly plan plus usage Individual USD 25 a month, Startup USD 299 a month; usage fees apply. | USD 25 | 2026-11-05 | | Arcjet Guards | Runtime security policies for agent frameworks and coding agents. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | Bot protection docs read 2026-11-05 mention IP data and reverse DNS only. | | Published IP list | Supports | Arcjet verifies detected bots by checking IP data. (docs.arcjet.com) | | Reverse DNS check | Supports | Arcjet performs reverse DNS lookups to catch spoofed bots. (docs.arcjet.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Arcjet](https://agentreadycrawlers.com/vendors/arcjet/alternatives/) graded on the rubric ## Sources - [arcjet.com: arcjet.com](https://arcjet.com) read 2026-11-04 - [arcjet.com: pricing](https://arcjet.com/pricing) read 2026-11-04 - [docs.arcjet.com: concepts](https://docs.arcjet.com/bot-protection/concepts) read 2026-11-04 - [docs.arcjet.com: reference](https://docs.arcjet.com/bot-protection/reference) read 2026-11-04 - [arcjet.com: introducing agent runtime security observe enforce audit](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit) read 2026-11-04 - [arcjet.com: introducing arcjet for coding agents](https://arcjet.com/blog/introducing-arcjet-for-coding-agents) read 2026-11-04 - [arcjet.com: how we built ai agent behavior drift analysis with cloudflare clef](https://arcjet.com/blog/how-we-built-ai-agent-behavior-drift-analysis-with-cloudflare-clef) read 2026-11-04 - [Identifying bots | Arcjet Docs](https://docs.arcjet.com/bot-protection/identifying-bots) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Arcjet | Agent Ready Crawlers" description: "Alternatives to Arcjet: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/arcjet/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Arcjet | Agent Ready Crawlers > Alternatives to Arcjet: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/arcjet/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Arcjet In short Arcjet and two other vendors, Auth0 for AI Agents and Imperva, share 11th place with 3 of 10 points. Head-to-head pages set it against Vercel and Cloudflare. Arcjet differs from the other two in deciding inside application code, and its pages describe no cryptographic way to verify an agent. Paid plans begin at USD 25 a month. ## Head-to-head, with the sourced difference - [**Arcjet vs Vercel**](https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid/). Arcjet verifies known bots by IP and reverse DNS inside application code through its SDK; Vercel verifies by IP, reverse DNS and Web Bot Auth at its edge and splits AI crawlers from AI assistants in a managed ruleset since 2026-10-29. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. - [**Arcjet vs Cloudflare**](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare/). Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. [Source](https://arcjet.com/pricing), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Arcjet's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Arcjet on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, charging, commerce agents, reporting | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy, charging, reporting | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, charging, commerce agents, reporting | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, charging, commerce agents, reporting | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy, charging, reporting | nothing | No public price; sales-led | | Skyfire | 6 | verifies agents, charging, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | charging, reporting | nothing | Free for publishers to start | | Vercel | 6 | verifies agents, per-purpose policy, reporting | nothing | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents, reporting | nothing | No public price; sales-led | | Fastly | 4 | charging, commerce agents | reporting | No public price; sales-led | | Auth0 for AI Agents | 3 | commerce agents | verifies agents | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | reporting | per-purpose policy | No public price; sales-led | | Stytch IsAgent | 2 | nothing | per-purpose policy | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Arcjet is not the fit - No Web Bot Auth or cryptographic agent verification in bot protection docs (checked 2026-11-04). - Does not run at the CDN edge; requests reach the app before Arcjet decides (concepts page). - No AI crawler monetization or pay-per-crawl (docs read 2026-11-04). ## Sources - [arcjet.com: pricing](https://arcjet.com/pricing) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Auth0 for AI Agents: bot management profile | Agent Ready Crawlers" description: "Auth0 for AI Agents: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources." canonical: https://agentreadycrawlers.com/vendors/auth0-ai-agents site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Auth0 for AI Agents: bot management profile | Agent Ready Crawlers > Auth0 for AI Agents: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Source: https://agentreadycrawlers.com/vendors/auth0-ai-agents Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 11 (tied) of 15 # Auth0 for AI Agents In short Auth0 for AI Agents secures agents a company builds, not the traffic reaching a website, which explains 3 of 10 points and a shared 11th place. Its cost is half the base plan added on top, with the smallest consumer tier of 500 users coming to USD 18 monthly. Per-purpose policy, commerce agents and reporting grade partial. Identity side of agent traffic: Auth0 gives agents that a company builds their own identities, scoped short-lived tokens, a Token Vault for third-party APIs and user approval for sensitive actions; it does not police inbound bots. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Auth0 for AI Agents](https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives/) · [auth0.com](https://auth0.com/ai) Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Operator Okta Homepage [auth0.com](https://auth0.com/ai) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Not its job | issues identities and scoped tokens to the customer's own agents; does not verify inbound third-party bots | Auth0 for AI Agents page. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | fine-grained permissions and authorization for RAG scoped to each agent task | Docs concepts and product page. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | No charging feature on product page. Source, read 2026-11-04. | | Lets signed commerce agents through | Partial | product page mentions agentic protocols for commerce and procurement; no named TAP or Agent Pay support found | Agent experiences section of product page. Source, read 2026-11-04. | | Cost | Not graded | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | Pricing page add-on tables. Source, read 2026-11-05. | | Reporting | Partial | audit trail for agent actions claimed | Product page agent identity section. Source, read 2026-11-04. | | Where it runs | Not graded | app: SDKs and tenant APIs inside the agent and its backend | GA post and docs. Source, read 2026-11-04. | ## Cost AI Agents add-on at 50 percent of the base plan price (USD 18 a month on B2C Essentials with 500 users, USD 75 on B2B Essentials); free plan available, paid tiers by monthly active users (checked 2026-11-05) [Source](https://auth0.com/pricing), read 2026-11-05. ## What Auth0 for AI Agents does not do - Does not detect, verify or block third-party crawlers or agents hitting a website; it secures agents the customer builds (product page). - Agent Gateway for MCP and Agent as Principal are listed as shipping next, not available (product page read 2026-11-04). - Auth0 does not classify, verify or block third-party crawlers and agents arriving at a website; its Bot Detection guards login and signup forms. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Auth0 for AI Agents became generally available on 19 Nov 2025. [Source](https://auth0.com/blog/auth0-for-ai-agents-generally-available/), read 2026-11-04, dated 2025-11-19. - Primary source The GA release lets agents identify the user, reach the user's data, connect to third-party apps on the user's behalf, ask the user to approve critical actions, and enforce fine-grained data access. [Source](https://auth0.com/blog/auth0-for-ai-agents-generally-available/), read 2026-11-04, dated 2025-11-19. - Primary source Auth0's AI docs cover four concepts: user authentication, delegated authorization, asynchronous authorization and authorization for RAG. [Source](https://auth0.com/ai/docs/intro/overview), read 2026-11-04. - Vendor claim Auth0 says each agent can get a unique digital identity tied to a verified user, keeping an audit trail when agents act on their own. [Source](https://auth0.com/ai), read 2026-11-04. - Vendor claim Auth0 says agent access uses fine-grained permissions and short-lived tokens, with sensitive tasks gated behind explicit user consent. [Source](https://auth0.com/ai), read 2026-11-04. - Primary source Auth0 lists Agent as Principal, Cross App Access (XAA) and an Agent Gateway for MCP servers as shipping next. [Source](https://auth0.com/ai), read 2026-11-04. - Primary source Auth0's pricing page lists Auth0 for AI Agents as an add-on that adds 50% to the base price: USD 18 a month on B2C Essentials with 500 monthly active users and USD 75 a month on B2B Essentials. [Source](https://auth0.com/pricing), read 2026-11-05. - Vendor claim Okta's 2026 Customer Identity Trends Report, cited by Auth0, says 67% of retailers are introducing AI-powered or agentic systems into customer-facing products. [Source](https://auth0.com/blog/who-will-consumers-trust-to-act-shop-pay-on-their-behalf/), read 2026-11-04, dated 2026-11-03. - Vendor claim The same report found 74% of 5,250 surveyed consumers would abandon a task if asked to create an account before seeing key information. [Source](https://auth0.com/blog/who-will-consumers-trust-to-act-shop-pay-on-their-behalf/), read 2026-11-04, dated 2026-11-03. - Vendor claim Okta announced on 2 Nov 2026 that it was named a Leader in the 2026 Gartner Magic Quadrant for Customer Identity and Access Management. [Source](https://www.okta.com/newsroom/press-releases/okta-named-a-leader-in-2026-gartner-magic-quadrant/), read 2026-11-04, dated 2026-11-02. ## Dated news - 2026-11-02**Okta named a Leader in 2026 Gartner Magic Quadrant for CIAM**. Okta, which owns Auth0, announced it was named a Leader in the 2026 Gartner Magic Quadrant for Customer Identity and Access Management. Auth0's blog tied the result to its customer identity work, including AI agent features. [Source](https://www.okta.com/newsroom/press-releases/okta-named-a-leader-in-2026-gartner-magic-quadrant/). - 2026-11-03**Auth0 publishes consumer trust data on agentic commerce**. Auth0 published findings from Okta's 2026 Customer Identity Trends Report: 67% of retailers are adding AI or agentic systems to customer-facing products, and most consumers will delegate actions to AI only when actions need prior approval and can be undone. [Source](https://auth0.com/blog/who-will-consumers-trust-to-act-shop-pay-on-their-behalf/). ## Compared with - [Stytch IsAgent vs Auth0 for AI Agents](https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents/): Stytch IsAgent flags inbound agent traffic on a site; Auth0 for AI Agents gives identities and scoped tokens to agents a company builds, and does not police inbound bots. - [Auth0 for AI Agents vs Skyfire](https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire/): Auth0 issues identities and short-lived tokens to agents a company builds for its own users; Skyfire issues KYA tokens that any third-party site can verify. ## What Auth0 for AI Agents does for this layer ### From the vendor's own pages Auth0's Free plan costs USD 0 a month for up to 25,000 monthly active users and lists "Secure Agentic AI workflows" among its features. ([auth0.com](https://auth0.com/pricing)) Token Vault stores the access and refresh tokens of a user's connected accounts, such as Google, Microsoft, Box, Slack and GitHub, and returns a provider token to an agent through a token exchange so the agent calls the API without the user re-authorising. ([auth0.com](https://auth0.com/docs/secure/call-apis-on-users-behalf/token-vault)) Auth0's Cross App Access implements the Identity Assertion Authorization Grant, an in-progress OAuth extension, and is the protocol behind MCP's Enterprise-Managed Authorization extension. ([auth0.com](https://auth0.com/docs/ai-agents-mcp/cross-app-access)) With Cross App Access an IT admin approves agent-to-app connections centrally, for example in the Okta Admin Console, instead of each employee answering an OAuth consent prompt. ([auth0.com](https://auth0.com/docs/ai-agents-mcp/cross-app-access)) Auth0 Bot Detection scores logins and signups against "60+ risk signals" and machine-learning models, and Supplemental Signals can add Akamai threat intelligence to Attack Protection decisions. ([auth0.com](https://auth0.com/platform/login-security.md)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Auth0 for AI Agents | Add-on for agent identity bound to a verified user, scoped short-lived tokens, async user approval and authorization for RAG. | add-on, 50% of the base plan price B2C Essentials at 500 MAUs, monthly; B2B from USD 75 | USD 18 | 2026-11-05 | | Token Vault | Stores third-party OAuth tokens per connected account and exchanges them for agents calling external APIs. | No price recorded | | Cross App Access (XAA) | Enterprise IdP issues identity assertions so agents reach other apps and MCP servers under admin policy. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | No Web Bot Auth signing or verification on the AI product page, pricing page or login security page (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | The AI page mentions "agentic protocols for commerce and procurement" without naming Trusted Agent Protocol (checked 2026-11-05). | | Mastercard Agent Pay | No statement | No Agent Pay mention on the pages read (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Auth0 for AI Agents (Okta)](https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives/) graded on the rubric ## Sources - [auth0.com: ai](https://auth0.com/ai) read 2026-11-04 - [auth0.com: pricing](https://auth0.com/pricing) read 2026-11-04 - [auth0.com: overview](https://auth0.com/ai/docs/intro/overview) read 2026-11-04 - [auth0.com: auth0 for ai agents generally available](https://auth0.com/blog/auth0-for-ai-agents-generally-available/) read 2026-11-04 - [auth0.com: who will consumers trust to act shop pay on their behalf](https://auth0.com/blog/who-will-consumers-trust-to-act-shop-pay-on-their-behalf/) read 2026-11-04 - [okta.com: okta named a leader in 2026 gartner magic quadrant](https://www.okta.com/newsroom/press-releases/okta-named-a-leader-in-2026-gartner-magic-quadrant/) read 2026-11-04 - [Auth0 for AI Agents (markdown product page)](https://auth0.com/ai.md) read 2026-11-05 - [Token Vault, Auth0 Docs](https://auth0.com/docs/secure/call-apis-on-users-behalf/token-vault) read 2026-11-05 - [Cross App Access (XAA), Auth0 Docs](https://auth0.com/docs/ai-agents-mcp/cross-app-access) read 2026-11-05 - [Auth0 login security (markdown)](https://auth0.com/platform/login-security.md) read 2026-11-05 - [Auth0 llms.txt](https://auth0.com/llms.txt) read 2026-11-05 - [Auth0: Ship AI agents, not identity work](https://auth0.com/solutions/ship-ai-agents.md) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Auth0 for AI Agents | Agent Ready Crawlers" description: "Alternatives to Auth0 for AI Agents: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head." canonical: https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Auth0 for AI Agents | Agent Ready Crawlers > Alternatives to Auth0 for AI Agents: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Source: https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Auth0 for AI Agents In short Auth0 for AI Agents shares 11th place on 3 of 10 points with Arcjet and Imperva, and two head-to-head pages compare it with Stytch IsAgent and Skyfire. The comparison is lopsided by design: Auth0 issues identities to agents a company builds, while those two work on agents that arrive at a site. Its add-on costs half the base plan. ## Head-to-head, with the sourced difference - [**Auth0 for AI Agents vs Stytch IsAgent**](https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents/). Stytch IsAgent flags inbound agent traffic on a site; Auth0 for AI Agents gives identities and scoped tokens to agents a company builds, and does not police inbound bots. [Source](https://auth0.com/ai), read 2026-11-04. - [**Auth0 for AI Agents vs Skyfire**](https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire/). Auth0 issues identities and short-lived tokens to agents a company builds for its own users; Skyfire issues KYA tokens that any third-party site can verify. [Source](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Auth0 for AI Agents's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Auth0 for AI Agents on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, charging, commerce agents, reporting | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy, charging, reporting | commerce agents | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, charging, reporting | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, charging, reporting | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy, charging, reporting | commerce agents | No public price; sales-led | | Skyfire | 6 | verifies agents, charging, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | verifies agents, charging, reporting | commerce agents | Free for publishers to start | | Vercel | 6 | verifies agents, per-purpose policy, reporting | commerce agents | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents, reporting | commerce agents | No public price; sales-led | | Fastly | 4 | verifies agents, charging | reporting | No public price; sales-led | | Arcjet | 3 | verifies agents | commerce agents | From USD 25/month plus USD 5 per 1M web requests | | Imperva | 3 | verifies agents, reporting | per-purpose policy, commerce agents | No public price; sales-led | | Stytch IsAgent | 2 | verifies agents | per-purpose policy, commerce agents | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | commerce agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Auth0 for AI Agents is not the fit - Does not detect, verify or block third-party crawlers or agents hitting a website; it secures agents the customer builds (product page). - Agent Gateway for MCP and Agent as Principal are listed as shipping next, not available (product page read 2026-11-04). ## Sources - [auth0.com: pricing](https://auth0.com/pricing) read 2026-11-04 - [auth0.com: ai](https://auth0.com/ai) read 2026-11-04 - [docs.skyfire.xyz: introduction to skyfire.md](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "AWS WAF Bot Control: bot management profile | Agent Ready Crawlers" description: "AWS WAF Bot Control: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources." canonical: https://agentreadycrawlers.com/vendors/aws-waf-bot-control site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # AWS WAF Bot Control: bot management profile | Agent Ready Crawlers > AWS WAF Bot Control: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Source: https://agentreadycrawlers.com/vendors/aws-waf-bot-control Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 1 (tied) of 15 # AWS WAF Bot Control In short AWS WAF Bot Control is Amazon's bot filter inside its web application firewall: it grades yes on agent verification, per-purpose policy, charging, through an x402 response, and reporting, for 8 of 10 points, with commerce agents not published. Pricing combines a flat monthly charge for each web ACL with a fee per million requests once a free allowance is used. Managed rule group in AWS WAF that labels and blocks bots, verifies AI agents with Web Bot Auth, and since June 2026 can charge AI bots per request with HTTP 402 on CloudFront. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/) · [aws.amazon.com](https://aws.amazon.com/waf/features/bot-control/) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Operator Amazon Homepage [aws.amazon.com](https://aws.amazon.com/waf/features/bot-control/) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | Web Bot Auth (rule set Version_4.0+), verified self-identifying bots | Bot Control docs section on Web Bot Authentication. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Yes | rules on bot category and name labels; prices can differ by bot identity and verification status | Monetize action uses Bot Control labels for pricing. Source, read 2026-11-04. | | Charging for access | Yes | HTTP 402 x402 monetization on CloudFront, Coinbase x402 settlement in stablecoins | What's new post 2026-06-15. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned in Bot Control docs or announcements read. Source, read 2026-11-04. | | Cost | Not graded | published: USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | WAF pricing page. Source, read 2026-11-04. | | Reporting | Yes | Bot Control console dashboard, AI activity dashboard, CloudWatch metrics and logs | Bot Control docs and Feb 2026 announcement. Source, read 2026-11-04. | | Where it runs | Not graded | edge or regional: CloudFront edge and regional resources such as load balancers and API Gateway | Web Bot Auth applies to CloudFront and Regional resources. Source, read 2026-11-04. | ## Cost Bot Control USD 10/month per web ACL. Common level: first 10M requests/month free, then USD 1.00 per million. Targeted level: first 1M requests/month free, then USD 10.00 per million. Standard AWS WAF web ACL and rule charges also apply; AI traffic monetization has no extra charge. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. ## What AWS WAF Bot Control does not do - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). - AWS WAF Bot Control cannot verify bots that reach it through a proxy or load balancer unless the customer adds a rule ahead of the rule group. - AgentCore Browser signing does not guarantee access; AWS says each site's bot policy decides. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Common Bot Control costs USD 10 per month per web ACL with the first 10 million requests a month free, then USD 1 per million requests. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - Primary source Targeted Bot Control costs USD 10 per month per web ACL with the first 1 million requests a month free, then USD 10 per million requests. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - Primary source AI traffic monetization carries no additional charge beyond standard AWS WAF fees. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - Primary source Bot Control supports Web Bot Auth for bots and AI agents on CloudFront and Regional resources in commercial Regions; it needs AWSManagedRulesBotControlRuleSet Version\_4.0 or later, explicitly selected. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. - Primary source AWS announced Web Bot Auth support in AWS WAF in November 2025. [Source](https://aws.amazon.com/about-aws/whats-new/2025/11/aws-waf-web-bot-auth-support), read 2026-11-04, dated 2025-11-21. - Primary source AI traffic monetization (15 Jun 2026) returns a machine-readable HTTP 402 under the x402 protocol with prices, accepted payment methods and license terms. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04, dated 2026-06-15. - Primary source Settlement runs through Coinbase's x402 Facilitator with stablecoin payouts; Stripe direct payouts and Machine Payments Protocol support are listed as coming soon. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04, dated 2026-06-15. - Primary source The Monetize action works only on web ACLs attached to CloudFront distributions and takes a PriceMultiplier from 1 to 100. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. - Primary source AWS warns that monetization classification is probabilistic and recommends Test mode before live charging. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. - Primary source Labels from Bot Control carry bot category and bot name and flow into CloudWatch metrics and web ACL logs. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. - Primary source Targeted protection adds browser interrogation, fingerprinting, behavior heuristics and optional machine learning for bots that do not self-identify. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. - Primary source AWS launched an AI activity dashboard for AWS WAF in February 2026 for visibility into AI bot and agent traffic. [Source](https://aws.amazon.com/about-aws/whats-new/2026/02/aws-waf-ai-activity-dashboard/), read 2026-11-04, dated 2026-02-24. ## Compared with - [Cloudflare vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control/): AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. - [AWS WAF Bot Control vs Akamai](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai/): AWS WAF publishes Bot Control prices (USD 10 a month per web ACL, USD 1 per million common requests after 10 million) and charges bots natively over x402; Akamai is sales-led and charges through partners. - [AWS WAF Bot Control vs DataDome](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome/): AWS WAF charges AI bots natively over x402 at no extra fee beyond WAF charges; DataDome charges through partners from its USD 8,670 Advanced plan. - [TollBit vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control/): TollBit sells publisher analytics and a paywall reached by redirect; AWS WAF returns HTTP 402 itself and settles through Coinbase. - [Vercel vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control/): Vercel splits AI crawlers from AI assistants on all plans for free; AWS WAF labels bots by category and name and can charge them, at USD 10 a month per web ACL plus request fees. ## What AWS WAF Bot Control does for this layer ### From the vendor's own pages AWS WAF Bot Control records each Web Bot Auth check as a label: web\_bot\_auth:verified, web\_bot\_auth:invalid, web\_bot\_auth:expired or web\_bot\_auth:unknown\_bot. ([docs.aws.amazon.com](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html)) When the web\_bot\_auth:verified label is present, Bot Control's category rules and its TGT\_TokenAbsent rule do not match, so verified signers pass. ([docs.aws.amazon.com](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html)) Bot Control adds a bot:account label holding an opaque hash of the AWS account that owns an AgentCore agent, so rules can allow, block or rate-limit agents per account without exposing account IDs. ([docs.aws.amazon.com](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html)) Bot Control treats bots labelled bot:user\_triggered:verified, which end users may invoke directly, like unverified bots. ([docs.aws.amazon.com](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html)) Bot Control decides whether a bot is verified from the request's origin IP and cannot be set to read a forwarded IP header instead. ([docs.aws.amazon.com](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html)) Amazon Bedrock AgentCore Browser's Web Bot Auth signing is in Preview and, when enabled, adds Signature, Signature-Agent and Signature-Input headers to every request. ([docs.aws.amazon.com](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html)) AWS says AgentCore registers with bot control vendors such as Cloudflare and reCAPTCHA and gives them public keys for verification. ([docs.aws.amazon.com](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html)) AWS's AgentCore guide warns that not every website recognises signed agent traffic and that domain owners may still block, monitor or rate-limit it. ([docs.aws.amazon.com](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | AWS WAF Bot Control | Managed rule group with Common and Targeted levels, bot labels, Web Bot Auth verification and the Monetize action. | per web ACL plus per request Common level includes 10M requests a month free; Targeted includes 1M; AI traffic monetization has no extra charge. | USD 10 | 2026-11-05 | | Amazon Bedrock AgentCore Browser | Hosted browser for agents that can sign requests with Web Bot Auth (Preview). | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Bot Control Version_4.0 or later verifies Web Bot Auth and labels the result. (docs.aws.amazon.com) | | Published IP list | Supports | Bot Control uses the request origin IP to verify bots. (docs.aws.amazon.com) | | Reverse DNS check | No statement | The rule group page read 2026-11-05 does not name reverse DNS. | | Visa Trusted Agent Protocol | No statement | No statement in the AWS WAF docs read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement in the AWS WAF docs read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Amazon Bedrock AgentCore Browser](https://agentreadycrawlers.com/crawlers/agentcore-browser/) no token - [Alternatives to Amazon Web Services (AWS WAF Bot Control, AgentCore Browser)](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives/) graded on the rubric ## Sources - [aws.amazon.com: bot control](https://aws.amazon.com/waf/features/bot-control/) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [aws.amazon.com: aws waf web bot auth support](https://aws.amazon.com/about-aws/whats-new/2025/11/aws-waf-web-bot-auth-support) read 2026-11-04 - [aws.amazon.com: aws waf ai activity dashboard](https://aws.amazon.com/about-aws/whats-new/2026/02/aws-waf-ai-activity-dashboard/) read 2026-11-04 - [AWS WAF Bot Control rule group - AWS WAF Developer Guide](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html) read 2026-11-05 - [Reducing CAPTCHAs with Web Bot Auth - Amazon Bedrock AgentCore](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to AWS WAF Bot Control | Agent Ready Crawlers" description: "Alternatives to AWS WAF Bot Control: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head." canonical: https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to AWS WAF Bot Control | Agent Ready Crawlers > Alternatives to AWS WAF Bot Control: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Source: https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to AWS WAF Bot Control In short AWS WAF Bot Control, tied for first at 8 of 10 points with Akamai, Cloudflare and DataDome, has five head-to-head pages: Cloudflare, Akamai, DataDome, TollBit and Vercel. It is the alternative that charges AI bots through HTTP 402 inside a cloud WAF, at USD 10 a month per web ACL plus request fees. ## Head-to-head, with the sourced difference - [**AWS WAF Bot Control vs Cloudflare**](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control/). AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. - [**AWS WAF Bot Control vs Akamai**](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai/). AWS WAF publishes Bot Control prices (USD 10 a month per web ACL, USD 1 per million common requests after 10 million) and charges bots natively over x402; Akamai is sales-led and charges through partners. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - [**AWS WAF Bot Control vs DataDome**](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome/). AWS WAF charges AI bots natively over x402 at no extra fee beyond WAF charges; DataDome charges through partners from its USD 8,670 Advanced plan. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - [**AWS WAF Bot Control vs TollBit**](https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control/). TollBit sells publisher analytics and a paywall reached by redirect; AWS WAF returns HTTP 402 itself and settles through Coinbase. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. - [**AWS WAF Bot Control vs Vercel**](https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control/). Vercel splits AI crawlers from AI assistants on all plans for free; AWS WAF labels bots by category and name and can charge them, at USD 10 a month per web ACL plus request fees. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with AWS WAF Bot Control's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than AWS WAF Bot Control on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | commerce agents | per-purpose policy, charging | No public price; sales-led | | Cloudflare | 8 | commerce agents | charging | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | commerce agents | per-purpose policy | From USD 3,830/month | | HUMAN Security | 7 | nothing | charging | No public price; sales-led | | Skyfire | 6 | commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | nothing | verifies agents, per-purpose policy | Free for publishers to start | | Vercel | 6 | nothing | charging | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | nothing | per-purpose policy, charging | No public price; sales-led | | Fastly | 4 | commerce agents | verifies agents, per-purpose policy, charging, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, per-purpose policy, charging, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | commerce agents | verifies agents, per-purpose policy, charging, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy, charging | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, charging, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, per-purpose policy, charging, reporting | Platform plans from USD 0; extension price not stated | ## Where AWS WAF Bot Control is not the fit - Monetize only works with CloudFront-associated web ACLs, not regional resources (Bot Control docs). - Stripe payouts and Machine Payments Protocol are not live yet; settlement is via Coinbase x402 in stablecoins (what's new, 2026-06-15). - Web Bot Auth is off unless the rule group version is pinned to Version\_4.0 or later (Bot Control docs). ## Sources - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Block: crawler operator profile | Agent Ready Crawlers" description: "Block: crawler operator profile covering goose. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/block site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Block: crawler operator profile | Agent Ready Crawlers > Block: crawler operator profile covering goose. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/block Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Block In short Block wrote goose, an open-source agent that runs on the user's own machine and that Cloudflare listed among its first signed agents in 2025. Its code now lives under the Agentic AI Foundation, part of the Linux Foundation, rather than at Block. Neither Block nor the goose docs offer a site owner any way to verify a visit. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [block.xyz](https://block.xyz) In one line Company that built goose, an open-source local AI agent Cloudflare classed as a signed agent in 2025; Block has since donated goose to the Linux Foundation's Agentic AI Foundation and publishes no key directory for it. Last read 2026-11-05 ## What Block does for this layer ### From the vendor's own pages Block has donated goose to the Agentic AI Foundation at the Linux Foundation, alongside Anthropic's MCP and OpenAI's AGENTS.md, and the goose blog post of 2026-04-07 says the repository moved from block/goose to aaif-goose/goose. ([goose-docs.ai](https://goose-docs.ai/blog/2026/04/07/goose-moves-to-aaif/), dated 2026-04-07) goose runs on the user's machine as a desktop app for macOS, Linux and Windows, a CLI and an API, and is built in Rust, per its documentation home page. ([goose-docs.ai](https://goose-docs.ai/)) goose works with 15+ model providers, including Anthropic, OpenAI, Google, Ollama, OpenRouter, Azure and Bedrock, and can use existing Claude, ChatGPT or Gemini subscriptions through ACP. ([goose-docs.ai](https://goose-docs.ai/)) The goose repository is Apache-2.0 licensed and showed 56.2k stars on 2026-11-05. ([github.com](https://github.com/aaif-goose/goose)) ### Reported by a third party Cloudflare listed goose from Block in its first cohort of signed agents on 2025-08-28, recognised through Web Bot Auth signatures. ([blog.cloudflare.com](https://blog.cloudflare.com/signed-agents/), dated 2025-08-28) In Cloudflare's signed-agents post, a Block staff software engineer says goose was designed to run locally and that Web Bot Auth lets third-party sites trust goose as it acts on users' behalf. ([blog.cloudflare.com](https://blog.cloudflare.com/signed-agents/), dated 2025-08-28) ### What our crawl saw The goose documentation index read on 2026-11-05 has no page on Web Bot Auth, request signing or a user agent for site owners. ([goose-docs.ai](https://goose-docs.ai/llms.txt)) A GET for block.xyz/.well-known/http-message-signatures-directory returned HTTP 404 on 2026-11-05. ([block.xyz](https://block.xyz/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | goose | Open-source general-purpose AI agent (desktop, CLI, API), now governed by the Agentic AI Foundation. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | No robots.txt statement in the goose docs index (2026-11-05). | | Web Bot Auth | Announced | Cloudflare's 2025-08-28 post names goose as a signed agent; no goose or Block page documents the signing. (blog.cloudflare.com) | | Cloudflare verified bots | Announced | Named in Cloudflare's first signed-agent cohort. (blog.cloudflare.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://block.xyz/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Block does not do - Block publishes no key directory, user agent string or verification page for goose (checked 2026-11-05), so its signed-agent status rests on Cloudflare's 2025 announcement. - Block no longer hosts goose's code; the repository now sits in the aaif-goose organisation under the Agentic AI Foundation. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Goose (open-source agent)](https://agentreadycrawlers.com/crawlers/goose/) no token ## Sources - [block.xyz](https://block.xyz) read 2026-11-05 - [The age of agents: cryptographically recognizing agent traffic | Cloudflare Blog](https://blog.cloudflare.com/signed-agents/) read 2026-11-05 - [goose has a new home - the Agentic AI Foundation (AAIF)](https://goose-docs.ai/blog/2026/04/07/goose-moves-to-aaif/) read 2026-11-05 - [goose | Your open source AI agent](https://goose-docs.ai/) read 2026-11-05 - [aaif-goose/goose on GitHub](https://github.com/aaif-goose/goose) read 2026-11-05 - [goose documentation index (llms.txt)](https://goose-docs.ai/llms.txt) read 2026-11-05 - [Block key directory probe](https://block.xyz/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Browser Use: crawler operator profile | Agent Ready Crawlers" description: "Browser Use: crawler operator profile covering Browser Infrastructure (cloud browsers), Browser Use Agents and browser-use (open source). Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/browser-use site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Browser Use: crawler operator profile | Agent Ready Crawlers > Browser Use: crawler operator profile covering Browser Infrastructure (cloud browsers), Browser Use Agents and browser-use (open source). Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/browser-use Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Browser Use In short Browser Use sells hosted cloud browsers that run a hardened Chromium with stealth fingerprints, residential proxies and CAPTCHA solving switched on by default, at USD 0.02 an hour plus proxy traffic. Nothing in its documentation names these browsers to a site, so an allow rule at a WAF has nothing to match. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [browser-use.com](https://browser-use.com) In one line Maker of the open-source browser-use agent library and a hosted cloud browser for agents that ships with stealth fingerprinting, residential proxies and CAPTCHA solving on by default, and no published identifier for site owners. Last read 2026-11-05 ## What Browser Use does for this layer ### From the vendor's own pages Browser Use sells Browser Use Agents, which complete web tasks, and Browser Infrastructure, a cloud browser that developers connect to with Playwright or Puppeteer over CDP. ([browser-use.com](https://browser-use.com/)) Browser Use's stealth docs say every cloud session runs a hardened Chromium fork with stealth enabled by default, randomising canvas, WebGL, font and navigator fingerprints per session "to appear as a real user". ([docs.browser-use.com](https://docs.browser-use.com/cloud/browser/stealth.md)) Browser Use enables residential proxies by default across 195+ countries, which its docs say makes sessions "appear as real users from the target geography". ([docs.browser-use.com](https://docs.browser-use.com/cloud/browser/stealth.md)) Browser Use's docs say its cloud browsers solve CAPTCHAs automatically and help with sites protected by Cloudflare, PerimeterX and other detection services, while access to every site is not guaranteed. ([docs.browser-use.com](https://docs.browser-use.com/cloud/browser/stealth.md)) Browser Use fixes the browser window size by default because, its docs say, "resizing makes a browser easier to detect". ([docs.browser-use.com](https://docs.browser-use.com/cloud/browser/stealth.md)) Browser Use prices cloud browsers at USD 0.02 an hour plus USD 5 per GB of residential proxy traffic, and web agents at model cost plus a 20 percent fee. ([browser-use.com](https://browser-use.com/pricing)) ### Claims the vendor makes about itself Browser Use markets its cloud browser as "The cloud browser that gets past bot walls" and says that across 200 visits to 100 bot-protected sites headless Chrome got through on 3.0 percent and Browser Use on 90.5 percent. ([browser-use.com](https://browser-use.com/)) Browser Use's homepage shows the open-source library under the MIT licence with 119,000 GitHub stars. ([browser-use.com](https://browser-use.com/)) ### What our crawl saw The Browser Use documentation index read on 2026-11-05 lists pages for stealth, proxies and CAPTCHA handling and none for a user agent token, robots.txt policy or Web Bot Auth signing. ([docs.browser-use.com](https://docs.browser-use.com/llms.txt)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Browser Infrastructure (cloud browsers) | Stealth Chromium sessions driven over CDP, with residential proxies and CAPTCHA solving. | usage-based Per browser-hour, plus USD 5 per GB of residential proxy traffic. | USD 0.02 | 2026-11-05 | | Browser Use Agents | Hosted agents that complete web tasks. | usage-based Plus browser time and traffic. | Model cost plus 20 percent | 2026-11-05 | | browser-use (open source) | MIT-licensed Python library for browser agents. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | No robots.txt page in the docs index (2026-11-05). | | Web Bot Auth | No statement | No signing feature in the docs index; key directory 404 at browser-use.com (2026-11-05). | | Published IP list | No statement | No egress IP list; sessions use residential proxies by default. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://browser-use.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Browser Use does not do - Browser Use publishes no user agent token, robots.txt policy or signing keys for its cloud browsers; its docs describe the aim of appearing as a real user instead (checked 2026-11-05). - A site cannot allow Browser Use as a named agent at its WAF, because the vendor publishes no identifier or key directory. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Browser Use (library and cloud)](https://agentreadycrawlers.com/crawlers/browser-use/) no token ## Sources - [browser-use.com](https://browser-use.com) read 2026-11-05 - [Browser Use Agents & Browser Infrastructure | Browser Use](https://browser-use.com/) read 2026-11-05 - [Stealth | Browser Use docs](https://docs.browser-use.com/cloud/browser/stealth.md) read 2026-11-05 - [Pricing - Browser Use](https://browser-use.com/pricing) read 2026-11-05 - [Browser Use documentation index (llms.txt)](https://docs.browser-use.com/llms.txt) read 2026-11-05 - [Browser Use key directory probe](https://browser-use.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Browserbase: crawler operator profile | Agent Ready Crawlers" description: "Browserbase: crawler operator profile covering Browserbase cloud browsers and Stagehand. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/browserbase site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Browserbase: crawler operator profile | Agent Ready Crawlers > Browserbase: crawler operator profile covering Browserbase cloud browsers and Stagehand. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/browserbase Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Browserbase In short Browserbase hosts headless browsers for agents and publishes the Stagehand SDK under the MIT licence. Web Bot Auth signing is a beta enabled per account on request, with two Ed25519 keys served from www.browserbase.com, so some sessions arrive signed and others do not. Whether an agent obeys a site's rules is left to the customer. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [browserbase.com](https://www.browserbase.com) In one line Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned. Last read 2026-11-05 ## What Browserbase does for this layer ### From the vendor's own pages Browserbase groups its access features under Browserbase Identity: Verified browsers, Web Bot Auth, CAPTCHA solving, residential or datacenter proxies, and human verification with AgentKit for x402 sessions. ([docs.browserbase.com](https://docs.browserbase.com/platform/identity/overview)) Browserbase's Verified sessions use a purpose-built Chromium with "real browser fingerprints" that its bot protection partners recognise; Verified is sold on the Scale plan and defaults to a Windows operating system. ([docs.browserbase.com](https://docs.browserbase.com/platform/identity/overview)) Browserbase offers Web Bot Auth in beta, enabled per account on request, and its identity docs say a valid signature does not grant access or sign the agent into a user account. ([docs.browserbase.com](https://docs.browserbase.com/platform/identity/overview)) Browserbase's Web Bot Auth page lists the Signature-Agent, Signature-Input and Signature headers and says that signing @authority with created and expires parameters limits where and when a signature works without preventing all replay. ([docs.browserbase.com](https://docs.browserbase.com/platform/identity/web-bot-auth.md)) Browserbase's guidance for site compliance tells customers to review a site's terms of service and to check robots.txt "when applicable" before an agent visits. ([docs.browserbase.com](https://docs.browserbase.com/platform/identity/overview)) Browserbase's Developer plan includes 100 browser hours then USD 0.12 per browser hour and 1 GB of proxies then USD 12 per GB; the Startup plan includes 500 hours then USD 0.10 and 5 GB then USD 10 per GB. ([browserbase.com](https://www.browserbase.com/pricing)) Browserbase's docs index lists AgentKit by Tools for Humanity, which lets an agent prove with a cryptographic proof-of-humanity that a real person backs it. ([docs.browserbase.com](https://docs.browserbase.com/llms.txt)) Stagehand, Browserbase's SDK for browser agents, is MIT-licensed and showed 26.0k GitHub stars on 2026-11-05. ([github.com](https://github.com/browserbase/stagehand)) Stagehand combines natural-language act, extract and observe steps with Playwright-style page calls, in TypeScript, Python and Go. ([docs.stagehand.dev](https://docs.stagehand.dev/v4/first-steps/introduction)) ### Reported by a third party Cloudflare named Browserbase in its first cohort of signed agents on 2025-08-28, alongside ChatGPT agent, goose from Block and Anchor Browser. ([blog.cloudflare.com](https://blog.cloudflare.com/signed-agents/), dated 2025-08-28) ### What our crawl saw Browserbase's key directory at www.browserbase.com/.well-known/http-message-signatures-directory returned HTTP 200 with content type application/http-message-signatures-directory+json and two Ed25519 keys on 2026-11-05. ([browserbase.com](https://www.browserbase.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Browserbase cloud browsers | Hosted headless browsers for agents, with Verified, proxies, CAPTCHA solving and optional Web Bot Auth. | subscription plus usage Page title lists Free, USD 20, USD 99 or custom; overage USD 0.12 or 0.10 per browser hour by plan. | USD 20 | 2026-11-05 | | Stagehand | MIT-licensed SDK for browser agents; identity comes from the browser it runs in. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | Advice to customers to check robots.txt when applicable; no crawl policy of its own. (docs.browserbase.com) | | Web Bot Auth | Supports | Web Bot Auth signing in beta per account; key directory served on www.browserbase.com. (docs.browserbase.com) | | Published IP list | No statement | No egress IP list; proxies are residential or datacenter (2026-11-05). | | Cloudflare verified bots | Supports | Listed by Cloudflare as a signed agent in the first cohort, 2025-08-28. (blog.cloudflare.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 2 Ed25519 keys | GET https://www.browserbase.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Browserbase does not do - Browserbase signs requests only for accounts in its Web Bot Auth beta, so a site cannot expect every Browserbase session to carry a signature. - Browserbase publishes no robots.txt token or crawl policy of its own; compliance is left to each customer's code. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Browserbase in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/browserbase/) 2 rows - [Browserbase cloud browsers](https://agentreadycrawlers.com/crawlers/browserbase/) no token - [Stagehand (open-source framework)](https://agentreadycrawlers.com/crawlers/stagehand/) no token ## Sources - [browserbase.com](https://www.browserbase.com) read 2026-11-05 - [Agent Identity - Browserbase Documentation](https://docs.browserbase.com/platform/identity/overview) read 2026-11-05 - [Web Bot Auth - Browserbase Documentation](https://docs.browserbase.com/platform/identity/web-bot-auth.md) read 2026-11-05 - [Browserbase key directory](https://www.browserbase.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [The age of agents: cryptographically recognizing agent traffic | Cloudflare Blog](https://blog.cloudflare.com/signed-agents/) read 2026-11-05 - [Browserbase Pricing: Free, $20, $99, or Custom | Browserbase](https://www.browserbase.com/pricing) read 2026-11-05 - [Browserbase documentation index (llms.txt)](https://docs.browserbase.com/llms.txt) read 2026-11-05 - [browserbase/stagehand on GitHub](https://github.com/browserbase/stagehand) read 2026-11-05 - [Introducing Stagehand - Stagehand](https://docs.stagehand.dev/v4/first-steps/introduction) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Bot management vendors: 23 vendor profiles | Agent Ready Crawlers" description: "Bot management vendors: 23 profiles with standards, dated prices, sources and what each does not do. 14 of 23 document support for Web Bot Auth." canonical: https://agentreadycrawlers.com/vendors/by-type/bot-management site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Bot management vendors: 23 vendor profiles | Agent Ready Crawlers > Bot management vendors: 23 profiles with standards, dated prices, sources and what each does not do. 14 of 23 document support for Web Bot Auth. Source: https://agentreadycrawlers.com/vendors/by-type/bot-management Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendors by type # Bot management vendors In short 23 bot management vendors are profiled, from CDNs such as Akamai and Cloudflare to detection specialists such as Kasada and Verisoul. 14 of 23 document support for Web Bot Auth, 7 for published IP lists and 4 for reverse DNS. The 23 bot management vendors run alphabetically from Akamai to Verisoul, and carry 181 sourced facts between them. The price models on record include sales-led; per web ACL plus per request; monthly plan plus usage; usage-based plans; quoted per deployment; and 13 more. 1 profile carries our own probes of key directories or IP files. Each profile lists what the vendor does not do next to what it does. - [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) - [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/) - [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) - [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) - [All 73](https://agentreadycrawlers.com/vendors/) | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Akamai | Bot management | CDN and security company whose Bot & Agent Control solutions judge AI agents at the edge, take identity from Visa's Trusted Agent Protocol and Know Your Agent tokens, and send AI traffic to TollBit or Skyfire for paid access. | 2 web-bot-auth, visa-tap | sales-led | Homepage HTTP 403 2026-11-05 | 2026-11-05 | | AWS WAF Bot Control | Bot management | Cloud provider whose WAF Bot Control labels and verifies bots, checks Web Bot Auth signatures and can charge AI bots with HTTP 402, and whose AgentCore Browser signs its own agent traffic with Web Bot Auth. | 2 web-bot-auth, ip-list | per web ACL plus per request | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Arcjet | Bot management | Developer security SDK for JavaScript, Python and Go that detects bots inside application code, lets developers allow or deny named bots or categories such as CATEGORY:AI, and adds runtime guards for agent frameworks. | 2 ip-list, reverse-dns | monthly plan plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Castle | Bot management | Bot and account abuse API whose risk responses include an agent object that names AI agents and browser automation, identified by a verified Web Bot Auth signature, a browser fingerprint or a user-agent string. | 1 web-bot-auth | usage-based plans | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Centinel Analytica | Bot management | Bot and AI crawler protection vendor whose edge integration verifies declared search and AI crawlers against operators' published IP ranges and detects scrapers that run real browsers behind residential proxies. | 1 ip-list | quoted per deployment | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Cequence Security | Bot management | Application and API protection vendor whose Bot Management includes Agent Trust for personal AI shopping agents, and whose AI Gateway governs enterprise agents' tool calls behind OAuth 2.1 authentication. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Cloudflare | Bot management | Connectivity cloud whose AI Crawl Control, Bot Management and verified bot program sort crawlers by purpose, verify them by IP list or Web Bot Auth signature, and let site owners charge crawlers per request in a private beta. | 5 rfc9309, web-bot-auth, content-signals, ip-list, cloudflare-verified-bots | included in every plan; site owner sets one per-request price; 1 more | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | DataDome | Bot management | Bot and fraud protection vendor whose Agentic Trust layer names each AI agent, rates how strong that identification is, applies per-agent and per-endpoint policies, and hands chosen agents to TollBit for paid access. | 3 web-bot-auth, ip-list, reverse-dns | monthly, by request volume | Homepage HTTP 403 2026-11-05 | 2026-11-05 | | F5 | Bot management | Application delivery and security company whose Distributed Cloud Bot Defense puts humans, trusted AI agents and malicious bots under one policy, accepts Web Bot Auth and Skyfire's Know Your Agent tokens, and runs on F5's platform, BIG-IP or public cloud. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Fastly | Bot management | Edge cloud vendor whose Bot Management adds edge detection (ContentGuard), TLS fingerprinting, client challenges and a verified bots signal, with separate AI crawler and AI fetcher signals for training bots and user-triggered fetchers. | 0 | contact sales | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Fingerprint | Bot management | Device intelligence vendor whose Bot Detection and Authorized AI Agent Detection signals separate good bots, bad bots and AI agents, and verify signed agents from providers such as OpenAI, AWS AgentCore and Browserbase. | 1 web-bot-auth | monthly plan plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Google Cloud Fraud Defense | Bot management | Google Cloud's bot and fraud service, the evolution of reCAPTCHA, which verifies AI agents by Web Bot Auth signature or by user agent matched to published IP ranges or reverse DNS, and reports verified and suspected agent sessions. | 3 web-bot-auth, ip-list, reverse-dns | per assessment | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | hCaptcha | Bot management | Bot detection and CAPTCHA vendor whose Enterprise Agent Controls identify AI agents, verify partner platforms that sign with Web Bot Auth, and let sites allow browsing while requiring verification before an agent reserves inventory. | 1 web-bot-auth | contact sales; monthly plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | HUMAN Security | Bot management | Bot and fraud defense vendor whose AgenticTrust module in Sightline names AI agents, grades them High, Medium or Low trust, verifies signed agents, and lets sites set what each agent may do from browsing to checkout. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Imperva | Bot management | Application security vendor owned by Thales whose Advanced Bot Protection sorts human, good bot and bad bot traffic across web, mobile and API; verified-agent controls so far ship in a controlled-availability Google Cloud offering. | 0 | sales-led, free trial | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Kasada | Bot management | Bot defense vendor founded in Sydney whose AI Agent Trust adds a verified agent directory with Web Bot Auth support and endpoint-level access rules, enforced at the edge before automated traffic reaches the application. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Known Agents | Bot management | Agent analytics and access-control service, formerly Dark Visitors, that identifies AI agents and crawlers from server-side data, keeps an automatic robots.txt current, and blocks bots that ignore it or spoof trusted identities. | 1 rfc9309 | monthly by events | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Link11 | Bot management | German security provider whose WAAP platform joins WAF, API security, Bot Management and Web DDoS protection, with a setting to monitor or block AI crawlers; the reblaze.com domain now redirects to Link11. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Netacea | Bot management | Server-side bot protection vendor that integrates at the CDN or load balancer without client code, classifies humans, good bots, bad bots and AI agents, and enforces access by intent rather than declared identity. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Netlify | Bot management | Web hosting platform that blocks chosen AI crawlers by user-agent string through an Edge Function extension and offers IP, geography and header-based traffic rules and rate limiting; it does not verify bot identity. | 0 | plan-based | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Radware | Bot management | Application security vendor whose Bot Manager and Agent Trust Management identify Web Bot Auth-compliant and non-compliant agents, capture an agent's declared intent in conversation, and restrict actions such as login or checkout. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Vercel | Bot management | Hosting platform whose firewall verifies known bots by IP, reverse DNS and Web Bot Auth, offers an AI Bots managed ruleset that treats AI crawlers and AI assistants separately, and sells BotID checks for high-value routes. | 3 web-bot-auth, ip-list, reverse-dns | included; per call | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Verisoul | Bot management | Fake account and fraud detection platform whose Agent & Bot Detection names AI agents by signature, self-declared user agent or inference, and whose public Agent Atlas documents how each agent shows up on a site. | 1 web-bot-auth | sales-led, 30-day trial | Homepage HTTP 200 2026-11-05 | 2026-11-05 | ## Sources - [The full vendor list (this site)](https://agentreadycrawlers.com/vendors/) read 2026-11-04 - [Akamai: akamai.com](https://www.akamai.com) read 2026-11-05 - [AWS WAF Bot Control: aws.amazon.com](https://aws.amazon.com/waf/features/bot-control/) read 2026-11-05 - [Arcjet: arcjet.com](https://arcjet.com) read 2026-11-05 - [Castle: castle.io](https://castle.io) read 2026-11-05 - [Centinel Analytica: centinelanalytica.com](https://www.centinelanalytica.com) read 2026-11-05 - [Cequence Security: cequence.ai](https://www.cequence.ai) read 2026-11-05 - [Cloudflare: cloudflare.com](https://www.cloudflare.com) read 2026-11-05 - [DataDome: datadome.co](https://datadome.co) read 2026-11-05 - [F5: f5.com](https://www.f5.com) read 2026-11-05 - [Fastly: fastly.com](https://www.fastly.com) read 2026-11-05 - [Fingerprint: fingerprint.com](https://fingerprint.com) read 2026-11-05 - [Google Cloud Fraud Defense: cloud.google.com](https://cloud.google.com/security/products/recaptcha) read 2026-11-05 - [hCaptcha: hcaptcha.com](https://www.hcaptcha.com) read 2026-11-05 - [HUMAN Security: humansecurity.com](https://www.humansecurity.com) read 2026-11-05 - [Imperva: imperva.com](https://www.imperva.com) read 2026-11-05 - [Kasada: kasada.io](https://www.kasada.io) read 2026-11-05 - [Known Agents: knownagents.com](https://knownagents.com) read 2026-11-05 - [Link11: link11.com](https://www.link11.com) read 2026-11-05 - [Netacea: netacea.com](https://netacea.com) read 2026-11-05 - [Netlify: netlify.com](https://www.netlify.com) read 2026-11-05 - [Radware: radware.com](https://www.radware.com) read 2026-11-05 - [Vercel: vercel.com](https://vercel.com) read 2026-11-05 - [Verisoul: verisoul.ai](https://www.verisoul.ai) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Crawler operators: 31 vendor profiles | Agent Ready Crawlers" description: "Crawler operators: 31 profiles with standards, dated prices, sources and what each does not do. 16 of 31 document support for RFC 9309. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/by-type/crawler-operator site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Crawler operators: 31 vendor profiles | Agent Ready Crawlers > Crawler operators: 31 profiles with standards, dated prices, sources and what each does not do. 16 of 31 document support for RFC 9309. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/by-type/crawler-operator Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendors by type # Crawler operators In short 31 crawler operators are profiled, from OpenAI and Google to browser-agent companies such as Kernel and Notte. 16 of 31 document support for RFC 9309, 9 for Web Bot Auth and 9 for published IP lists. A site owner can read down the standards column to see which of them publish a signature, an IP file or neither. The 31 crawler operators run alphabetically from Amazon to You.com, and carry 298 sourced facts between them. The price models on record include subscription plus credits; platform usage; subscription plus usage; usage-based; subscription with credits; and 4 more. 30 profiles carry our own probes of key directories or IP files. Each profile lists what the vendor does not do next to what it does. - [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) - [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/) - [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) - [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) - [All 73](https://agentreadycrawlers.com/vendors/) | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Amazon | Crawler operator | Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 202, 4 probes 2026-11-05 | 2026-11-05 | | Anchor Browser | Crawler operator | Cloud browser infrastructure for computer-use agents that can sign sessions with Cloudflare Web Bot Auth on its upper plans, and also sells stealth fingerprinting, proxies, CAPTCHA bypass and a pay-on-success web unlocker. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Anthropic | Crawler operator | Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Apify | Crawler operator | Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. | 0; partial: rfc9309, web-bot-auth, cloudflare-verified-bots | platform usage; subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Apple | Crawler operator | Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Block | Crawler operator | Company that built goose, an open-source local AI agent Cloudflare classed as a signed agent in 2025; Block has since donated goose to the Linux Foundation's Agentic AI Foundation and publishes no key directory for it. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Browser Use | Crawler operator | Maker of the open-source browser-use agent library and a hosted cloud browser for agents that ships with stealth fingerprinting, residential proxies and CAPTCHA solving on by default, and no published identifier for site owners. | 0 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Browserbase | Crawler operator | Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned. | 2 web-bot-auth, cloudflare-verified-bots; partial: rfc9309 | subscription plus usage | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Cohere | Crawler operator | Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page. | 1 rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Common Crawl | Crawler operator | Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Diffbot | Crawler operator | Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. | 1 rfc9309 | subscription with credits; credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | DuckDuckGo | Crawler operator | DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Exa | Crawler operator | AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities. | 2 rfc9309, web-bot-auth | pay as you go | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Firecrawl | Crawler operator | Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends. | 1 rfc9309 | credit subscription | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Google | Crawler operator | Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic. | 2 ip-list, reverse-dns; partial: rfc9309, web-bot-auth | No price recorded | Homepage HTTP 200, 5 probes 2026-11-05 | 2026-11-05 | | Hive | Crawler operator | AI model company, renamed from Hive to Deploy in October 2026, whose ImageSift unit runs ImagesiftBot to collect public images and page text for a reverse image search index. | 1 rfc9309 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Huawei | Crawler operator | Huawei runs PetalBot for the Petal search engine, whose index also feeds Huawei Assistant and AI Search; PetalBot follows robots.txt and is verified by reverse DNS under aspiegel.com or an IP lookup tool. | 2 rfc9309, reverse-dns | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Hyperbrowser | Crawler operator | Cloud browser platform for AI agents whose sessions can sign requests with Web Bot Auth under the fixed Signature-Agent www.hyperbrowser.ai, on request per team, alongside stealth, proxies and CAPTCHA solving. | 1 web-bot-auth | credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Kernel | Crawler operator | Cloud browser company for AI agents that runs two signed identities, Kernel Agent for user-directed browsing and KernelSearchBot for search crawling, each with its own Web Bot Auth key directory so sites can treat them separately. | 3 rfc9309, web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Manus | Crawler operator | General-purpose AI agent, owned by Meta since December 2025, that browses either in an isolated cloud browser or through an extension inside the user's own browser, and publishes no identifier, robots.txt policy or signing keys for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Meta | Crawler operator | Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt. | 0; partial: rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Microsoft | Crawler operator | Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Mistral AI | Crawler operator | Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two. | 1 rfc9309; partial: ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Notte | Crawler operator | Browser platform for AI agents (sessions, agents and serverless browser functions) that says it is registered as a verified bot with Cloudflare and signs session requests with Web Bot Auth when a customer turns it on. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | OpenAI | Crawler operator | OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. | 2 web-bot-auth, ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 403, 5 probes 2026-11-05 | 2026-11-05 | | Parallel Web Systems | Crawler operator | Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only. | 2 rfc9309, web-bot-auth; partial: ip-list | pay per request | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Perplexity | Crawler operator | Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Timpi | Crawler operator | Operator of an independent search index built by a decentralised network of contributor nodes; its crawler appears in robots.txt files as Timpibot, but Timpi publishes no crawler documentation for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Webz.io | Crawler operator | Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training. | 1 rfc9309 | pay as you go | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | xAI | Crawler operator | xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic. | 0 | No price recorded | Homepage HTTP 403, 1 probe 2026-11-05 | 2026-11-05 | | You.com | Crawler operator | You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | ## Sources - [The full vendor list (this site)](https://agentreadycrawlers.com/vendors/) read 2026-11-04 - [Amazon: amazon.com](https://www.amazon.com) read 2026-11-05 - [Anchor Browser: anchorbrowser.io](https://anchorbrowser.io) read 2026-11-05 - [Anthropic: anthropic.com](https://www.anthropic.com) read 2026-11-05 - [Apify: apify.com](https://apify.com) read 2026-11-05 - [Apple: apple.com](https://www.apple.com) read 2026-11-05 - [Block: block.xyz](https://block.xyz) read 2026-11-05 - [Browser Use: browser-use.com](https://browser-use.com) read 2026-11-05 - [Browserbase: browserbase.com](https://www.browserbase.com) read 2026-11-05 - [Cohere: cohere.com](https://cohere.com) read 2026-11-05 - [Common Crawl: commoncrawl.org](https://commoncrawl.org) read 2026-11-05 - [Diffbot: diffbot.com](https://www.diffbot.com) read 2026-11-05 - [DuckDuckGo: duckduckgo.com](https://duckduckgo.com) read 2026-11-05 - [Exa: exa.ai](https://exa.ai) read 2026-11-05 - [Firecrawl: firecrawl.dev](https://www.firecrawl.dev) read 2026-11-05 - [Google: google.com](https://www.google.com) read 2026-11-05 - [Hive: deploy.com](https://deploy.com) read 2026-11-05 - [Huawei: huawei.com](https://www.huawei.com) read 2026-11-05 - [Hyperbrowser: hyperbrowser.ai](https://www.hyperbrowser.ai) read 2026-11-05 - [Kernel: kernel.sh](https://www.kernel.sh) read 2026-11-05 - [Manus: manus.im](https://manus.im) read 2026-11-05 - [Meta: meta.com](https://www.meta.com) read 2026-11-05 - [Microsoft: microsoft.com](https://www.microsoft.com) read 2026-11-05 - [Mistral AI: mistral.ai](https://mistral.ai) read 2026-11-05 - [Notte: notte.cc](https://www.notte.cc) read 2026-11-05 - [OpenAI: openai.com](https://openai.com) read 2026-11-05 - [Parallel Web Systems: parallel.ai](https://parallel.ai) read 2026-11-05 - [Perplexity: perplexity.ai](https://www.perplexity.ai) read 2026-11-05 - [Timpi: timpi.io](https://timpi.io) read 2026-11-05 - [Webz.io: webz.io](https://webz.io) read 2026-11-05 - [xAI: x.ai](https://x.ai) read 2026-11-05 - [You.com: you.com](https://you.com) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Licensing intermediaries: 8 vendor profiles | Agent Ready Crawlers" description: "Licensing intermediaries: 8 profiles with standards, dated prices, sources and what each does not do. 3 of 8 document support for RFC 9309. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Licensing intermediaries: 8 vendor profiles | Agent Ready Crawlers > Licensing intermediaries: 8 profiles with standards, dated prices, sources and what each does not do. 3 of 8 document support for RFC 9309. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendors by type # Licensing intermediaries In short 8 licensing intermediaries are profiled, from marketplaces such as TollBit and Dappier to the RSL standard and IAB Tech Lab's CoMP. 3 of 8 document support for RFC 9309, 1 for Web Bot Auth and 1 for reverse DNS. Most of them price access without blocking anyone, and enforcement stays with the publisher's CDN or bot manager. The 8 licensing intermediaries run alphabetically from Dappier to TollBit, and carry 85 sourced facts between them. The price models on record include publisher-set price per query; open specification; sales-led; annual subscription; free with Tools Portal registration; and 5 more. None of the profiles carries a key directory or IP file probe of our own. Each profile lists what the vendor does not do next to what it does. - [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) - [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/) - [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) - [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) - [All 73](https://agentreadycrawlers.com/vendors/) | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Dappier | Licensing intermediary | Content marketplace where publishers list their content as priced RAG models and AI agents buy query access through an agent storefront that takes x402, MPP or card payments without signup. | 0; partial: rfc9309 | publisher-set price per query | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | FetchRight | Licensing intermediary | Licensing service behind the open Peek-Then-Pay protocol: a site publishes /.well-known/peek.json, AI agents get a priced HTTP 203 preview, buy a JWT license per intent and present it to an edge enforcer. | 0 | open specification; sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | IAB Tech Lab | Licensing intermediary | Ad-industry standards body whose Content Monetization Protocols (CoMP) 1.0 define how a licensed AI crawler asks a publisher for content and gets a token, alongside a paid Spiders and Bots List and a free Agent Registry. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | open specification; annual subscription; 1 more | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | ProRata | Licensing intermediary | AI attribution company that splits an AI answer into claims and measures each source's share as a basis for licensing and payment, and runs Gist Answers on a network of licensed publisher titles. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | RSL Collective | Licensing intermediary | Nonprofit collective rights organisation and co-developer of the RSL standard, a machine-readable license file a site links from robots.txt so crawlers can see, buy and prove a license for training, AI input, search or crawling. | 1 rfc9309 | open standard; free to join for publishers | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Skyfire | Licensing intermediary | Agent identity and payment network whose signed KYA and pay tokens let a website, API or bot manager admit a verified agent and charge it per request, with Skyfire vetting the agent platform and the person or business behind the agent. | 0 | subscription for verification (Individual or Organization) | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Supertab | Licensing intermediary | Managed RSL licensing service: Supertab Connect hosts a publisher's license.xml and RSL license server, issues license tokens to crawlers, checks them at the CDN with its SDK and bills the use, priced from USD 99 a month. | 1 rfc9309 | monthly subscription by visitors and licensees | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | TollBit | Licensing intermediary | Licensing marketplace that sends AI bots, by user agent at the publisher's CDN, to an Agent Site on a tollbit subdomain, where bots holding a short-lived TollBit JWT get licensed content at publisher-set rates per 1,000 pages. | 0; partial: rfc9309 | free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee | Homepage HTTP 200 2026-11-05 | 2026-11-05 | ## Sources - [The full vendor list (this site)](https://agentreadycrawlers.com/vendors/) read 2026-11-04 - [Dappier: dappier.com](https://dappier.com) read 2026-11-05 - [FetchRight: fetchright.ai](https://fetchright.ai) read 2026-11-05 - [IAB Tech Lab: iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/) read 2026-11-05 - [ProRata: prorata.ai](https://prorata.ai) read 2026-11-05 - [RSL Collective: rslcollective.org](https://rslcollective.org) read 2026-11-05 - [Skyfire: skyfire.xyz](https://skyfire.xyz) read 2026-11-05 - [Supertab: supertab.co](https://www.supertab.co) read 2026-11-05 - [TollBit: tollbit.com](https://tollbit.com) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Verification providers: 11 vendor profiles | Agent Ready Crawlers" description: "Verification providers: 11 profiles with standards, dated prices, sources and what each does not do. 3 of 11 document support for Web Bot Auth." canonical: https://agentreadycrawlers.com/vendors/by-type/verification-provider site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Verification providers: 11 vendor profiles | Agent Ready Crawlers > Verification providers: 11 profiles with standards, dated prices, sources and what each does not do. 3 of 11 document support for Web Bot Auth. Source: https://agentreadycrawlers.com/vendors/by-type/verification-provider Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendors by type # Verification providers In short 11 verification providers are profiled, from card networks such as Visa and Mastercard to identity companies such as Prove and Vouched. 3 of 11 document support for Web Bot Auth, 1 for Visa TAP and 1 for Mastercard Agent Pay. The 11 verification providers run alphabetically from American Express to World, and carry 107 sourced facts between them. The price models on record include add-on, 50% of the base plan price; free tier, then monthly plans; sales-led; free tier, then contact sales; per fingerprint above the included volume; and 4 more. 1 profile carries our own probes of key directories or IP files. Each profile lists what the vendor does not do next to what it does. - [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) - [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/) - [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) - [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) - [All 73](https://agentreadycrawlers.com/vendors/) | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | American Express | Verification provider | Card network whose ACE developer kit plans an agent registration service so only verified AI agents get tokenized Amex credentials, with Agent Purchase Protection for registered agent purchases; the registration spec is still in development. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Auth0 for AI Agents | Verification provider | Okta's customer identity platform; its AI Agents add-on gives agents a company builds a user-bound identity, scoped tokens, a Token Vault for third-party APIs and Cross App Access, and it verifies no inbound crawlers. | 0 | add-on, 50% of the base plan price | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Descope | Verification provider | Customer and agent identity platform whose Agentic Identity Hub gives AI agents and MCP clients OAuth 2.1 access, short-lived scoped credentials and an agent directory with policies, for agents calling a company's APIs rather than crawling its site. | 0 | free tier, then monthly plans | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Mastercard | Verification provider | Card network whose Agent Pay registers AI agents and has them sign requests with Web Bot Auth under the tag Agent-pay-auth, so a merchant or its CDN can check the agent's key in a network directory before letting it buy. | 2 web-bot-auth, mastercard-agent-pay; partial: visa-tap | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Microsoft Entra Agent ID | Verification provider | Microsoft's identity service for AI agents inside an Entra tenant: each agent gets its own service principal created from a blueprint, with tokens, a human sponsor and audit, for agents an organisation runs rather than agents visiting a website. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Prove | Verification provider | Identity verification company whose Verified Agent product binds an AI agent to a verified person or business with identity-bound tokens, checks participants against a shared trust registry and logs co-signed consent for agent purchases. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Stytch IsAgent | Verification provider | Identity platform, part of Twilio since November 2025, that verifies Web Bot Auth signatures through its IsAgent SDK and Device Fingerprinting, and issues scoped OAuth tokens to AI agents and MCP clients through Connected Apps. | 1 web-bot-auth | free tier, then contact sales; per fingerprint above the included volume | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Trulioo | Verification provider | Identity verification company whose Know Your Agent programme, in alpha, runs business and AML checks on the company behind an AI agent, binds the agent's fingerprint to it in a signed Digital Agent Profile and lets counterparties check it by API or MCP. | 0 | alpha, access on request | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Visa | Verification provider | Card network whose Trusted Agent Protocol lets a merchant or its CDN verify, from RFC 9421 signatures tagged agent-browser-auth or agent-payer-auth, that a visiting agent is approved under Visa Intelligent Commerce and acting for a known consumer. | 2 web-bot-auth, visa-tap | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Vouched | Verification provider | Identity verification company whose Know Your Agent suite detects AI agents on a site by behaviour, enforces allow, challenge, block or redirect rules, and ties each agent action to a verified person through signed delegations. | 0 | free; monthly fee plus per transaction | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | World | Verification provider | Proof-of-human network whose AgentKit lets a person with a verified World ID register an agent's wallet, so x402-protected sites can tell a human-backed agent from a script and limit access per unique person. | 0 | developer preview | Homepage HTTP 200 2026-11-05 | 2026-11-05 | ## Sources - [The full vendor list (this site)](https://agentreadycrawlers.com/vendors/) read 2026-11-04 - [American Express: americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/) read 2026-11-05 - [Auth0 for AI Agents: auth0.com](https://auth0.com/ai) read 2026-11-05 - [Descope: descope.com](https://www.descope.com/use-cases/ai) read 2026-11-05 - [Mastercard: developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/) read 2026-11-05 - [Microsoft Entra Agent ID: learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities) read 2026-11-05 - [Prove: prove.com](https://www.prove.com/solutions/agentic-suite) read 2026-11-05 - [Stytch IsAgent: stytch.com](https://stytch.com) read 2026-11-05 - [Trulioo: trulioo.com](https://www.trulioo.com/products/kya-verification) read 2026-11-05 - [Visa: developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol) read 2026-11-05 - [Vouched: vouched.id](https://www.vouched.id/know-your-agent) read 2026-11-05 - [World: world.org](https://world.org) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Castle: bot management vendor profile | Agent Ready Crawlers" description: "Castle: bot management vendor profile covering Castle Risk API with AI agent identification. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/castle site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Castle: bot management vendor profile | Agent Ready Crawlers > Castle: bot management vendor profile covering Castle Risk API with AI agent identification. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/castle Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Castle In short Castle's agent object carries the agent's name, operator and category, and records how it knew, in order of strength: a verified Web Bot Auth signature, a browser fingerprint, then the user agent string. Its docs warn that a missing object does not prove a human sent the request, and the Free plan covers about 1,000 Risk API calls. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [castle.io](https://castle.io) In one line Bot and account abuse API whose risk responses include an agent object that names AI agents and browser automation, identified by a verified Web Bot Auth signature, a browser fingerprint or a user-agent string. Last read 2026-11-05 ## What Castle does for this layer ### From the vendor's own pages Castle adds an optional agent object to its API response with name, operator, category (ai\_agent, browser\_automation or ai\_browser) and identified\_by. ([docs.castle.io](https://docs.castle.io/docs/ai-agent-identification)) Castle's identified\_by values are web\_bot\_auth, fingerprint and user\_agent; a verified signature takes precedence over a fingerprint match, which takes precedence over a user-agent match. ([docs.castle.io](https://docs.castle.io/docs/ai-agent-identification)) Castle names Meta Muse, Grok Bot, ChatGPT agent and Manus among the personal AI agents it identifies. ([docs.castle.io](https://docs.castle.io/docs/ai-agent-identification)) To verify Web Bot Auth, Castle asks customers to forward the Signature-Input, Signature, Signature-Agent and original Host headers in context.headers. ([docs.castle.io](https://docs.castle.io/docs/ai-agent-identification)) Castle's docs warn that an omitted agent object does not establish that a request came from a human. ([docs.castle.io](https://docs.castle.io/docs/ai-agent-identification)) Castle's Free plan includes USD 5 of API usage, about 5,000 IP lookups or 1,000 Risk API calls, and Pro costs USD 200 a month with USD 200 of usage. ([castle.io](https://castle.io/pricing/)) Castle's homepage marks AI agent detection as new and describes it as identifying AI agents and browser automation tools in an application. ([castle.io](https://castle.io/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Castle Risk API with AI agent identification | Bot and abuse scores plus an agent object naming the agent and how it was identified. | usage-based plans Free includes USD 5 of usage; Pro USD 200 a month. | USD 0 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Castle verifies Web Bot Auth signatures for supported signers. (docs.castle.io) | | Visa Trusted Agent Protocol | No statement | No statement in the Castle docs read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement in the Castle docs read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Castle does not do - Castle does not block requests at the edge; it returns scores and agent identity to the customer's backend, which decides. - Castle does not use agent identity to change its bot or abuse scores. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) no token - [Grok Bot (agent)](https://agentreadycrawlers.com/crawlers/grok-bot/) no token - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token - [Manus cloud browser](https://agentreadycrawlers.com/crawlers/manus/) no token ## Sources - [castle.io](https://castle.io) read 2026-11-05 - [AI agent identification](https://docs.castle.io/docs/ai-agent-identification) read 2026-11-05 - [Pricing – Castle](https://castle.io/pricing/) read 2026-11-05 - [Castle – Stop bots and account abuse at scale](https://castle.io/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Centinel Analytica: vendor profile | Agent Ready Crawlers" description: "Centinel Analytica: bot management vendor profile covering Centinel bot management and Bot directory and AI crawler list. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/centinel-analytica site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Centinel Analytica: vendor profile | Agent Ready Crawlers > Centinel Analytica: bot management vendor profile covering Centinel bot management and Bot directory and AI crawler list. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/centinel-analytica Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Centinel Analytica In short Centinel Analytica runs its check at the site's own edge, on whichever CDN, server or WordPress setup the customer already has, before the page is served. It claims that most requests calling themselves GPTBot are impostors, and checks them against OpenAI's published ranges. Prices are quoted per deployment, and a free audit looks only at scraping exposure. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [centinelanalytica.com](https://www.centinelanalytica.com) In one line Bot and AI crawler protection vendor whose edge integration verifies declared search and AI crawlers against operators' published IP ranges and detects scrapers that run real browsers behind residential proxies. Last read 2026-11-05 ## What Centinel Analytica does for this layer ### From the vendor's own pages Centinel's GPTBot page tells site owners to verify GPTBot by checking that the source IP falls in the ranges OpenAI publishes at openai.com/gptbot.json, and shows 2026-09-30 as the last verified date. ([centinelanalytica.com](https://www.centinelanalytica.com/bots/gptbot)) Centinel integrates through AWS CloudFront, Akamai EdgeWorkers, Cloudflare Workers, Fastly Compute, Fastly VCL, NGINX or OpenResty, and WordPress. ([centinelanalytica.com](https://www.centinelanalytica.com/integrations)) In Centinel's design the site's edge asks Centinel to analyse each request before serving the page, then applies the policy decision Centinel returns. ([centinelanalytica.com](https://www.centinelanalytica.com/)) ### Claims the vendor makes about itself Centinel reports that for the median organisation 94% of requests claiming to be GPTBot failed verification, measured between 14 Jul and 10 Oct 2026. ([centinelanalytica.com](https://www.centinelanalytica.com/bots/gptbot), dated 2026-10-10) Centinel's homepage says that on a typical news site 84% of requests claiming to be GPTBot were fake, as were 14% of those claiming to be Googlebot. ([centinelanalytica.com](https://www.centinelanalytica.com/)) Centinel claims it detected 8 of 8 automation tools in its September 2026 benchmark, against at most 5 for six other vendors. ([centinelanalytica.com](https://www.centinelanalytica.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Centinel bot management | Crawler verification, scripted-client and browser-automation detection, and per-crawler policy at the edge. | quoted per deployment Free site audit first. | Not published | 2026-11-05 | | Bot directory and AI crawler list | Public pages per crawler with user agent, verification method and failed-verification rates. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | Centinel pages read 2026-11-05 name no Web Bot Auth signature check. | | Published IP list | Supports | Centinel verifies GPTBot against OpenAI's published IP ranges. (centinelanalytica.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Centinel Analytica does not do - Centinel's free site audit checks scraping exposure only, not account takeover or API abuse. - Centinel's pages read name no Web Bot Auth signature verification (read 2026-11-05). ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) token GPTBot - [Bytespider](https://agentreadycrawlers.com/crawlers/bytespider/) token Bytespider - [Comet (browser assistant)](https://agentreadycrawlers.com/crawlers/perplexity-comet/) no token ## Sources - [centinelanalytica.com](https://www.centinelanalytica.com) read 2026-11-05 - [GPTBot | Centinel bot directory](https://www.centinelanalytica.com/bots/gptbot) read 2026-11-05 - [Centinel Analytica | Bot and AI crawler protection](https://www.centinelanalytica.com/) read 2026-11-05 - [Integrations: Cloudflare, Fastly, NGINX and more | Centinel](https://www.centinelanalytica.com/integrations) read 2026-11-05 - [Pricing and how to start | Centinel](https://www.centinelanalytica.com/pricing) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Cequence Security: vendor profile | Agent Ready Crawlers" description: "Cequence Security: bot management vendor profile covering Bot Management with Agent Trust and AI Gateway. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/cequence site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Cequence Security: vendor profile | Agent Ready Crawlers > Cequence Security: bot management vendor profile covering Bot Management with Agent Trust and AI Gateway. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/cequence Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Cequence Security In short Cequence Security offers bot management with an Agent Trust feature aimed at shoppers' personal AI agents, Meta Muse and OpenAI Dots among them, yet its pages never say how an agent's identity is checked. Sensors can sit inline to block, rate limit or deceive, or watch passively, on premises or in the cloud, and no price is listed. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [cequence.ai](https://www.cequence.ai) In one line Application and API protection vendor whose Bot Management includes Agent Trust for personal AI shopping agents, and whose AI Gateway governs enterprise agents' tool calls behind OAuth 2.1 authentication. Last read 2026-11-05 ## What Cequence Security does for this layer ### From the vendor's own pages Cequence's bot management page says Agent Trust verifies each agent's identity, keeps analysing its behavior, and stops scraping or account abuse while allowing legitimate actions. ([cequence.ai](https://www.cequence.ai/products/bot-management)) Cequence names Meta Muse and OpenAI Dots as personal AI agents that now shop and buy on customers' behalf. ([cequence.ai](https://www.cequence.ai/products/bot-management)) Cequence lists blocking, rate limiting, header injection and deception as its real-time bot mitigation options. ([cequence.ai](https://www.cequence.ai/products/bot-management)) Cequence deploys on premises, in the cloud or hybrid, with software sensors that inspect traffic passively or sit inline for real-time mitigation. ([cequence.ai](https://www.cequence.ai/products/bot-management)) Cequence AI Gateway authenticates each enterprise agent through the customer's identity infrastructure and applies DLP scanning to agent requests and MCP server responses with more than 100 built-in detection types. ([cequence.ai](https://www.cequence.ai/learn/ai-agent-identity)) ### Claims the vendor makes about itself Cequence's homepage reports 10 billion daily application interactions and 200 million daily agentic interactions on its platform. ([cequence.ai](https://www.cequence.ai/)) Cequence says it blocked tens of millions of inventory-scraping requests for Ulta Beauty. ([cequence.ai](https://www.cequence.ai/products/bot-management)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot Management with Agent Trust | Bot detection, fraud prevention and agent identity checks with behavior monitoring. | sales-led Pages offer a personalized demo. | Not published | 2026-11-05 | | AI Gateway | Governance layer between enterprise AI agents and applications, data and APIs, with OAuth 2.1 and per-tool limits. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | Cequence pages read 2026-11-05 do not say which verification method Agent Trust uses. | | Visa Trusted Agent Protocol | No statement | No statement on the Cequence pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the Cequence pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Cequence Security does not do - Cequence's pages read do not name the method Agent Trust uses to verify an agent: no Web Bot Auth, IP list or reverse DNS statement (read 2026-11-05). - Cequence publishes no price for Bot Management or AI Gateway. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) no token ## Sources - [cequence.ai](https://www.cequence.ai) read 2026-11-05 - [Bot Management | Bot Detection, Mitigation, and Fraud Prevention | Cequence](https://www.cequence.ai/products/bot-management) read 2026-11-05 - [Advanced Application, API, and AI Protection](https://www.cequence.ai/) read 2026-11-05 - [How AI Agent Identity Works, Top 7 Risks & Critical Best Practices](https://www.cequence.ai/learn/ai-agent-identity) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Cloudflare: bot management profile | Agent Ready Crawlers" description: "Cloudflare: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15." canonical: https://agentreadycrawlers.com/vendors/cloudflare site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Cloudflare: bot management profile | Agent Ready Crawlers > Cloudflare: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. Source: https://agentreadycrawlers.com/vendors/cloudflare Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 1 (tied) of 15 # Cloudflare In short Cloudflare lets a publisher treat search, training and agent crawlers differently on any plan, free included, and scores 8 of 10 points. Charging and commerce agents grade partial: per-request charging for crawlers is a private beta and the Visa and Mastercard agent rules are announced but unconfirmed, with Pro from USD 20 a month. CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/) · [cloudflare.com](https://www.cloudflare.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Based in United States Homepage [cloudflare.com](https://www.cloudflare.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | Web Bot Auth signatures, IP validation, verified bot and signed agent directory (BotBase) | Verified bots must validate with Web Bot Auth or IP validation; signed agents tracked with a Direct versus Intermediary field. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Yes | separate Search, Training and Agent controls on all plans, plus Disallow AI Training since 2026-09-15 | Training control has Allow, Block and Disallow AI Training; Search and Agent are set separately per zone. Source, read 2026-11-04. | | Charging for access | Partial | Pay per crawl with HTTP 402, Cloudflare as Merchant of Record, private beta | Joining requires contacting Cloudflare; not self-serve. Cloudflare's overview page calls it a private beta. Source, read 2026-11-04. | | Lets signed commerce agents through | Partial | Visa TAP and Mastercard Agent Pay built on Web Bot Auth, managed rulesets announced Oct 2025; current rule name not confirmed in docs (checked 2026-11-04) | Blog describes planned managed rulesets that admit Visa or Mastercard registered agents. Source, read 2026-11-04. | | Cost | Not graded | free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | Plan table lists Free, Pro and Business prices; Enterprise is custom. Source, read 2026-11-04. | | Reporting | Yes | AI Crawl Control analytics on crawler activity, request patterns and robots.txt compliance | Docs list Analyze AI traffic and Track robots.txt features. Source, read 2026-11-04. | | Where it runs | Not graded | edge: enforced on Cloudflare's reverse proxy, no app code needed | Controls are set per zone under Security Settings. Source, read 2026-11-04. | ## Cost AI Crawl Control and the Search, Training and Agent controls are on all plans including Free (USD 0). Pro USD 20/month billed annually or USD 25 monthly; Business USD 200/month annually or USD 250 monthly; Enterprise Bot Management and BotBase are sales-led. Pay per crawl prices are set by the site owner (private beta). [Source](https://www.cloudflare.com/plans/network-cdn.md), read 2026-11-04. ## What Cloudflare does not do - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. - Cloudflare's pay per crawl cannot price crawlers differently: one Charge price covers every crawler set to Charge (FAQ updated 2026-04-23). - Cloudflare's managed robots.txt only states a preference; Cloudflare's own docs say crawlers may disregard it, so blocking needs AI Crawl Control rules. - No managed ruleset for Trusted Agent Protocol or Agent Pay agents appears in Cloudflare's bot docs index read on 2026-11-05, a year after the October 2025 announcement. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source AI Crawl Control (formerly AI Audit) is available on all plans and lets site owners set allow or block policies per AI crawler, review crawler activity and check which crawlers follow robots.txt. [Source](https://developers.cloudflare.com/ai-crawl-control/), read 2026-11-04, dated 2026-08-14. - Primary source Under pay per crawl a crawler either sends payment intent in request headers and gets HTTP 200, or receives HTTP 402 with pricing, and Cloudflare acts as Merchant of Record. [Source](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/), read 2026-11-04, dated 2026-07-28. - Primary source Disallow AI Training, launched 15 Sep 2026, publishes a no-training preference in robots.txt, keeps crawlers labelled Accountable allowed for search, and blocks training-only crawlers. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source To be labelled Accountable, a crawler operator must offer a training opt-out, a way to opt out of AI summaries, URL-level visibility into pages used for training, and an assurance that opting out of training does not affect search results. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source Cloudflare labels Googlebot, Applebot and Bingbot as Accountable mixed-use crawlers, and also labels the separate search and training crawlers of Amazon, Anthropic, Meta and OpenAI. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source Disallow AI Training does not yet pass a no-training preference to Bing through robots.txt; Cloudflare says Microsoft targets early 2027 for domain-level support. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source The Block AI Bots toggle is being replaced by separate Search, Training and Agent controls, and Managed Robots.txt is being replaced by Bot Preference Sync; existing Training settings of Block move to Disallow AI Training. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04, dated 2026-09-15. - Primary source Verified bots must validate with Web Bot Auth or IP validation; since 1 Jul 2026 signed agents sit under the verified bot umbrella with a Direct versus Intermediary metadata field. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04, dated 2026-07-01. - Primary source Cloudflare's behavior taxonomy has 11 entries: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social / Link Preview, Feed Fetching, and Monitoring & Operations. [Source](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-11-04, dated 2026-07-01. - Primary source BotBase, the directory of known bots and agents, is available only to Enterprise Bot Management customers and is a visibility tool; blocking needs Security rules or the AI traffic options. [Source](https://developers.cloudflare.com/bots/botbase/), read 2026-11-04, dated 2026-10-08. - Primary source Bots register for Web Bot Auth by hosting a signed key directory at /.well-known/http-message-signatures-directory; Cloudflare accepts valid Ed25519 keys and currently does no nonce validation. [Source](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/), read 2026-11-04, dated 2026-10-08. - Primary source Visa's Trusted Agent Protocol and Mastercard's Agent Pay both use Web Bot Auth, tagging requests agent-browser-auth or agent-payer-auth, and Cloudflare said it would ship managed rulesets to admit registered agents. [Source](https://blog.cloudflare.com/secure-agentic-commerce/), read 2026-11-04, dated 2025-10-24. ## Dated news - 2026-09-15**Cloudflare launches Disallow AI Training and the Accountable crawler label**. Cloudflare added a Disallow AI Training option to its Training control. It writes a no-training preference into robots.txt, keeps Accountable mixed-use crawlers such as Googlebot, Applebot and Bingbot allowed for search, and blocks training-only crawlers. Block AI Bots and Managed Robots.txt are being deprecated. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/). - 2026-09-28**Coverage: Cloudflare asks Google to skip AI training without blocking search**. Trade coverage of the Disallow AI Training rollout, describing how the setting relies on robots.txt tokens such as Google-Extended so pages stay in search while opting out of model training, and noting the Bing gap until Microsoft adds support. [Source](https://winbuzzer.com/2026/09/28/cloudflare-ask-google-skip-ai-training-without-blocking-search-a002-xcxwbn/). ## Compared with - [Cloudflare vs DataDome](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome/): Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. - [Akamai vs Cloudflare](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare/): Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). - [TollBit vs Cloudflare](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/): TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. - [Cloudflare vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control/): AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. - [Cloudflare vs Vercel](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid/): Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. - [Fastly vs Cloudflare](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare/): Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. - [Cloudflare vs Netlify](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify/): Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. - [Cloudflare vs HUMAN Security](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security/): Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. - [Arcjet vs Cloudflare](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare/): Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. - [Imperva vs Cloudflare](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare/): Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. - [Skyfire vs Cloudflare](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare/): Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. ## What Cloudflare does for this layer ### From the vendor's own pages Cloudflare's pay per crawl FAQ lists /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json as always free to crawl, and says error responses are never billed. ([developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/index.md), dated 2026-04-23) A crawler that wants to pay through Cloudflare must sign with Web Bot Auth, follow the verified bots policy and be submitted through the Bot Submission Form; until then it receives HTTP 402 with a crawler-price header such as "USD 0.01". ([developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler/index.md), dated 2026-07-01) Cloudflare's Bot Preference Sync prepends managed robots.txt directives and the Content Signals Policy only when at least one Search, Agent or Training policy is set to Block, Block on pages with ads, or, for Training, Disallow. ([developers.cloudflare.com](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), dated 2026-11-02) Cloudflare's robots.txt documentation states that robots.txt compliance is voluntary and sends site owners to AI Crawl Control to enforce crawler blocks. ([developers.cloudflare.com](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), dated 2026-11-02) Cloudflare published the Content Signals Policy on 24 Sep 2025 under a CC0 licence, defining three signals for robots.txt: search, ai-input and ai-train. ([blog.cloudflare.com](https://blog.cloudflare.com/content-signals-policy/), dated 2025-09-24) Cloudflare Browser Run, formerly Browser Rendering, signs its requests with Web Bot Auth and adds headers that cannot be removed (cf-brapi-request-id, cf-brapi-devtools, cf-biso-devtools); its crawl endpoint uses the fixed user agent CloudflareBrowserRenderingCrawler/1.0. ([developers.cloudflare.com](https://developers.cloudflare.com/browser-run/reference/automatic-request-headers/), dated 2026-10-30) Cloudflare's 24 Oct 2025 post says Visa and Mastercard will each host a key directory for their registered agents, and that commerce agents must include a nonce in the signature to guard against replay. ([blog.cloudflare.com](https://blog.cloudflare.com/secure-agentic-commerce/), dated 2025-10-24) Cloudflare's plan page grades bot mitigation by tier: simple bots on Free, easy-to-detect bots on Pro, sophisticated bots with basic bot analytics on Business, and anomaly detection with advanced bot analytics on Contract. ([cloudflare.com](https://www.cloudflare.com/plans/network-cdn.md)) Cloudflare announced on 15 Jan 2026 that it had acquired Human Native, which its release calls "an AI data marketplace that connects creators and AI developers", to help publishers prepare and price content for AI companies willing to pay. ([cloudflare.com](https://www.cloudflare.com/press/press-releases/2026/cloudflare-strengthens-content-offering-to-ai-companies-with-acquisition-of-human-native/), dated 2026-01-15) ### Claims the vendor makes about itself At the Content Signals launch Cloudflare said its customers had turned on managed robots.txt for more than 3.8 million domains, and that it would not serve an ai-input signal on their behalf. ([blog.cloudflare.com](https://blog.cloudflare.com/content-signals-policy/), dated 2025-09-24) ### What our crawl saw A GET with the crawl user agent on 2026-11-05 to the Browser Run key directory at web-bot-auth.cloudflare-browser-rendering-085.workers.dev returned HTTP 200, media type application/http-message-signatures-directory+json, and three Ed25519 keys. ([web-bot-auth.cloudflare-browser-rendering-085.workers.dev](https://web-bot-auth.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | AI Crawl Control | Per-crawler allow or block policies, AI traffic analytics and robots.txt compliance tracking. | included in every plan Overview page says available on all plans. | USD 0 | 2026-11-05 | | Pay per crawl | HTTP 402 charging for verified AI crawlers, Stripe payouts, Cloudflare as Merchant of Record; private beta. | site owner sets one per-request price Example header in the docs: crawler-price USD 0.01. | Set by the site owner | 2026-11-05 | | Bot Management (Contract plan) | Bot scores, anomaly detection, custom CAPTCHAs and advanced bot analytics; BotBase directory for Enterprise. | contract Pro USD 20 a month billed annually (USD 25 monthly); Business USD 200 a month billed annually (USD 250 monthly). | Custom | 2026-11-05 | | Browser Run (formerly Browser Rendering) | Headless browser service whose requests are signed with Web Bot Auth; Cloudflare is the operator of this agent. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Bot Preference Sync writes managed robots.txt directives from the Search, Agent and Training policies. (developers.cloudflare.com) | | Web Bot Auth | Supports | Pay per crawl requires crawlers to sign with Web Bot Auth; Browser Run signs its own requests. (developers.cloudflare.com) | | Content Signals | Supports | Cloudflare wrote the Content Signals Policy (CC0) and serves it in managed robots.txt. (blog.cloudflare.com) | | Published IP list | Supports | The bots docs index lists IP validation: matching request IPs against published bot IP lists. (developers.cloudflare.com) | | Reverse DNS check | No statement | The bots docs index (read 2026-11-05) lists IP validation and Web Bot Auth, not reverse DNS. | | Cloudflare verified bots | Supports | Cloudflare runs the verified bots program; paying crawlers must be added to it. (developers.cloudflare.com) | | Visa Trusted Agent Protocol | Announced | Cloudflare said in Oct 2025 it would create managed rulesets for agents using the Trusted Agent Protocol. (blog.cloudflare.com) | | Mastercard Agent Pay | Announced | The same Oct 2025 post announced managed rulesets for agents using Agent Pay. (blog.cloudflare.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Browser Run Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 3 Ed25519 keys | GET with the crawl user agent | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Cloudflare Browser Rendering](https://agentreadycrawlers.com/crawlers/cloudflare-browser-rendering/) no token - [Alternatives to Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/alternatives/) graded on the rubric ## Sources - [cloudflare.com: cloudflare.com](https://www.cloudflare.com) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [developers.cloudflare.com: verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [developers.cloudflare.com: what is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [developers.cloudflare.com: ai crawl control](https://developers.cloudflare.com/ai-crawl-control/) read 2026-11-04 - [developers.cloudflare.com: botbase](https://developers.cloudflare.com/bots/botbase/) read 2026-11-04 - [developers.cloudflare.com: web bot auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) read 2026-11-04 - [winbuzzer.com: cloudflare ask google skip ai training without blocking search a002 xcxwbn](https://winbuzzer.com/2026/09/28/cloudflare-ask-google-skip-ai-training-without-blocking-search-a002-xcxwbn/) read 2026-11-04 - [Pay Per Crawl FAQ · Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/index.md) read 2026-11-05 - [Verify your AI crawler · Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler/index.md) read 2026-11-05 - [robots.txt setting · Cloudflare bot solutions docs](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) read 2026-11-05 - [Giving users choice with Cloudflare’s new Content Signals Policy | Cloudflare Blog](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-05 - [Automatic request headers · Cloudflare Browser Run docs](https://developers.cloudflare.com/browser-run/reference/automatic-request-headers/) read 2026-11-05 - [Cloudflare Browser Run Web Bot Auth key directory (JSON)](https://web-bot-auth.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory) read 2026-11-05 - [Cloudflare Strengthens Content Offering to AI Companies with Acquisition of Human Native](https://www.cloudflare.com/press/press-releases/2026/cloudflare-strengthens-content-offering-to-ai-companies-with-acquisition-of-human-native/) read 2026-11-05 - [Cloudflare bot solutions docs index (llms.txt)](https://developers.cloudflare.com/bots/llms.txt) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Cloudflare | Agent Ready Crawlers" description: "Alternatives to Cloudflare: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/cloudflare/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Cloudflare | Agent Ready Crawlers > Alternatives to Cloudflare: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/cloudflare/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Cloudflare In short Cloudflare shares first place with 8 of 10 points and has more head-to-head pages than any other vendor: eleven, from DataDome and Akamai to Netlify and Skyfire. Its tie partners are Akamai, AWS WAF Bot Control and DataDome. Charging and commerce agents grade partial, since pay per crawl is a private beta. ## Head-to-head, with the sourced difference - [**Cloudflare vs DataDome**](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome/). Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. [Source](https://datadome.co/pricing/), read 2026-11-04. - [**Cloudflare vs Akamai**](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare/). Akamai routes AI bot charging through partners TollBit and Skyfire; Cloudflare runs its own pay per crawl with HTTP 402 and acts as Merchant of Record (private beta). [Source](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue), read 2026-11-04. - [**Cloudflare vs TollBit**](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/). TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. [Source](https://blog.cloudflare.com/introducing-pay-per-crawl/), read 2026-11-04. - [**Cloudflare vs AWS WAF Bot Control**](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control/). AWS WAF charges AI bots with an x402 HTTP 402 and settles in stablecoins through Coinbase (since 2026-06-15, CloudFront only); Cloudflare charges through pay per crawl with itself as Merchant of Record. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. - [**Cloudflare vs Vercel**](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid/). Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. - [**Cloudflare vs Fastly**](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare/). Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. - [**Cloudflare vs Netlify**](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify/). Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04. - [**Cloudflare vs HUMAN Security**](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security/). Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. - [**Cloudflare vs Arcjet**](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare/). Arcjet runs in application code with published prices from USD 25 a month; Cloudflare runs at its edge with AI controls on the free plan. [Source](https://arcjet.com/pricing), read 2026-11-04. - [**Cloudflare vs Imperva**](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare/). Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. [Source](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), read 2026-11-04. - [**Cloudflare vs Skyfire**](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare/). Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. Skyfire documents enforcing its tokens on Cloudflare with a Worker. [Source](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Cloudflare's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Cloudflare on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | commerce agents | per-purpose policy | No public price; sales-led | | AWS WAF Bot Control | 8 | charging | commerce agents | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | DataDome | 8 | charging | per-purpose policy | From USD 3,830/month | | HUMAN Security | 7 | nothing | commerce agents | No public price; sales-led | | Skyfire | 6 | charging, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | charging | verifies agents, per-purpose policy, commerce agents | Free for publishers to start | | Vercel | 6 | nothing | charging, commerce agents | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | nothing | per-purpose policy, charging, commerce agents | No public price; sales-led | | Fastly | 4 | nothing | verifies agents, per-purpose policy, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, per-purpose policy, charging, commerce agents, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | nothing | verifies agents, per-purpose policy, charging, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy, charging, commerce agents | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, charging, commerce agents, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, per-purpose policy, charging, commerce agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Cloudflare is not the fit - BotBase does not block traffic itself; mitigation needs Security rules or AI traffic options (BotBase docs, updated 2026-10-08). - Disallow AI Training cannot pass a no-training preference to Bing until Microsoft ships robots.txt support, targeted early 2027 (Cloudflare blog, 2026-09-15). - Web Bot Auth verification does no nonce validation today; Cloudflare relies on short expires values (Web Bot Auth docs). - Pay per crawl is not self-serve: the docs (updated 2026-07-28) tell site owners to reach out to Cloudflare to join the beta. ## Sources - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [docs.netlify.com: block ai crawlers](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [arcjet.com: pricing](https://arcjet.com/pricing) read 2026-11-04 - [cpl.thalesgroup.com: thales google protect ai agent ecosystem](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem) read 2026-11-04 - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Cohere: crawler operator profile | Agent Ready Crawlers" description: "Cohere: crawler operator profile. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/cohere site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Cohere: crawler operator profile | Agent Ready Crawlers > Cohere: crawler operator profile. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/cohere Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Cohere In short Cohere has told site owners that it currently runs no crawler gathering pages for model training, and expects any future one to be built around robots.txt, with page changes announced in its docs changelog. Fetches its products make on a user's behalf fall outside that page, and the sample opt-out it shows names Coherebot rather than cohere-ai. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [cohere.com](https://cohere.com) In one line Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page. Last read 2026-11-05 ## What Cohere does for this layer ### From the vendor's own pages Cohere's crawler table listed N/A for both bot name and description when read on 2026-11-05. ([docs.cohere.com](https://docs.cohere.com/docs/cohere-web-crawlers)) Cohere states that its policy is to require that crawlers be designed to respect robots.txt. ([docs.cohere.com](https://docs.cohere.com/docs/cohere-web-crawlers)) Cohere's example robots.txt block for a future crawler uses the token Coherebot; the page does not mention cohere-ai. ([docs.cohere.com](https://docs.cohere.com/docs/cohere-web-crawlers)) Cohere warns that blocking IP addresses may not give a reliable or lasting opt-out, because it prevents a crawler from reading robots.txt. ([docs.cohere.com](https://docs.cohere.com/docs/cohere-web-crawlers)) Cohere says changes to the crawler page will be announced in the changelog on its docs site, which offers an RSS feed. ([docs.cohere.com](https://docs.cohere.com/docs/cohere-web-crawlers)) ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Cohere's policy requires crawlers designed to respect robots.txt. (docs.cohere.com) | | Web Bot Auth | No statement | Key directory path 404 on cohere.com (observed 2026-11-05). | | Published IP list | No statement | No crawler, so no IP list (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | ## What Cohere does not do - Cohere publishes no user agent string, IP list or verification method, because it says it runs no training crawler. - Cohere's crawler page covers training crawlers only and says nothing about fetches made by its products at a user's request. ## Reading notes The [cohere-ai row](https://agentreadycrawlers.com/crawlers/cohere-ai/) in the Crawler Index takes its token from a third party. Cohere's own crawler page names no crawler and uses Coherebot only in an example, so the two disagree. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [cohere-ai](https://agentreadycrawlers.com/crawlers/cohere-ai/) token cohere-ai ## Sources - [cohere.com](https://cohere.com) read 2026-11-05 - [Cohere Web Crawlers](https://docs.cohere.com/docs/cohere-web-crawlers) read 2026-11-05 - [cohere.com Web Bot Auth key directory path (GET probe)](https://cohere.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Common Crawl: crawler operator profile | Agent Ready Crawlers" description: "Common Crawl: crawler operator profile covering CCBot and Opt-Out Ledger. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/common-crawl site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Common Crawl: crawler operator profile | Agent Ready Crawlers > Common Crawl: crawler operator profile covering CCBot and Opt-Out Ledger. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/common-crawl Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Common Crawl In short Common Crawl's CCBot identifies as version 2.0, follows up to four redirects, runs no JavaScript and sets no cookies, and can be verified by reverse DNS on its dedicated ranges, except over IPv6. Since 17 September 2025 the non-profit has published each legal opt-out request it receives, in what is now called the Opt-Out Ledger. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [commoncrawl.org](https://commoncrawl.org) In one line Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests. Last read 2026-11-05 ## What Common Crawl does for this layer ### From the vendor's own pages CCBot identifies itself as CCBot/2.0 with a link to commoncrawl.org/faq/; Common Crawl's older bot sent CCBot/1.0. ([commoncrawl.org](https://commoncrawl.org/faq)) Common Crawl says CCBot now runs on dedicated IP ranges with reverse DNS under crawl.commoncrawl.org, except over IPv6, where reverse DNS is not yet supported. ([commoncrawl.org](https://commoncrawl.org/ccbot)) Common Crawl publishes CCBot's IPv4 and IPv6 ranges as JSON at index.commoncrawl.org/ccbot.json. ([commoncrawl.org](https://commoncrawl.org/ccbot)) Common Crawl warns that it knows of crawlers that falsely identify themselves as CCBot and recommends verifying the user agent. ([commoncrawl.org](https://commoncrawl.org/ccbot)) Common Crawl says CCBot obeys Crawl-delay in robots.txt and uses an adaptive back-off that slows requests when a server answers HTTP 429 or 5xx. ([commoncrawl.org](https://commoncrawl.org/faq)) CCBot follows up to four consecutive redirects, or five when fetching robots.txt in line with RFC 9309, and does not execute JavaScript or use cookies. ([commoncrawl.org](https://commoncrawl.org/faq)) Common Crawl's FAQ says the dataset is a sample of the web rather than whole sites, and that CCBot uses sitemaps announced in robots.txt. ([commoncrawl.org](https://commoncrawl.org/faq)) Common Crawl began publishing every legal opt-out request it had received on 2025-09-17 as the Opt-Out Registry, now renamed the Opt-Out Ledger. ([commoncrawl.org](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry), dated 2025-09-17) ### Claims the vendor makes about itself Common Crawl describes itself as a non-profit foundation that maintains an open repository of web crawl data that anyone can access and analyse. ([commoncrawl.org](https://commoncrawl.org/ccbot)) ### What our crawl saw commoncrawl.org returned HTTP 404 at the Web Bot Auth key directory path on 2026-11-05. ([commoncrawl.org](https://commoncrawl.org/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | CCBot | Nutch-based crawler that builds Common Crawl's open web archive. | No price recorded | | Opt-Out Ledger | Public list of the legal opt-out requests Common Crawl has received. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | CCBot checks robots.txt before fetching, obeys Crawl-delay and cites RFC 9309 for redirect handling. (commoncrawl.org) | | Web Bot Auth | No statement | Not on the CCBot page; key directory path 404 on commoncrawl.org (observed 2026-11-05). | | Published IP list | Supports | Ranges published at index.commoncrawl.org/ccbot.json. (commoncrawl.org) | | Reverse DNS check | Supports | Dedicated ranges resolve under crawl.commoncrawl.org with forward confirmation (IPv4 only). (commoncrawl.org) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | ## What Common Crawl does not do - CCBot has no reverse DNS over IPv6, so IPv6 requests can only be checked against the range file. - A robots.txt Disallow stops future CCBot crawls; Common Crawl's FAQ does not say it removes pages from earlier crawl archives. ## Reading notes The robots.txt at index.commoncrawl.org disallows our crawl token, so we did not fetch the [CCBot](https://agentreadycrawlers.com/crawlers/ccbot/) range file ourselves and counted no prefixes in it. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [CCBot](https://agentreadycrawlers.com/crawlers/ccbot/) token CCBot ## Sources - [commoncrawl.org](https://commoncrawl.org) read 2026-11-05 - [Common Crawl: CCBot](https://commoncrawl.org/ccbot) read 2026-11-05 - [Common Crawl: Frequently Asked Questions](https://commoncrawl.org/faq) read 2026-11-05 - [Common Crawl Foundation Opt-Out Registry](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry) read 2026-11-05 - [commoncrawl.org Web Bot Auth key directory path (GET probe)](https://commoncrawl.org/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Dappier: licensing intermediary profile | Agent Ready Crawlers" description: "Dappier: licensing intermediary profile covering RAG API Marketplace, Agent storefront and Ask AI and WordPress plugin. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/dappier site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Dappier: licensing intermediary profile | Agent Ready Crawlers > Dappier: licensing intermediary profile covering RAG API Marketplace, Agent storefront and Ask AI and WordPress plugin. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/dappier Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Licensing intermediary # Dappier In short Dappier's storefront for agents, which listed nine API services on 5 November 2026, sells access without signup and settles by x402, MPP or card. Publishers choose their own per-query price for the branded RAG models they list, while Dappier gives only general advice, such as publishing robots.txt and rate limiting, for keeping bots out. Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [dappier.com](https://dappier.com) In one line Content marketplace where publishers list their content as priced RAG models and AI agents buy query access through an agent storefront that takes x402, MPP or card payments without signup. Last read 2026-11-05 ## What Dappier does for this layer ### From the vendor's own pages Dappier's publisher guide has a publisher list its content as a branded RAG model in the Dappier RAG Marketplace and set its own price per query. ([docs.dappier.com](https://docs.dappier.com/publish-and-monetize.md)) Dappier says the models in its marketplace at marketplace.dappier.com are vetted by Dappier and draw on the publisher's data, trusted sources and real-time data before an LLM writes the answer. ([docs.dappier.com](https://docs.dappier.com/rag-marketplace.md)) Dappier's homepage tells visiting agents that its storefront at agents.dappier.com/llms.txt publishes prices, payment instructions and setup, and that agents can buy without signing up. ([dappier.com](https://dappier.com/)) Dappier's agent storefront listed nine API services on 2026-11-05: 9 and 10 News, Benzinga, CafeMom Parenting, iHeartCats AI, iHeartDogs AI, MethodShop, Real Time Search and WISH-TV AI. ([agents.dappier.com](https://agents.dappier.com/llms.txt)) Dappier's storefront accepts payment by x402, MPP or card and routes card checkouts through Stripe-hosted pages. ([agents.dappier.com](https://agents.dappier.com/llms.txt)) Dappier's ZeroClick endpoints for real-time search and AI recommendations authenticate each call by verifying a zc-signature HMAC over the raw request bytes. ([docs.dappier.com](https://docs.dappier.com/llms.txt)) Dappier's documentation includes a WordPress plugin and an Ask AI widget so publishers can run the same content models on their own sites. ([docs.dappier.com](https://docs.dappier.com/llms.txt)) Dappier's bot deterrence guide gives publishers generic advice to publish robots.txt, add CAPTCHAs, rate-limit requests and monitor traffic, without a Dappier enforcement product. ([docs.dappier.com](https://docs.dappier.com/bot-deterrence.md)) ### Reported by a third party IAB Tech Lab reported that Dappier's co-founders presented AI bot traffic data at its August 2025 workshop that led to the CoMP working group. ([iabtechlab.com](https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/), dated 2025-08-19) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | RAG API Marketplace | Marketplace of publisher content models sold to AI developers per query. | publisher-set price per query Dappier's own fee not stated | Not published | 2026-11-05 | | Agent storefront | llms.txt-based storefront where agents look up and buy Dappier API access with x402, MPP or card. | No price recorded | | Ask AI and WordPress plugin | On-site AI search widget and plugin for publishers. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | Dappier's bot deterrence guide recommends publishing robots.txt; Dappier does not enforce it. (docs.dappier.com) | | Web Bot Auth | No statement | Agents authenticate to Dappier with ZeroClick HMAC signatures or storefront identity, not Web Bot Auth (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Dappier does not do - Dappier does not stand at the publisher's door; it sells licensed access through its own APIs and leaves crawler blocking on the publisher's site to the publisher. - Dappier's docs do not state the fee Dappier keeps from per-query sales (read 2026-11-05). ## Reading notes Dappier's storefront llms.txt contains instructions addressed to visiting agents, with look-up and purchase steps. They were read as data only, and nothing was bought or posted. ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [dappier.com](https://dappier.com) read 2026-11-05 - [Dappier docs: Monetize Your Content](https://docs.dappier.com/publish-and-monetize.md) read 2026-11-05 - [Dappier docs: RAG API Marketplace](https://docs.dappier.com/rag-marketplace.md) read 2026-11-05 - [Dappier: AI Monetization for Publishers and Advertisers](https://dappier.com/) read 2026-11-05 - [Dappier agent storefront (llms.txt)](https://agents.dappier.com/llms.txt) read 2026-11-05 - [Dappier API Guide index (llms.txt)](https://docs.dappier.com/llms.txt) read 2026-11-05 - [Dappier docs: Bot Deterrence](https://docs.dappier.com/bot-deterrence.md) read 2026-11-05 - [Announcing the Content Monetization Protocols (CoMP) for AI Working Group](https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "DataDome: bot management profile | Agent Ready Crawlers" description: "DataDome: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15." canonical: https://agentreadycrawlers.com/vendors/datadome site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # DataDome: bot management profile | Agent Ready Crawlers > DataDome: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. Source: https://agentreadycrawlers.com/vendors/datadome Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 1 (tied) of 15 # DataDome In short DataDome sells bot and fraud protection from USD 3,830 a month, grading yes on agent verification, charging and reporting and partial on per-purpose policy and commerce agents, for 8 of 10 points. Its Agent Trust layer needs the full Bot Protect installation underneath it. Bot and fraud protection vendor whose Agent Trust layer identifies AI agents, scores them per customer, applies per-agent policies and connects to monetization partners, with prices published from USD 3,830 per month. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to DataDome](https://agentreadycrawlers.com/vendors/datadome/alternatives/) · [datadome.co](https://datadome.co) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [datadome.co](https://datadome.co) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | Web Bot Auth, KYA tokens, official IP lists, reverse DNS, plus fingerprinting fallback | Strong identification methods listed in Agentic Trust docs. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | policies per agent, trust score and intent; no published search versus training switch (checked 2026-11-04) | Access control from global default to custom rules on subsets of traffic. Source, read 2026-11-04. | | Charging for access | Yes | AI crawler and agent monetization through partners, from the Advanced plan | Pricing table lists AI crawler and agent monetization from Advanced. Source, read 2026-11-04. | | Lets signed commerce agents through | Partial | TAP and KYA trust-authority integrations listed; Experian Agent Trust member since 2026-10-01 | Pricing page integrations row and Experian release. Source, read 2026-11-04. | | Cost | Not graded | published: from USD 3,830/month | Essentials plan price. Source, read 2026-11-04. | | Reporting | Yes | agent traffic by intent, identity, geo, URL and endpoint type | Traffic Insights section of Agentic Trust docs. Source, read 2026-11-04. | | Where it runs | Not graded | both: server-side module on web servers, CDNs or API gateways plus client-side tag | Prerequisites section of Agentic Trust docs. Source, read 2026-11-04. | ## Cost Bot Protect Essentials USD 3,830/month (under 100M requests/month), Advanced USD 8,670/month (under 200M), Premium USD 10,160/month (under 300M), Enterprise from USD 13,270/month (over 300M). Agent Trust capability listed from Essentials; monetization from Advanced. [Source](https://datadome.co/pricing/), read 2026-11-04. ## What DataDome does not do - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). - DataDome's partner docs list Skyfire as the only supported trust authority; Visa's Trusted Agent Protocol and Mastercard Agent Pay are not named (read 2026-11-05). - A DataDome workspace cannot split paid agent traffic across several monetization partners; one active partner is allowed. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Agentic Trust is built on Bot Protect and needs both the server-side module and the client-side tag installed; incomplete integrations may give partial or missing agent data. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04, dated 2026-05-05. - Primary source For strong identification DataDome uses Web Bot Authentication, KYA (Know Your Agent), official IP source lists and reverse DNS, and falls back to behavioral and technical fingerprinting for agents that do not support them. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04, dated 2026-05-05. - Primary source Each identified agent gets an identification strength score, and a trust score combines identification strength with fraud history, adjusted per customer. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04, dated 2026-05-05. - Primary source Intent detection models look for vulnerability scanning, credential stuffing, fake account creation, payment fraud, spamming, scalping and scraping, and block them even when the agent is strongly authenticated. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04, dated 2026-05-05. - Primary source Agent traffic can be explored by intent, agent identity, geo-location, domain and URL, and endpoint type, with full request details for every HTTP request. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04, dated 2026-05-05. - Primary source Bot Protect Essentials is listed at USD 3,830 per month for under 100 million requests a month and includes the Agent Trust capability. [Source](https://datadome.co/pricing/), read 2026-11-04. - Primary source AI crawler and agent monetization starts on the Advanced plan at USD 8,670 per month; protection for agentic endpoints such as MCP starts on Premium at USD 10,160 per month; Enterprise starts at USD 13,270 per month. [Source](https://datadome.co/pricing/), read 2026-11-04. - Primary source Historical data retention is 7 days on Essentials, 15 days on Advanced and 30 days on Premium and Enterprise. [Source](https://datadome.co/pricing/), read 2026-11-04. - Vendor claim The pricing comparison lists third-party trust authority integrations such as TAP and KYA as an integrations row. [Source](https://datadome.co/pricing/), read 2026-11-04. - Primary source On 1 Oct 2026 Experian announced DataDome joined its Experian Agent Trust ecosystem, using verified Know Your Agent signals for continuous intent checks during a session. [Source](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous), read 2026-11-04, dated 2026-10-01. - Secondary source DataDome's Q3 2026 AI Traffic Report counted nearly 25 billion AI agent requests, up 41.5% quarter over quarter, with Meta-ExternalAgent and Meta-WebIndexer at 49.7% of the total. [Source](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report), read 2026-11-04, dated 2026-10-15. - Vendor claim DataDome says it was named a Leader in The Forrester Wave: Bot and Agent Trust Management Software, Q2 2026. [Source](https://datadome.co/pricing/), read 2026-11-04. ## Dated news - 2026-10-01**DataDome joins Experian Agent Trust ecosystem**. Experian announced DataDome as a member of its open Agent Trust framework for agentic commerce. DataDome adds continuous intent verification, using verified Know Your Agent signals to check whether an agent's actions stay within its delegated authority during a session. [Source](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous). - 2026-10-15**DataDome Q3 2026 AI Traffic Report**. DataDome reported nearly 25 billion AI agent requests in Q3 2026, up 41.5% on Q2. Meta's crawlers made up 49.7% of AI agent traffic, and Meta's Muse agentic browser sent 29.1 million requests in under two weeks without identifying itself as an AI agent. [Source](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report). ## Compared with - [Cloudflare vs DataDome](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome/): Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. - [DataDome vs HUMAN Security](https://agentreadycrawlers.com/compare/datadome-vs-human-security/): DataDome publishes plan prices (from USD 3,830 a month); HUMAN publishes none and rates agents High, Medium or Low trust, with High reserved for agents that carry a cryptographic key on each request. - [DataDome vs Akamai](https://agentreadycrawlers.com/compare/datadome-vs-akamai/): DataDome publishes plan prices and lists AI monetization from USD 8,670 a month; Akamai publishes no prices and names Visa TAP, Skyfire and Experian KYA, Auth0 and TollBit in its June 2026 agentic framework. - [DataDome vs Kasada](https://agentreadycrawlers.com/compare/datadome-vs-kasada/): DataDome publishes prices and monetization tiers; Kasada publishes none but says it can deploy behind an incumbent bot manager. - [AWS WAF Bot Control vs DataDome](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome/): AWS WAF charges AI bots natively over x402 at no extra fee beyond WAF charges; DataDome charges through partners from its USD 8,670 Advanced plan. - [Imperva vs DataDome](https://agentreadycrawlers.com/compare/imperva-vs-datadome/): Imperva's product page names agentic AI as a traffic class but lists no AI crawler or agent controls; DataDome documents Web Bot Auth, KYA, IP lists and reverse DNS for agent identification. ## What DataDome does for this layer ### From the vendor's own pages DataDome rates Web Bot Authentication, cryptographic token verification, trusted IP source lists and reverse DNS as high-strength identification, network attribution and client-side fingerprinting as medium, and server-side fingerprinting as low. ([docs.datadome.co](https://docs.datadome.co/docs/ai-agent-identification)) DataDome adds each new agent detection model to its catalog switched off and gives customers a two-week window to review its description, Trust Score and recommended policy before it activates. ([docs.datadome.co](https://docs.datadome.co/docs/ai-agent-identification)) DataDome lists Skyfire as its supported trust authority: DataDome verifies Skyfire's cryptographic signatures and uses Skyfire's agent registry to name the agent. ([docs.datadome.co](https://docs.datadome.co/docs/trust-monetization-partners)) TollBit is DataDome's supported monetization partner; a workspace can have only one active monetization partner, and the Monetize response replaces Block or Challenge on the models it is set for. ([docs.datadome.co](https://docs.datadome.co/docs/trust-monetization-partners)) DataDome's Bot Authentication page offers bot operators five ways to be recognised: Web Bot Auth, two-way reverse DNS, a static IP list, a dynamic IP list file in CSV, JSON or plain text, and a private AS number. ([docs.datadome.co](https://docs.datadome.co/docs/bot-authentication)) DataDome blocks requests from unauthenticated bots and AI agents by default as Threat Detection, and asks operators to use a dedicated user agent before applying for its Verified Bot category. ([docs.datadome.co](https://docs.datadome.co/docs/bot-authentication)) DataDome stores trust authority fields with each request, including trustauthorityagentid, trustauthorityagentName and trustauthorityverificationstatus, for filtering and investigation. ([docs.datadome.co](https://docs.datadome.co/docs/trust-monetization-partners)) DataDome says Agentic Trust features, from agent identification to monetization, depend on traffic being routed through the DataDome integration. ([docs.datadome.co](https://docs.datadome.co/docs/agentic-trust-getting-started)) ### What our crawl saw datadome.co answered the crawl user agent with HTTP 403 on 2026-11-05, while docs.datadome.co served its documentation. ([datadome.co](https://datadome.co/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot Protect | Server-side module plus client-side tag for bot and fraud detection; Agentic Trust runs on top of it. | monthly, by request volume Essentials under 100M requests a month; monetization from Advanced at USD 8,670 a month (pricing page read 2026-11-04). | USD 3830 | 2026-11-04 | | Agentic Trust | Agent identification, trust scores, access control and monetization through partners. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Bot Authentication page: DataDome can cryptographically verify Web Bot Auth signatures. (docs.datadome.co) | | Published IP list | Supports | Static and dynamic IP lists are accepted; trusted IP source lists rate high strength. (docs.datadome.co) | | Reverse DNS check | Supports | Two-way RDNS/DNS lookup is one of the five recognition methods. (docs.datadome.co) | | Visa Trusted Agent Protocol | No statement | Partner docs read 2026-11-05 list Skyfire as the only trust authority. | | Mastercard Agent Pay | No statement | Partner docs read 2026-11-05 list Skyfire as the only trust authority. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to DataDome (Bot Protect, Agentic Trust)](https://agentreadycrawlers.com/vendors/datadome/alternatives/) graded on the rubric ## Sources - [datadome.co: datadome.co](https://datadome.co) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [experianplc.com: datadome joins experian agent trust ecosystem to add continuous](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous) read 2026-11-04 - [helpnetsecurity.com: datadome ai agent traffic report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report) read 2026-11-04 - [AI Agent Identification](https://docs.datadome.co/docs/ai-agent-identification) read 2026-11-05 - [Agentic Trust Dashboard Access Control](https://docs.datadome.co/docs/agentic-trust-dashboard-access-control) read 2026-11-05 - [Trust and Monetization Partners](https://docs.datadome.co/docs/trust-monetization-partners) read 2026-11-05 - [Bot Authentication](https://docs.datadome.co/docs/bot-authentication) read 2026-11-05 - [DataDome (homepage, HTTP 403 to the crawl user agent)](https://datadome.co/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to DataDome | Agent Ready Crawlers" description: "Alternatives to DataDome: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/datadome/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to DataDome | Agent Ready Crawlers > Alternatives to DataDome: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/datadome/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to DataDome In short DataDome ties with Akamai, Cloudflare and AWS WAF Bot Control on 8 of 10 points, and six head-to-head pages compare it with Cloudflare, HUMAN Security, Akamai, Kasada, AWS WAF Bot Control and Imperva. Plans start at USD 3,830 a month, and Agent Trust needs Bot Protect installed in full. ## Head-to-head, with the sourced difference - [**DataDome vs Cloudflare**](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome/). Cloudflare puts per-purpose Search, Training and Agent controls on every plan including Free; DataDome sells Agent Trust from USD 3,830 a month and only adds AI crawler monetization on its USD 8,670 Advanced plan. [Source](https://datadome.co/pricing/), read 2026-11-04. - [**DataDome vs HUMAN Security**](https://agentreadycrawlers.com/compare/datadome-vs-human-security/). DataDome publishes plan prices (from USD 3,830 a month); HUMAN publishes none and rates agents High, Medium or Low trust, with High reserved for agents that carry a cryptographic key on each request. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04. - [**DataDome vs Akamai**](https://agentreadycrawlers.com/compare/datadome-vs-akamai/). DataDome publishes plan prices and lists AI monetization from USD 8,670 a month; Akamai publishes no prices and names Visa TAP, Skyfire and Experian KYA, Auth0 and TollBit in its June 2026 agentic framework. [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04. - [**DataDome vs Kasada**](https://agentreadycrawlers.com/compare/datadome-vs-kasada/). DataDome publishes prices and monetization tiers; Kasada publishes none but says it can deploy behind an incumbent bot manager. [Source](https://www.kasada.io/), read 2026-11-04. - [**DataDome vs AWS WAF Bot Control**](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome/). AWS WAF charges AI bots natively over x402 at no extra fee beyond WAF charges; DataDome charges through partners from its USD 8,670 Advanced plan. [Source](https://aws.amazon.com/waf/pricing/), read 2026-11-04. - [**DataDome vs Imperva**](https://agentreadycrawlers.com/compare/imperva-vs-datadome/). Imperva's product page names agentic AI as a traffic class but lists no AI crawler or agent controls; DataDome documents Web Bot Auth, KYA, IP lists and reverse DNS for agent identification. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with DataDome's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than DataDome on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | commerce agents | charging | No public price; sales-led | | AWS WAF Bot Control | 8 | per-purpose policy | commerce agents | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | per-purpose policy | charging | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | HUMAN Security | 7 | per-purpose policy | charging, commerce agents | No public price; sales-led | | Skyfire | 6 | commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | nothing | verifies agents, commerce agents | Free for publishers to start | | Vercel | 6 | per-purpose policy | charging, commerce agents | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | nothing | charging, commerce agents | No public price; sales-led | | Fastly | 4 | nothing | verifies agents, charging, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, charging, commerce agents, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | nothing | verifies agents, charging, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy, charging, commerce agents | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, charging, commerce agents, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, charging, commerce agents, reporting | Platform plans from USD 0; extension price not stated | ## Where DataDome is not the fit - Agent Trust does not run on its own; Bot Protect must be fully installed with server and client components (Agentic Trust docs). - Monetization is not in the Essentials plan; it starts at Advanced (pricing page). - No published control that separates search crawling from training crawling as distinct purposes (docs and pricing read 2026-11-04). ## Sources - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Descope: agent verification provider profile | Agent Ready Crawlers" description: "Descope: agent verification provider profile covering Agentic Identity Hub. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/descope site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Descope: agent verification provider profile | Agent Ready Crawlers > Descope: agent verification provider profile covering Agentic Identity Hub. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/descope Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Descope In short Descope's October 2026 release lets organisations route requests for new internal agents through review before anything is provisioned, and lets those agents authenticate with certificates in place of static secrets, with MCP gateway integrations. The company says its hub already handles authentication for more than a thousand MCP servers. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [descope.com](https://www.descope.com/use-cases/ai) In one line Customer and agent identity platform whose Agentic Identity Hub gives AI agents and MCP clients OAuth 2.1 access, short-lived scoped credentials and an agent directory with policies, for agents calling a company's APIs rather than crawling its site. Last read 2026-11-05 ## What Descope does for this layer ### From the vendor's own pages Descope announced Agentic Identity Hub 3.0 on 15 Oct 2026 from Los Altos, California, adding governance for the AI agents and MCP servers organisations run internally. ([descope.com](https://www.descope.com/press-release/agentic-identity-hub-3), dated 2026-10-15) Agentic Identity Hub 3.0 added an Agentic Governance Suite in which employees request a new agent and the resources it needs, and the request is reviewed before anything is provisioned. ([descope.com](https://www.descope.com/press-release/agentic-identity-hub-3), dated 2026-10-15) Descope's 3.0 release integrates with MCP gateways including agentgateway, an open source project of the Agentic AI Foundation, to put one identity and policy layer in front of agents and MCP servers. ([descope.com](https://www.descope.com/press-release/agentic-identity-hub-3), dated 2026-10-15) Descope's 3.0 release lets agents authenticate with keys and certificates instead of static secrets and issues ID-JAG tokens so internal agents reach MCP servers through the organisation's existing identity provider. ([descope.com](https://www.descope.com/press-release/agentic-identity-hub-3), dated 2026-10-15) Descope lets a company register its product APIs as OAuth-protected resource servers so AI agents get scoped, delegated access without changes to the API logic. ([descope.com](https://www.descope.com/use-cases/ai)) Descope exposes MCP servers to AI agents with OAuth 2.1, client registration, consent and Cross-App Access through the customer's existing identity providers. ([descope.com](https://www.descope.com/use-cases/ai)) Descope's Free Forever plan costs USD 0 for 7,500 monthly active users, 2,000 monthly active consents and 2,000 monthly active tokens; the next plan starts at USD 249 a month. ([descope.com](https://www.descope.com/pricing)) ### Claims the vendor makes about itself Descope says its Agentic Identity Hub handles authentication and access control for 1000+ MCP servers and millions of monthly agentic transactions. ([descope.com](https://www.descope.com/press-release/agentic-identity-hub-3), dated 2026-10-15) Descope's AI use-case page shows Databricks, You.com, 6Sense and Cequence among its AI customers. ([descope.com](https://www.descope.com/use-cases/ai)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Agentic Identity Hub | OAuth 2.1 for MCP servers and APIs, agent credentials, agent directory, policies and audit for AI agents. | free tier, then monthly plans paid plans from USD 249 a month | USD 0 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The AI use-case page and 3.0 release cover OAuth, ID-JAG and client certificates, not Web Bot Auth (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Descope does not do - Descope does not verify or block crawlers and browsing agents arriving at a public website; it authenticates agents that call APIs and MCP servers the customer runs. - Descope's governance suite covers agents an organisation registers; agents built and run by outside vendors stay unknown unless they pass through an integrated MCP gateway. ## Reading notes Descope lists You.com as a customer, which does not mean Descope identifies traffic from [YouBot](https://agentreadycrawlers.com/crawlers/youbot/). ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [AI Agent and MCP Identity, Descope Agentic Identity Hub](https://www.descope.com/use-cases/ai) read 2026-11-05 - [Descope Unveils Agentic Identity Hub 3.0](https://www.descope.com/press-release/agentic-identity-hub-3) read 2026-11-05 - [Descope pricing](https://www.descope.com/pricing) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Diffbot: crawler operator profile | Agent Ready Crawlers" description: "Diffbot: crawler operator profile covering Crawl (Crawlbot), Extract APIs and Knowledge Graph and Web Search. Sources, standards and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/diffbot site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Diffbot: crawler operator profile | Agent Ready Crawlers > Diffbot: crawler operator profile covering Crawl (Crawlbot), Extract APIs and Knowledge Graph and Web Search. Sources, standards and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/diffbot Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Diffbot In short Diffbot's own whole-web crawl feeds its Knowledge Graph and, its FAQ says, is not used to train generative AI; a second token, Diffbot-User, covers pages a person asks to see. Customers can run hosted Crawlbot jobs from USD 899 a month, and those follow robots.txt only until the customer switches that setting off. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [diffbot.com](https://www.diffbot.com) In one line Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. Last read 2026-11-05 ## What Diffbot does for this layer ### From the vendor's own pages Diffbot's robots.txt FAQ names two user agents: Diffbot, for proactive crawling that feeds its Knowledge Graph and web search, and Diffbot-User, for requests made on behalf of a person browsing a URL with Diffbot software. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot says its web crawls follow robots.txt by default, including the disallow and crawl-delay directives. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot's FAQ says the Diffbot crawler "is not used for AI training" and that "Diffbot does not crawl pages for training generative AI foundation models". ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot states that a robots.txt instruction "can be ignored/overridden" in specific cases, typically when the customer has a partnership or agreement with the site being crawled. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot's crawl troubleshooting guide offers an "ignore robots.txt" setting in crawl settings or the Crawl API, and tells customers whose seed URL refuses connections to try turning on proxies. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/crawl-not-crawling)) For Extract and Crawlbot jobs, Diffbot recommends that customers set a User-Agent naming their own organisation and notes that robots.txt adherence in Crawlbot is on by default. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot's custom user agent guide shows a customer adding an agreed word to the User-Agent header for one domain, so that the site can allow that customer's requests and refuse other Diffbot customers. ([diffbot.com](https://www.diffbot.com/docs/custom-api/faq/custom-user-agents)) Diffbot's pricing page lists Free at USD 0 with 10,000 credits a month and 5 requests a minute, Startup at USD 299 a month and Plus at USD 899 a month, the first plan with access to Crawl (25 active crawls). ([diffbot.com](https://www.diffbot.com/pricing)) Diffbot charges 0 credits to spider a site for links and 2 credits instead of 1 to extract a page through a datacenter proxy, per its pricing table. ([diffbot.com](https://www.diffbot.com/pricing)) ### Claims the vendor makes about itself Diffbot says it runs its own crawl of the public web, independent from Google and Bing, on custom-assembled hardware in its own data center in California. ([diffbot.com](https://www.diffbot.com/company/about)) Diffbot says it has maintained a whole-web crawl "for over 20 years" and that its Global Index revisits pages on a freshness policy set per page type. ([diffbot.com](https://www.diffbot.com/technology)) ### What our crawl saw A GET for www.diffbot.com/.well-known/http-message-signatures-directory with the crawl user agent returned HTTP 404 on 2026-11-05, so Diffbot serves no Web Bot Auth key directory at that host. ([diffbot.com](https://www.diffbot.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Crawl (Crawlbot) | Hosted crawl jobs that spider a site from seed URLs and pass pages to an Extract API; robots.txt adherence on by default, customer can turn it off. | subscription with credits Crawl starts on the Plus plan, USD 899 a month, 25 active crawls. | USD 899 | 2026-11-05 | | Extract APIs | Classify a page and return structured JSON; custom user agents can be set per domain. | credits Free plan 10,000 credits a month; 1 credit per page, 2 through a datacenter proxy. | USD 0 | 2026-11-05 | | Knowledge Graph and Web Search | Entity graph and web search built from Diffbot's own crawl of the public web. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Diffbot's crawls follow robots.txt by default, including crawl-delay; customers can override it for their own jobs. (diffbot.com) | | Web Bot Auth | No statement | No key directory at www.diffbot.com (HTTP 404, 2026-11-05). | | Published IP list | No statement | The robots.txt FAQ names user agents only; no IP list was found (checked 2026-11-05). | | Reverse DNS check | No statement | No reverse DNS pattern published (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://www.diffbot.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Diffbot does not do - Diffbot publishes no IP range list, reverse DNS pattern or signature keys for its crawlers, so a request claiming to be Diffbot can be checked only against the user agent name (checked 2026-11-05). - Diffbot does not hold its customers to robots.txt: a customer crawl job can turn adherence off. ## Reading notes The [Diffbot row](https://agentreadycrawlers.com/crawlers/diffbot/) in the Crawler Index, read on 4 November 2026, says the full user agent string was not published. Diffbot's robots.txt FAQ, read the next day, adds the Diffbot-User token and the statement that the crawler is not used for AI training. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Diffbot](https://agentreadycrawlers.com/crawlers/diffbot/) token Diffbot ## Sources - [diffbot.com](https://www.diffbot.com) read 2026-11-05 - [Does Diffbot respect robots.txt? | Diffbot](https://www.diffbot.com/docs/crawl/faq/robots-txt) read 2026-11-05 - [Why is my crawl not crawling (and other uncommon crawl problems)? | Diffbot](https://www.diffbot.com/docs/crawl/faq/crawl-not-crawling) read 2026-11-05 - [How to Use Custom User Agents with Extract APIs | Diffbot](https://www.diffbot.com/docs/custom-api/faq/custom-user-agents) read 2026-11-05 - [About Diffbot | Diffbot](https://www.diffbot.com/company/about) read 2026-11-05 - [Technology | Diffbot](https://www.diffbot.com/technology) read 2026-11-05 - [Plans & Pricing | Diffbot](https://www.diffbot.com/pricing) read 2026-11-05 - [Diffbot key directory probe](https://www.diffbot.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "DuckDuckGo: crawler operator profile | Agent Ready Crawlers" description: "DuckDuckGo: crawler operator profile covering DuckDuckBot and DuckAssistBot. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/duckduckgo site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # DuckDuckGo: crawler operator profile | Agent Ready Crawlers > DuckDuckGo: crawler operator profile covering DuckDuckBot and DuckAssistBot. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/duckduckgo Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # DuckDuckGo In short DuckDuckGo's DuckAssistBot retrieves pages while an answer is composed, cites them, and is not used for model training; a Disallow takes 72 hours to apply. Its search crawler, DuckDuckBot, has an IP file of its own, but on 5 November 2026 the two files matched byte for byte, so addresses alone cannot separate the bots. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [duckduckgo.com](https://duckduckgo.com) In one line DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models. Last read 2026-11-05 ## What DuckDuckGo does for this layer ### From the vendor's own pages DuckDuckGo says a robots.txt Disallow for DuckAssistBot takes effect after 72 hours. ([duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)) DuckDuckGo says opting out of DuckAssistBot does not affect organic rankings or whether a site appears in DuckDuckGo search results. ([duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)) DuckAssistBot identifies itself as DuckAssistBot/1.2 with a link to duckduckgo.com/duckassistbot.html. ([duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)) DuckDuckGo says DuckDuckBot respects the rules at robotstxt.org and identifies itself as DuckDuckBot/1.1. ([duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot)) ### What our crawl saw DuckDuckGo's duckassistbot.json and duckduckbot.json were byte-identical on 2026-11-05 (same SHA-1), each with 486 IPv4 prefixes and a creationTime of 2026-09-01. ([duckduckgo.com](https://duckduckgo.com/duckassistbot.json), dated 2026-09-01) duckduckgo.com answered HTTP 404 for the Web Bot Auth key directory path on 2026-11-05. ([duckduckgo.com](https://duckduckgo.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | DuckDuckBot | Crawler for DuckDuckGo search results. | No price recorded | | DuckAssistBot | Real-time crawler for AI-assisted answers; not used for model training. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Both help pages describe robots.txt control; DuckAssistBot opt-out takes 72 hours. (duckduckgo.com) | | Web Bot Auth | No statement | Not on either help page; key directory path 404 (observed 2026-11-05). | | Published IP list | Supports | duckassistbot.json and duckduckbot.json. (duckduckgo.com) | | Reverse DNS check | No statement | Neither help page gives a reverse DNS pattern (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | duckassistbot.json and duckduckbot.json | identical files, 486 IPv4 prefixes each, creationTime 2026-09-01 | GET both and compare SHA-1 | 2026-11-05 | ## What DuckDuckGo does not do - DuckDuckGo's two range files were identical on 2026-11-05, so an IP check alone cannot tell DuckAssistBot from DuckDuckBot. - DuckDuckGo publishes no reverse DNS pattern or signature for either crawler (read 2026-11-05). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [DuckDuckGo in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/duckduckgo/) 2 rows - [DuckAssistBot](https://agentreadycrawlers.com/crawlers/duckassistbot/) token DuckAssistBot - [DuckDuckBot](https://agentreadycrawlers.com/crawlers/duckduckbot/) token DuckDuckBot ## Sources - [duckduckgo.com](https://duckduckgo.com) read 2026-11-05 - [Is DuckAssistBot related to DuckDuckGo?](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) read 2026-11-05 - [Is DuckDuckBot related to DuckDuckGo?](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) read 2026-11-05 - [DuckDuckGo duckassistbot.json IP ranges](https://duckduckgo.com/duckassistbot.json) read 2026-11-05 - [duckduckgo.com Web Bot Auth key directory path (GET probe)](https://duckduckgo.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [DuckDuckGo duckduckbot.json IP ranges](https://duckduckgo.com/duckduckbot.json) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Exa: crawler operator profile | Agent Ready Crawlers" description: "Exa: crawler operator profile covering Exa Search, Contents and Answer APIs. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/exa site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Exa: crawler operator profile | Agent Ready Crawlers > Exa: crawler operator profile covering Exa Search, Contents and Answer APIs. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/exa Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Exa In short Exa's search API draws on its own crawler, ExaSearchBot, whose requests are signed with a single published Ed25519 key; there is no IP list. A noindex tag removes a page from the index only after ExaSearchBot's next visit, since robots.txt governs crawling, not indexing. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [exa.ai](https://exa.ai) In one line AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities. Last read 2026-11-05 ## What Exa does for this layer ### From the vendor's own pages Exa's crawler page names three identities: ExaSearchBot for crawling, ExaUser for pages fetched on a user's request, and ExaSearchBot-Images for images on the pages it crawls. ([crawler.exa.ai](https://crawler.exa.ai/)) ExaSearchBot sends the user agent "Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)" and its robots.txt product token is ExaSearchBot. ([crawler.exa.ai](https://crawler.exa.ai/)) Exa signs every ExaSearchBot request with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme, carrying Signature, Signature-Input and Signature-Agent headers, and publishes its Ed25519 keys on crawler.exa.ai. ([crawler.exa.ai](https://crawler.exa.ai/)) When no robots.txt group addresses ExaSearchBot, Exa says the crawler follows the rules a site sets for major search engines, and only otherwise the \* group. ([crawler.exa.ai](https://crawler.exa.ai/)) Exa says robots.txt controls crawling but not indexing, and that it drops an indexed page after its next re-fetch when the page carries noindex in a robots meta tag or an X-Robots-Tag header. ([crawler.exa.ai](https://crawler.exa.ai/)) Exa says ExaSearchBot is rate-limited per site, does not try to bypass logins, paywalls or CAPTCHAs, and submits no forms. ([crawler.exa.ai](https://crawler.exa.ai/)) Exa's FAQ confirms the company operates ExaSearchBot to discover and refresh public pages for search and retrieval and points site owners to crawler.exa.ai for crawler controls. ([exa.ai](https://exa.ai/docs/admin/faqs.md)) Exa is pay-as-you-go: a Free Tier of USD 10 in credits that resets monthly, search from USD 4 per 1,000 requests, Contents at USD 1 per 1,000 pages, and an annual Enterprise plan at USD 12,000 paid upfront. ([exa.ai](https://exa.ai/docs/admin/pricing.md)) ### Claims the vendor makes about itself Exa says ExaSearchBot is validated automatically at the edge on Cloudflare, Akamai and other providers that support Web Bot Auth, with nothing for the site owner to configure. ([crawler.exa.ai](https://crawler.exa.ai/)) ### What our crawl saw ExaSearchBot's key directory at crawler.exa.ai/.well-known/http-message-signatures-directory returned HTTP 200, content type application/http-message-signatures-directory+json, with one Ed25519 key on 2026-11-05. ([crawler.exa.ai](https://crawler.exa.ai/.well-known/http-message-signatures-directory)) Exa's own robots.txt, read on 2026-11-05, declares Content-Signal "search=yes, ai-input=yes" for the whole site and adds ai-train=yes for /docs and /llms.txt. ([exa.ai](https://exa.ai/robots.txt)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Exa Search, Contents and Answer APIs | Web search and page retrieval for agents, built on the ExaSearchBot index. | pay as you go Per 1,000 search requests (lite or instant); free USD 10 credits each month. | USD 4 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | ExaSearchBot respects the Robots Exclusion Protocol, with its own token and a fallback to search-engine rules. (crawler.exa.ai) | | Web Bot Auth | Supports | Every ExaSearchBot request is signed; keys at crawler.exa.ai. (crawler.exa.ai) | | Content Signals | No statement | Exa uses Content-Signal lines in its own robots.txt, but the crawler page does not say whether ExaSearchBot reads them. | | Published IP list | No statement | The crawler page relies on signatures and says verification holds regardless of IP; no IP list published. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key | GET https://crawler.exa.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404 | GET https://exa.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Exa does not do - Exa publishes no IP range list for ExaSearchBot; its crawler page relies on signatures instead (read 2026-11-05). - Exa's crawler page does not say whether ExaUser requests follow robots.txt. ## Reading notes The [ExaSearchBot row](https://agentreadycrawlers.com/crawlers/exasearchbot/) in the Crawler Index, read on 4 November 2026, says Exa publishes no bot documentation. The page at crawler.exa.ai, read on 5 November, now documents the bot, its keys and its robots.txt rules. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [ExaSearchBot](https://agentreadycrawlers.com/crawlers/exasearchbot/) token ExaSearchBot ## Sources - [exa.ai](https://exa.ai) read 2026-11-05 - [Exa Search Crawler](https://crawler.exa.ai/) read 2026-11-05 - [FAQs | Exa docs](https://exa.ai/docs/admin/faqs.md) read 2026-11-05 - [ExaSearchBot key directory](https://crawler.exa.ai/.well-known/http-message-signatures-directory) read 2026-11-05 - [exa.ai robots.txt](https://exa.ai/robots.txt) read 2026-11-05 - [Pricing | Exa docs](https://exa.ai/docs/admin/pricing.md) read 2026-11-05 - [exa.ai key directory probe](https://exa.ai/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "F5: bot management vendor profile | Agent Ready Crawlers" description: "F5: bot management vendor profile covering F5 Distributed Cloud Bot Defense. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/f5 site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # F5: bot management vendor profile | Agent Ready Crawlers > F5: bot management vendor profile covering F5 Distributed Cloud Bot Defense. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/f5 Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # F5 In short F5 and Skyfire announced on 18 March 2026 that Bot Defense recognises agents carrying Skyfire's Know Your Agent tokens, which merchants can let through checkout, and policies can allow, challenge, rate-limit or block a risky session. F5 publishes no price, and its device intelligence features arrive in limited availability late in 2026. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [f5.com](https://www.f5.com) In one line Application delivery and security company whose Distributed Cloud Bot Defense puts humans, trusted AI agents and malicious bots under one policy, accepts Web Bot Auth and Skyfire's Know Your Agent tokens, and runs on F5's platform, BIG-IP or public cloud. Last read 2026-11-05 ## What F5 does for this layer ### From the vendor's own pages F5 and Skyfire announced on 18 Mar 2026 that Distributed Cloud Bot Defense recognises agents carrying Skyfire KYA identity tokens, which are standard JSON Web Tokens. ([f5.com](https://www.f5.com/company/news/press-releases/f5-skyfire-secure-agentic-commerce), dated 2026-03-18) F5 says merchants on Bot Defense can let Skyfire-verified agents complete standard checkout flows with Skyfire's tokenized payment credentials, with no re-platforming. ([f5.com](https://www.f5.com/company/news/press-releases/f5-skyfire-secure-agentic-commerce), dated 2026-03-18) On 15 Sep 2026 F5 added an agent-aware policy framework that classifies humans, trusted AI agents and malicious bots in a single policy, available that day. ([f5.com](https://www.f5.com/company/news/press-releases/bot-defense-device-intelligence-agentic-ai-detection), dated 2026-09-15) F5 lists allow, step-up challenge, rate-limit and block as the policy actions Bot Defense can apply to risky sessions. ([f5.com](https://www.f5.com/company/news/press-releases/bot-defense-device-intelligence-agentic-ai-detection), dated 2026-09-15) F5 deploys Distributed Cloud Bot Defense through its Distributed Cloud WAAP, as a BIG-IP native module or iApp, or in VMs or containers close to public cloud workloads. ([f5.com](https://www.f5.com/products/distributed-cloud-services/bot-defense)) F5 sends Bot Defense data and inferences to Syslog, SIEM systems or cloud buckets for security and fraud teams. ([f5.com](https://www.f5.com/products/distributed-cloud-services/bot-defense)) ### Claims the vendor makes about itself F5's Bot Defense page says it identifies and controls AI agents "based on behavior and intent, not static signatures or identity claims." ([f5.com](https://www.f5.com/products/distributed-cloud-services/bot-defense)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | F5 Distributed Cloud Bot Defense | Bot and agent detection with behavioral analysis, client-side telemetry, API and mobile protection. | sales-led Product page leads to sales contact. | Not published | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | F5 says Bot Defense supports Web Bot Auth and KYA. (f5.com) | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol statement on the F5 pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No Agent Pay statement on the F5 pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What F5 does not do - F5 publishes no price for Distributed Cloud Bot Defense. - F5's device intelligence capability was not generally available on 2026-11-05; F5 gave the fourth quarter of 2026 for limited availability. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [f5.com](https://www.f5.com) read 2026-11-05 - [F5 Distributed Cloud Bot Defense | F5](https://www.f5.com/products/distributed-cloud-services/bot-defense) read 2026-11-05 - [Managing AI agents with trust, control, and confidence | F5](https://www.f5.com/company/blog/managing-ai-agents-with-trust-control-and-confidence) read 2026-11-05 - [F5 and Skyfire partner to advance secure agentic commerce for the enterprise | F5](https://www.f5.com/company/news/press-releases/f5-skyfire-secure-agentic-commerce) read 2026-11-05 - [F5 Distributed Cloud Bot Defense adds device intelligence and agentic AI detection to guard against automated account abuse | F5](https://www.f5.com/company/news/press-releases/bot-defense-device-intelligence-agentic-ai-detection) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Fastly: bot management profile | Agent Ready Crawlers" description: "Fastly: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 10 of 15." canonical: https://agentreadycrawlers.com/vendors/fastly site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Fastly: bot management profile | Agent Ready Crawlers > Fastly: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 10 of 15. Source: https://agentreadycrawlers.com/vendors/fastly Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 10 of 15 # Fastly In short Fastly is an edge cloud whose WAF signals and Bot Management product cover agents only in part: verification, per-purpose policy, charging and commerce agents each grade partial, and reporting is not published, for 4 of 10 points. Fastly publishes no price for Bot Management, which an Essential-tier Next-Gen WAF customer cannot buy at all. Edge cloud vendor whose Bot Management adds signals for suspected and verified AI crawlers and fetchers, lets sites block, throttle, deceive or allow them, and can redirect them to a TollBit paywall. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Fastly](https://agentreadycrawlers.com/vendors/fastly/alternatives/) · [fastly.com](https://www.fastly.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [fastly.com](https://www.fastly.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Partial | verified-bot signals for AI crawlers and fetchers; Skyfire credentials at the edge (announced); Web Bot Auth not found in docs (checked 2026-11-04) | Signals listed in Fastly-TollBit blog. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | crawler versus fetcher signals, suspected versus verified | Four separate AI bot signals. Source, read 2026-11-04. | | Charging for access | Partial | through TollBit paywall redirect | Fastly-TollBit blog. Source, read 2026-11-04. | | Lets signed commerce agents through | Partial | Skyfire KYA/KYAPay partnership (Jun 2026); no Visa TAP or Mastercard Agent Pay found | Skyfire partnership release. Source, read 2026-11-04. | | Cost | Not graded | no public price; sales-led | Pricing page lists Bot Management as contact sales. Source, read 2026-11-04. | | Reporting | Not published | AI bot reporting not described on pages read (checked 2026-11-04) | Product and docs pages read. Source, read 2026-11-04. | | Where it runs | Not graded | edge: classification and actions on Fastly's edge network | AI Bot Management and TollBit flow. Source, read 2026-11-04. | ## Cost no public price for Bot Management or AI Bot Management; contact sales (checked 2026-11-04). Free tier covers up to 100,000 requests/month for delivery. [Source](https://www.fastly.com/pricing), read 2026-11-04. ## What Fastly does not do - Bot Management is not available on the Essential platform for Next-Gen WAF (Bot Management docs). - No native per-crawl billing; charging relies on TollBit (Fastly-TollBit blog). - No public price for bot products (pricing page). - Fastly does not sell Bot Management on its own; Full-Site Delivery or Streaming Delivery is a prerequisite. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Fastly Bot Management identifies bots and reduces unwanted bot activity; the Essential platform for Next-Gen WAF does not support Bot Management. [Source](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/), read 2026-11-04. - Primary source Bot Management deployments can optionally add client challenges and advanced client-side detections. [Source](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/), read 2026-11-04. - Vendor claim Fastly says AI Bot Management lets sites detect, block, deceive or allow AI bots, and allow, throttle, monitor or block requests. [Source](https://www.fastly.com/products/fastly-ai-bot-management), read 2026-11-04. - Primary source Fastly exposes separate signals SUSPECTED-BOT.AI-CRAWLER, SUSPECTED-BOT.AI-FETCHER, VERIFIED-BOT.AI-CRAWLER and VERIFIED-BOT.AI-FETCHER. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. - Primary source With the TollBit integration Fastly classifies an AI bot and redirects it to the publisher's TollBit Bot Paywall instead of blocking; bots with valid TollBit tokens get the content. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. - Vendor claim The AI Bot Management page mentions ContentGuard for pre-cache inspection. [Source](https://www.fastly.com/products/fastly-ai-bot-management), read 2026-11-04. - Primary source Bot Management, Next-Gen WAF and the Security Core packages are contact-sales only; the free tier covers up to 100,000 requests a month. [Source](https://www.fastly.com/pricing), read 2026-11-04. - Primary source On 10 Jun 2026 Fastly and Skyfire partnered to bring Skyfire's identity and payment-backed agent credentials to Fastly's edge. [Source](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce), read 2026-11-04, dated 2026-06-10. - Primary source On 21 Sep 2026 Fastly launched AI Firewall (prompt-injection filtering before models) and AI Runtime Control (routing, token spend, rate limits and budgets for model calls). [Source](https://www.fastly.com/press/press-releases/fastly-launches-ai-firewall-and-ai-runtime-control-to-secure-and-scale-ai), read 2026-11-04, dated 2026-09-21. - Vendor claim Fastly reported AI requests grew about 6.5 times faster than human traffic in the first months of 2026. [Source](https://www.businesswire.com/news/home/20260609376872/en/AI-Traffic-Grew-6.5x-Faster-Than-Human-Traffic-This-Year-Creating-New-Business-Challenges-and-Opportunities), read 2026-11-04, dated 2026-06-09. ## Dated news - 2026-09-21**Fastly launches AI Firewall and AI Runtime Control**. Fastly added AI Firewall, which inspects prompts in the request path to stop attacks such as prompt injection, and AI Runtime Control, which routes model calls and manages token spend, rate limits, budgets and failover. Fastly positions both next to Bot Management for agent traffic. [Source](https://www.fastly.com/press/press-releases/fastly-launches-ai-firewall-and-ai-runtime-control-to-secure-and-scale-ai). - 2026-10-14**Fastly named a Leader in 2026 IDC MarketScape for WAAP**. Fastly announced it was named a Leader in the 2026 IDC MarketScape for Worldwide Web Application and API Protection platforms, which covers bot management among other capabilities. [Source](https://www.fastly.com/press/press-releases/fastly-named-a-leader-in-2026-idc-marketscape-for-worldwide-waap-platforms). ## Compared with - [Fastly vs Cloudflare](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare/): Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. - [Fastly vs Akamai](https://agentreadycrawlers.com/compare/fastly-vs-akamai/): Both route AI bot charging to TollBit, and both partner with Skyfire (Fastly 2026-06-10, Akamai 2026-06-15); Akamai documents Web Bot Auth verification and Visa TAP, Fastly docs read did not. ## What Fastly does for this layer ### From the vendor's own pages Fastly's client fingerprinting uses JA3 and JA4 TLS fingerprints to identify client types. ([docs.fastly.com](https://docs.fastly.com/products/bot-management), dated 2026-10-13) Fastly's client challenges come in dynamic, interactive and non-interactive forms; dynamic challenges can use Private Access Tokens and non-interactive ones use a JavaScript proof-of-work. ([docs.fastly.com](https://docs.fastly.com/products/bot-management), dated 2026-10-13) Bot operators apply for Fastly's Verified Bot categories through a submission form on community.fastly.com. ([docs.fastly.com](https://docs.fastly.com/products/bot-management), dated 2026-10-13) Fastly's VCL variable fastly.bot.category.is\_verified reports whether a detected bot is a self-identified bot that Fastly has validated. ([fastly.com](https://www.fastly.com/documentation/reference/vcl/variables/miscellaneous/fastly-bot-category-is-verified/)) ### Claims the vendor makes about itself Fastly's AI Bot Management page offers to block, intercept or deceive verified or suspected AI bots that scrape content without consent. ([fastly.com](https://www.fastly.com/products/fastly-ai-bot-management)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot Management | ContentGuard edge detection, client fingerprinting, client challenges, advanced client-side detections and verified bots. | contact sales Requires Full-Site Delivery or Streaming Delivery. | Not published | 2026-11-05 | | Next-Gen WAF AI bot signals | SUSPECTED-BOT and VERIFIED-BOT signals for AI crawlers and AI fetchers usable in lists, rate limit rules and request rules. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The Bot Management product description (updated 2026-10-13) does not mention Web Bot Auth or HTTP message signatures. | | Visa Trusted Agent Protocol | No statement | No statement found in the Fastly docs read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement found in the Fastly docs read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/alternatives/) graded on the rubric ## Sources - [fastly.com: fastly.com](https://www.fastly.com) read 2026-11-04 - [fastly.com: pricing](https://www.fastly.com/pricing) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [investors.fastly.com: fastly and skyfire partner enable trusted agentic commerce](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce) read 2026-11-04 - [fastly.com: fastly ai bot management](https://www.fastly.com/products/fastly-ai-bot-management) read 2026-11-04 - [fastly.com: about bot management](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/) read 2026-11-04 - [fastly.com: fastly launches ai firewall and ai runtime control to secure and scale ai](https://www.fastly.com/press/press-releases/fastly-launches-ai-firewall-and-ai-runtime-control-to-secure-and-scale-ai) read 2026-11-04 - [businesswire.com: AI Traffic Grew 6.5x Faster Than Human Traffic This Year Creating New Business Challenges and Opportunities](https://www.businesswire.com/news/home/20260609376872/en/AI-Traffic-Grew-6.5x-Faster-Than-Human-Traffic-This-Year-Creating-New-Business-Challenges-and-Opportunities) read 2026-11-04 - [fastly.com: fastly named a leader in 2026 idc marketscape for worldwide waap platforms](https://www.fastly.com/press/press-releases/fastly-named-a-leader-in-2026-idc-marketscape-for-worldwide-waap-platforms) read 2026-11-04 - [New Bot Signals: SUSPECTED-BOT.AI-CRAWLER, VERIFIED-BOT.AI-CRAWLER, SUSPECTED-BOT.AI-FETCHER, and VERIFIED-BOT.AI-FETCHER | Fastly Documentation](https://www.fastly.com/documentation/reference/changes/2025/04/new-bot-signals-suspected-botai-crawler-verified-botai-crawler-suspected-botai-fetcher-and-verified-botai-fetcher/) read 2026-11-05 - [Using system signals | Fastly Documentation](https://www.fastly.com/documentation/guides/next-gen-waf/signals/using-system-signals/) read 2026-11-05 - [Bot Management | Fastly Products](https://docs.fastly.com/products/bot-management) read 2026-11-05 - [fastly.bot.category.is\_verified | Fastly Documentation](https://www.fastly.com/documentation/reference/vcl/variables/miscellaneous/fastly-bot-category-is-verified/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Fastly | Agent Ready Crawlers" description: "Alternatives to Fastly: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/fastly/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Fastly | Agent Ready Crawlers > Alternatives to Fastly: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/fastly/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Fastly In short Fastly ranks 10th of 15 on 4 of 10 points, with Arcjet the nearest on points, and head-to-head pages set it against Cloudflare and Akamai. Four questions grade partial for Fastly. It gives no price for Bot Management, which Essential-tier Next-Gen WAF customers cannot use. ## Head-to-head, with the sourced difference - [**Fastly vs Cloudflare**](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare/). Fastly exposes separate signals for suspected and verified AI crawlers and AI fetchers and redirects to a TollBit paywall; Cloudflare runs its own pay per crawl and per-purpose Search, Training and Agent controls. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. - [**Fastly vs Akamai**](https://agentreadycrawlers.com/compare/fastly-vs-akamai/). Both route AI bot charging to TollBit, and both partner with Skyfire (Fastly 2026-06-10, Akamai 2026-06-15); Akamai documents Web Bot Auth verification and Visa TAP, Fastly docs read did not. [Source](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Fastly's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Fastly on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, commerce agents, reporting | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy, charging, reporting | commerce agents | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, reporting | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, charging, reporting | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy, reporting | commerce agents | No public price; sales-led | | Skyfire | 6 | verifies agents, charging, commerce agents | per-purpose policy | Not published (checked 2026-11-04) | | TollBit | 6 | charging, reporting | commerce agents | Free for publishers to start | | Vercel | 6 | verifies agents, per-purpose policy, reporting | charging, commerce agents | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents, reporting | charging, commerce agents | No public price; sales-led | | Arcjet | 3 | reporting | charging, commerce agents | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | reporting | verifies agents, charging | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | reporting | per-purpose policy, charging, commerce agents | No public price; sales-led | | Stytch IsAgent | 2 | reporting | per-purpose policy, charging, commerce agents | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, charging, commerce agents | Platform plans from USD 0; extension price not stated | ## Where Fastly is not the fit - Bot Management is not available on the Essential platform for Next-Gen WAF (Bot Management docs). - No native per-crawl billing; charging relies on TollBit (Fastly-TollBit blog). - No public price for bot products (pricing page). ## Sources - [fastly.com: pricing](https://www.fastly.com/pricing) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [investors.fastly.com: fastly and skyfire partner enable trusted agentic commerce](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "FetchRight: licensing intermediary profile | Agent Ready Crawlers" description: "FetchRight: licensing intermediary profile covering Peek-Then-Pay and FetchRight platform. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/fetchright site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # FetchRight: licensing intermediary profile | Agent Ready Crawlers > FetchRight: licensing intermediary profile covering Peek-Then-Pay and FetchRight platform. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/fetchright Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Licensing intermediary # FetchRight In short In Peek-Then-Pay a publisher signals participation in /.well-known/peek.json, naming the licence server and terms, and an agent answers a priced preview by buying a JWT licence for one intent, such as quote or embed. Edge enforcers check those licences on their own, and FetchRight claims previews cut AI compute by over 90%. Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [fetchright.ai](https://fetchright.ai) In one line Licensing service behind the open Peek-Then-Pay protocol: a site publishes /.well-known/peek.json, AI agents get a priced HTTP 203 preview, buy a JWT license per intent and present it to an edge enforcer. Last read 2026-11-05 ## What FetchRight does for this layer ### From the vendor's own pages FetchRight describes Peek-Then-Pay as an open protocol that separates discovery from access, so AI agents can evaluate content through structured previews without unrestricted retrieval. ([fetchright.ai](https://fetchright.ai/)) Under Peek-Then-Pay a publisher signals participation with a manifest at /.well-known/peek.json that names the license server, the terms URL, the supported intents and rate limits per IP. ([peekthenpay.org](https://peekthenpay.org/)) Peek-Then-Pay answers an AI agent's content request with HTTP 203 Non-Authoritative Information carrying a publisher-written preview plus pricing. ([github.com](https://github.com/PeekThenPay/peek-json-spec)) The Peek-Then-Pay specification chooses HTTP 203 over 402 because, in its words, most AI systems process 203 responses normally "unlike 402 which is often blocked". ([github.com](https://github.com/PeekThenPay/peek-json-spec)) Peek-Then-Pay defines six standard intents, peek, read, summarize, quote, embed and analyze, and agents buy a JWT license for a specific intent from the license server in the manifest. ([peekthenpay.org](https://peekthenpay.org/)) Peek-Then-Pay edge enforcers validate assertion-only JWTs and manage budgets locally without calling the license server, and both sides report usage asynchronously for billing and disputes. ([github.com](https://github.com/PeekThenPay/peek-json-spec)) The example peek.json on peekthenpay.org names https://api.fetchright.ai/v1/license as license issuer and lists /robots.txt and /sitemap.xml as paths that bypass enforcement. ([peekthenpay.org](https://peekthenpay.org/)) The peek.json specification is dedicated to the public domain under CC0 1.0 and published as the npm package @peekthenpay/peek-json-spec. ([github.com](https://github.com/PeekThenPay/peek-json-spec)) Peek-Then-Pay's sample pricing schema prices intents per request or per 1,000 tokens in US cents, and marks train and index uses as requiring a contract. ([peekthenpay.org](https://peekthenpay.org/)) ### Claims the vendor makes about itself FetchRight claims that structured peek responses cut AI compute by more than 90% compared with fetching raw HTML. ([fetchright.ai](https://fetchright.ai/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Peek-Then-Pay | Open protocol (CC0) for manifest discovery, HTTP 203 previews, intent-scoped JWT licenses and edge enforcement. | open specification CC0 1.0 | USD 0 | 2026-11-05 | | FetchRight platform | Peek and Pay services for publishers, affiliates, partner networks and brands; acts as a license server. | sales-led homepage offers "Start a Conversation" only | Not published | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | Peek-Then-Pay keeps /robots.txt reachable in its example manifest but does not build on robots.txt directives (checked 2026-11-05). | | Web Bot Auth | No statement | Agents prove entitlement with license JWTs; the spec page does not reference Web Bot Auth (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What FetchRight does not do - Peek-Then-Pay does not stop an agent that never reads peek.json; enforcement depends on the publisher running an edge enforcer. - FetchRight publishes no price list or fee for its license server (homepage read 2026-11-05). ## Reading notes peekthenpay.org links fetchright.ai as the company behind the protocol, and its example manifest names api.fetchright.ai as the licence issuer. ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [fetchright.ai](https://fetchright.ai) read 2026-11-05 - [FetchRight: Controlled Participation Across AI Systems](https://fetchright.ai/) read 2026-11-05 - [Peek-Then-Pay: The Open Standard for AI Content Discovery and Use](https://peekthenpay.org/) read 2026-11-05 - [PeekThenPay/peek-json-spec on GitHub](https://github.com/PeekThenPay/peek-json-spec) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Fingerprint: bot management vendor profile | Agent Ready Crawlers" description: "Fingerprint: bot management vendor profile covering Bot Detection (Smart Signal) and Authorized AI Agent Detection. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/fingerprint site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Fingerprint: bot management vendor profile | Agent Ready Crawlers > Fingerprint: bot management vendor profile covering Bot Detection (Smart Signal) and Authorized AI Agent Detection. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/fingerprint Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Fingerprint In short Fingerprint's agent verification relies on signing providers: it says agents from five named providers, OpenAI among them, can be checked cryptographically, and it runs a public tester for RFC 9421 signatures. A site can pick a different treatment for agent traffic, from allowing it to throttling or blocking it. Bot signals are available only on Pro Plus and above. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [fingerprint.com](https://fingerprint.com) In one line Device intelligence vendor whose Bot Detection and Authorized AI Agent Detection signals separate good bots, bad bots and AI agents, and verify signed agents from providers such as OpenAI, AWS AgentCore and Browserbase. Last read 2026-11-05 ## What Fingerprint does for this layer ### From the vendor's own pages Fingerprint says AI agents from OpenAI, AWS AgentCore, Browserbase, Manus and Anchor Browser can be cryptographically verified through its signed agent ecosystem. ([fingerprint.com](https://fingerprint.com/blog/product-update-ai-agent-detection/)) Fingerprint's bot detection response marks a bot as good when it is a search engine or friendly crawler, and separates good bots, bad bots and AI agents. ([fingerprint.com](https://fingerprint.com/products/bot-detection/)) Fingerprint hosts a Web Bot Authentication tester that verifies RFC 9421 HTTP Message Signatures from a browser or with a curl example. ([fingerprint.com](https://fingerprint.com/products/bot-detection/)) Fingerprint's browser bot detection and other Smart Signals require the Pro Plus or Enterprise plan. ([fingerprint.com](https://fingerprint.com/products/bot-detection/)) Fingerprint's AI agent detection page shows a signed request with Signature-Agent "https://openai.com" and offers Allow, Enrich, Throttle and Block handling for agent traffic. ([fingerprint.com](https://fingerprint.com/products/ai-agent-detection/)) Fingerprint Pro Plus starts at USD 99 a month for 20,000 API calls, then USD 4 per 1,000 calls; the free plan covers up to 1,000 API calls a month. ([fingerprint.com](https://fingerprint.com/pricing/)) ### Claims the vendor makes about itself Fingerprint claims Authorized AI Agent Detection identifies authorized agents with 100% certainty. ([fingerprint.com](https://fingerprint.com/blog/product-update-ai-agent-detection/)) Fingerprint says it is trusted by more than 6,000 companies. ([fingerprint.com](https://fingerprint.com/pricing/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot Detection (Smart Signal) | Client-side detection of good bots, bad bots and AI agents through the Fingerprint JavaScript agent. | monthly plan plus usage Pro Plus, 20K API calls included, then USD 4 per 1,000. | USD 99 | 2026-11-05 | | Authorized AI Agent Detection | Verification of agents signed by participating providers. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Fingerprint verifies RFC 9421 signatures and offers a Web Bot Auth tester. (fingerprint.com) | | Visa Trusted Agent Protocol | No statement | No statement on the Fingerprint pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the Fingerprint pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Fingerprint does not do - Fingerprint does not block traffic at the CDN by itself; it returns signals from its JavaScript agent and API, with proxy integrations listed for Enterprise. - Fingerprint offers no AI crawler charging or licensing. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token - [Browserbase cloud browsers](https://agentreadycrawlers.com/crawlers/browserbase/) no token - [Amazon Bedrock AgentCore Browser](https://agentreadycrawlers.com/crawlers/agentcore-browser/) no token - [Manus cloud browser](https://agentreadycrawlers.com/crawlers/manus/) no token - [Anchor Browser](https://agentreadycrawlers.com/crawlers/anchor-browser/) no token ## Sources - [fingerprint.com](https://fingerprint.com) read 2026-11-05 - [Identifying Authorized AI Agents vs. Bots on the Web](https://fingerprint.com/blog/product-update-ai-agent-detection/) read 2026-11-05 - [Browser Bot Detection Software | Fingerprint](https://fingerprint.com/products/bot-detection/) read 2026-11-05 - [Detect AI Agents with 100% Accuracy | Fingerprint](https://fingerprint.com/products/ai-agent-detection/) read 2026-11-05 - [Pricing & Plans | Fingerprint](https://fingerprint.com/pricing/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Firecrawl: crawler operator profile | Agent Ready Crawlers" description: "Firecrawl: crawler operator profile covering Firecrawl API (scrape, crawl, map, search). Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/firecrawl site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Firecrawl: crawler operator profile | Agent Ready Crawlers > Firecrawl: crawler operator profile covering Firecrawl API (scrape, crawl, map, search). Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/firecrawl Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Firecrawl In short Firecrawl turns pages into Markdown or JSON for AI applications, sending every request through proxies by default: basic ones in 26 countries, with enhanced proxies tried when those fail. The one address it publishes belongs to its API server, not its scrapers, and plans run from a free 1,000 credits a month to USD 83 for Standard billed annually. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [firecrawl.dev](https://www.firecrawl.dev) In one line Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends. Last read 2026-11-05 ## What Firecrawl does for this layer ### From the vendor's own pages Firecrawl reads robots.txt before it crawls a site and applies both the rules written for the FirecrawlAgent token and the rules for the \* group. ([docs.firecrawl.dev](https://docs.firecrawl.dev/advanced-scraping-guide)) Firecrawl's docs call FirecrawlAgent the token it matches in robots.txt, say not to assume it is also the User-Agent header on the fetch, and advise allow-listing Firecrawl with a robots.txt directive rather than a WAF rule on User-Agent. ([docs.firecrawl.dev](https://docs.firecrawl.dev/advanced-scraping-guide)) Firecrawl publishes one IP address, 35.245.250.27, which its docs describe as the API server address that customers' own firewalls must allow to reach api.firecrawl.dev. ([docs.firecrawl.dev](https://docs.firecrawl.dev/advanced-scraping-guide)) Firecrawl's proxy docs say all requests go through proxies by default, with US proxies when no location is set, basic proxies in 26 countries and enhanced proxies in the Netherlands and the United States. ([docs.firecrawl.dev](https://docs.firecrawl.dev/features/proxies.md)) Firecrawl's default proxy mode is auto, which tries basic proxies first and retries with enhanced proxies on failure; the docs mark the proxy parameter as deprecated. ([docs.firecrawl.dev](https://docs.firecrawl.dev/features/enhanced-mode.md)) Firecrawl's pricing page lists a Free plan of 1,000 credits a month and Hobby at USD 19 a month (USD 16 billed annually) for 5,000 credits, where one credit is one page on a basic scrape or crawl. ([firecrawl.dev](https://www.firecrawl.dev/pricing)) ### Claims the vendor makes about itself Firecrawl's pricing page carries a banner announcing "Alexandria" and a USD 75M Series B. ([firecrawl.dev](https://www.firecrawl.dev/pricing)) ### What our crawl saw GETs for the Web Bot Auth key directory at www.firecrawl.dev and api.firecrawl.dev both returned HTTP 404 on 2026-11-05. ([firecrawl.dev](https://www.firecrawl.dev/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Firecrawl API (scrape, crawl, map, search) | Turns pages into Markdown or JSON for AI applications. | credit subscription Hobby, per month billed annually; Free plan 1,000 credits a month. | USD 16 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Reads robots.txt before crawling and applies FirecrawlAgent and * rules. (docs.firecrawl.dev) | | Web Bot Auth | No statement | No signing feature documented; key directory 404 (2026-11-05). | | Published IP list | No statement | Only the API server IP is published, not crawler egress addresses. (docs.firecrawl.dev) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404 | GET https://www.firecrawl.dev/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, application/json | GET https://api.firecrawl.dev/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Firecrawl does not do - Firecrawl does not promise a fixed User-Agent header that a WAF rule can match, by its own documentation. - Firecrawl publishes no egress IP list or signature keys for its scraping traffic (checked 2026-11-05). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [FirecrawlAgent](https://agentreadycrawlers.com/crawlers/firecrawl/) token FirecrawlAgent ## Sources - [firecrawl.dev](https://www.firecrawl.dev) read 2026-11-05 - [Advanced Scraping Guide | Firecrawl](https://docs.firecrawl.dev/advanced-scraping-guide) read 2026-11-05 - [Proxies | Firecrawl docs](https://docs.firecrawl.dev/features/proxies.md) read 2026-11-05 - [Enhanced Mode | Firecrawl docs](https://docs.firecrawl.dev/features/enhanced-mode.md) read 2026-11-05 - [Pricing | Firecrawl](https://www.firecrawl.dev/pricing) read 2026-11-05 - [Firecrawl key directory probe](https://www.firecrawl.dev/.well-known/http-message-signatures-directory) read 2026-11-05 - [Firecrawl API key directory probe](https://api.firecrawl.dev/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Google: crawler operator profile | Agent Ready Crawlers" description: "Google: crawler operator profile covering Googlebot, Google-Extended, Google-Agent and 3 more. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/google site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Google: crawler operator profile | Agent Ready Crawlers > Google: crawler operator profile covering Googlebot, Google-Extended, Google-Agent and 3 more. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/google Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Google In short Google separates its fetchers into three classes, each with a reverse DNS mask and a JSON range file, five files in all. A request never carries Google-Extended, which exists purely as a control word in robots.txt for Gemini training and grounding. Google-Agent traffic is partly signed with Web Bot Auth, an experiment described on 13 October 2026. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [google.com](https://www.google.com) In one line Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic. Last read 2026-11-05 ## What Google does for this layer ### From the vendor's own pages Google says its common crawlers, Googlebot among them, always obey robots.txt rules when crawling automatically. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)) Google-Extended has no HTTP user agent of its own and works only as a robots.txt control token for Gemini model training and grounding; Google says it does not affect inclusion or ranking in Google Search. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)) Google states it intends to support the AIPREF shared training preference signal, written "user-agent: \_\_usage\_ai-training", in early 2027. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), dated 2026-10-30) Google's verification page sorts its requesters into common crawlers, special-case crawlers and user-triggered fetchers, each with its own reverse DNS mask and IP file. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)) Google says special-case crawlers such as AdsBot may or may not respect robots.txt, and that user-triggered fetchers ignore robots.txt because a user asked for the fetch. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)) Google calls its Web Bot Auth use experimental, signs only a subset of Google-Agent requests under the identity https://agent.bot.goog, and tells sites to keep IP and reverse DNS checks as a fallback. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth), dated 2026-10-13) Signed Google-Agent requests carry a Signature-Agent header set to g="https://agent.bot.goog", with the Signature and Signature-Input headers labelled g. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)) Google-Agent is the user agent for agents hosted on Google infrastructure that browse and act on user request, and it uses the ranges in user-triggered-agents.json. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)) Google's Gemini Notebook fetcher now sends the token Google-GeminiNotebook; the former Google-NotebookLM agent was supported until August 2026. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)) ### What our crawl saw Google's common-crawlers.json listed 317 prefixes, 170 IPv4 and 147 IPv6, with a creationTime of 2026-11-04 when fetched on 2026-11-05. ([developers.google.com](https://developers.google.com/static/crawling/ipranges/common-crawlers.json), dated 2026-11-04) Google's user-triggered-agents.json, the range file for Google-Agent, listed 20 prefixes (12 IPv4, 8 IPv6) on 2026-11-05. ([developers.google.com](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json), dated 2026-11-04) agent.bot.goog served a key directory with five Ed25519 keys and the content type application/http-message-signatures-directory+json on 2026-11-05. ([agent.bot.goog](https://agent.bot.goog/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Googlebot | Common crawler for Google Search, Images, Video, News and Discover. | No price recorded | | Google-Extended | Robots.txt token, with no user agent string, that controls Gemini training and grounding use. | No price recorded | | Google-Agent | User agent for Google-hosted agents acting on user request; some requests signed with Web Bot Auth. | No price recorded | | Google-CloudVertexBot | Crawler for site-owner-requested crawls that build Vertex AI Agents. | No price recorded | | Storebot-Google | Crawler for Google Shopping surfaces. | No price recorded | | Google-GeminiNotebook | User-triggered fetcher for sources added to Gemini Notebook projects. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | Common crawlers always obey robots.txt for automatic crawls; special-case crawlers may not; user-triggered fetchers ignore it. (developers.google.com) | | Web Bot Auth | Partial | Experimental; a subset of Google-Agent requests signed as https://agent.bot.goog. (developers.google.com) | | Content Signals | No statement | Google names the AIPREF signal for early 2027; Content-Signals is not mentioned (read 2026-11-05). | | Published IP list | Supports | Five published range files, including user-triggered-agents.json for Google-Agent. (developers.google.com) | | Reverse DNS check | Supports | Masks under googlebot.com, google.com and googleusercontent.com with a forward-confirm step. (developers.google.com) | | Cloudflare verified bots | No statement | Google's crawler pages do not mention Cloudflare's list (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | agent.bot.goog: 200, application/http-message-signatures-directory+json, 5 keys | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | common-crawlers.json | 317 prefixes (170 IPv4, 147 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 | | special-crawlers.json | 272 prefixes (136 IPv4, 136 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 | | user-triggered-agents.json | 20 prefixes (12 IPv4, 8 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 | | user-triggered-fetchers-google.json | 504 prefixes (252 IPv4, 252 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 | ## What Google does not do - Google does not sign every Google-Agent request, so a missing signature does not prove a request is fake. - Google-Extended cannot be blocked at the server by user agent, because Google sends no Google-Extended string in requests. - Google's common crawler and user-triggered fetcher pages list no token for Gemini auto browse in Chrome (read 2026-11-05). ## Reading notes Google's robots.txt disallows our crawl token for www.google.com/.well-known/, so that path was not probed for a key directory; the Web Bot Auth directory on agent.bot.goog was read instead. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Google in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/google/) 8 rows - [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) token Googlebot - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) no token ## Sources - [google.com](https://www.google.com) read 2026-11-05 - [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) read 2026-11-05 - [Verify Requests from Google Crawlers and Fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-05 - [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) read 2026-11-05 - [Google User-Triggered Fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-05 - [Google common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json) read 2026-11-05 - [Google user-triggered-agents.json](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json) read 2026-11-05 - [agent.bot.goog Web Bot Auth key directory path (GET probe)](https://agent.bot.goog/.well-known/http-message-signatures-directory) read 2026-11-05 - [Google special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json) read 2026-11-05 - [Google user-triggered-fetchers-google.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Google Cloud Fraud Defense: vendor profile | Agent Ready Crawlers" description: "Google Cloud Fraud Defense: bot management vendor profile covering Google Cloud Fraud Defense (reCAPTCHA) and Google Cloud Armor. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/google-cloud-fraud-defense site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Google Cloud Fraud Defense: vendor profile | Agent Ready Crawlers > Google Cloud Fraud Defense: bot management vendor profile covering Google Cloud Fraud Defense (reCAPTCHA) and Google Cloud Armor. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/google-cloud-fraud-defense Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Google Cloud Fraud Defense In short Google Cloud Fraud Defense reports verified and suspected agent sessions side by side, treating as suspected any session that declares itself through headers or behaves like an agent without being verifiable. It plugs into Cloud Armor, Fastly and Cloudflare at the WAF layer, deciding by signature where one exists or else by matching the user agent to published addresses. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Operator Google Homepage [cloud.google.com](https://cloud.google.com/security/products/recaptcha) In one line Google Cloud's bot and fraud service, the evolution of reCAPTCHA, which verifies AI agents by Web Bot Auth signature or by user agent matched to published IP ranges or reverse DNS, and reports verified and suspected agent sessions. Last read 2026-11-05 ## What Google Cloud Fraud Defense does for this layer ### From the vendor's own pages Google Cloud Fraud Defense classifies agent traffic with two verification methods: Web Bot Auth signatures, and the User-Agent string checked against the source IP address. ([docs.cloud.google.com](https://docs.cloud.google.com/recaptcha/docs/monitor-agents), dated 2026-10-26) Google counts sessions that self-declare as agents, for example through HTTP headers, or behave like agents but cannot be verified, as suspected agent sessions. ([docs.cloud.google.com](https://docs.cloud.google.com/recaptcha/docs/monitor-agents), dated 2026-10-26) Google's Fraud Defense post calls Web Bot Auth its preferred standard for agents and bot operators, and says Fraud Defense also validates declared user agents against published egress IP blocks and reverse DNS records. ([security.googlecloudcommunity.com](https://security.googlecloudcommunity.com/community-blog-42/how-google-cloud-fraud-defense-recaptcha-identifies-agents-and-bots-and-why-builders-should-sign-their-traffic-8449)) reCAPTCHA in Fraud Defense is free up to 10,000 assessments a month per organization, then a USD 8 flat fee up to 100,000, then USD 1 per 1,000 assessments. ([cloud.google.com](https://cloud.google.com/security/products/recaptcha)) Google's reCAPTCHA page says it integrates at the WAF layer with Google Cloud Armor, Fastly and Cloudflare, and combines with Cloud Armor and Apigee in Google's WAAP offering. ([cloud.google.com](https://cloud.google.com/security/products/recaptcha)) ### Claims the vendor makes about itself Google says an operator that signs with Web Bot Auth builds a reputation that extends across the 14 million+ websites protected by Fraud Defense. ([security.googlecloudcommunity.com](https://security.googlecloudcommunity.com/community-blog-42/how-google-cloud-fraud-defense-recaptcha-identifies-agents-and-bots-and-why-builders-should-sign-their-traffic-8449)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Google Cloud Fraud Defense (reCAPTCHA) | Bot and agent assessments with an Agent overview dashboard and policy controls. | per assessment Free to 10,000 assessments a month; USD 8 flat to 100,000; then USD 1 per 1,000. | USD 0 | 2026-11-05 | | Google Cloud Armor | WAF that can call reCAPTCHA for bot management at the edge. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Fraud Defense verifies Web Bot Auth signatures and calls it the preferred standard. (security.googlecloudcommunity.com) | | Published IP list | Supports | Declared user agents are checked against published egress IP blocks. (security.googlecloudcommunity.com) | | Reverse DNS check | Supports | Declared user agents are also checked against reverse DNS records. (security.googlecloudcommunity.com) | | Visa Trusted Agent Protocol | No statement | No statement on the Fraud Defense pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the Fraud Defense pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Google Cloud Fraud Defense does not do - Fraud Defense's Agent overview only covers the last 30 days or the current month. - Fraud Defense offers no charging or licensing route for AI crawlers. ## Reading notes Google also runs crawlers, covered on [Google's own profile](https://agentreadycrawlers.com/vendors/google/); this one covers only the bot-management product. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) no token ## Sources - [reCAPTCHA website security and fraud protection | Google Cloud](https://cloud.google.com/security/products/recaptcha) read 2026-11-05 - [Monitor agent traffic  |  Google Cloud Fraud Defense  |  Google Cloud Documentation](https://docs.cloud.google.com/recaptcha/docs/monitor-agents) read 2026-11-05 - [How Google Cloud Fraud Defense (reCAPTCHA) Identifies Agents and Bots-And Why Builders Should Sign Their Traffic | Community](https://security.googlecloudcommunity.com/community-blog-42/how-google-cloud-fraud-defense-recaptcha-identifies-agents-and-bots-and-why-builders-should-sign-their-traffic-8449) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "hCaptcha: bot management vendor profile | Agent Ready Crawlers" description: "hCaptcha: bot management vendor profile covering hCaptcha Enterprise with Agent Controls and hCaptcha Pro. Sources, standards and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/hcaptcha site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # hCaptcha: bot management vendor profile | Agent Ready Crawlers > hCaptcha: bot management vendor profile covering hCaptcha Enterprise with Agent Controls and hCaptcha Pro. Sources, standards and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/hcaptcha Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # hCaptcha In short hCaptcha limits agent identification to its Enterprise tier, whose rules can let an assistant browse a catalogue but demand verification before it reserves inventory, and suggests running such rules in logging mode before enforcing them. Enterprise carries no listed price, whereas the lighter Pro tier is billed monthly with a quota of evaluations. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Operator Intuition Machines, Inc. Homepage [hcaptcha.com](https://www.hcaptcha.com) In one line Bot detection and CAPTCHA vendor whose Enterprise Agent Controls identify AI agents, verify partner platforms that sign with Web Bot Auth, and let sites allow browsing while requiring verification before an agent reserves inventory. Last read 2026-11-05 ## What hCaptcha does for this layer ### From the vendor's own pages hCaptcha's Agent Controls page says hCaptcha Enterprise identifies agents even when they try to hide from detection, and supports Web Bot Auth verification for partner platforms that sign their requests. ([hcaptcha.com](https://www.hcaptcha.com/agent-controls)) hCaptcha's example agent policy lets an assistant browse a catalog, requires verification before it reserves inventory, and denies requests that break policy. ([hcaptcha.com](https://www.hcaptcha.com/agent-controls)) hCaptcha advises starting agent rules in logging mode, targeted by sitekey, before applying a response. ([hcaptcha.com](https://www.hcaptcha.com/agent-controls)) hCaptcha Pro costs USD 139 a month billed monthly with 100,000 evaluations included, then USD 0.99 per 1,000; the Basic plan is free. ([hcaptcha.com](https://www.hcaptcha.com/pricing)) hCaptcha describes a Zero-PII architecture and lists compliance with GDPR, CCPA, LGPD and PIPL. ([hcaptcha.com](https://www.hcaptcha.com/)) hCaptcha is a registered trademark of Intuition Machines, Inc., per the site footer. ([hcaptcha.com](https://www.hcaptcha.com/)) ### Claims the vendor makes about itself hCaptcha says its Advanced Threat Signatures cluster attackers across thousands of IPs and devices while grouping good traffic into one or a few signatures. ([hcaptcha.com](https://www.hcaptcha.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | hCaptcha Enterprise with Agent Controls | Agent identification, Web Bot Auth verification for signing partners, and a rules engine per sitekey. | contact sales Enterprise price not listed. | Not published | 2026-11-05 | | hCaptcha Pro | Low-friction CAPTCHA with passive mode. | monthly plus usage 100K evaluations included, then USD 0.99 per 1,000. | USD 139 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Agent Controls support Web Bot Auth verification for partner platforms that sign. (hcaptcha.com) | | Visa Trusted Agent Protocol | No statement | No statement on the hCaptcha pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the hCaptcha pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What hCaptcha does not do - hCaptcha describes agent identification for hCaptcha Enterprise; the free Basic plan is a CAPTCHA service without Agent Controls listed. - hCaptcha offers no charging or licensing route for AI crawlers. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [hcaptcha.com](https://www.hcaptcha.com) read 2026-11-05 - [Agent Controls: Detect and Manage Agents | hCaptcha](https://www.hcaptcha.com/agent-controls) read 2026-11-05 - [hCaptcha Pricing | Pro & Enterprise Plans](https://www.hcaptcha.com/pricing) read 2026-11-05 - [hCaptcha: Real-Time Bot Detection & Fraud Prevention](https://www.hcaptcha.com/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Hive: crawler operator profile | Agent Ready Crawlers" description: "Hive: crawler operator profile covering ImageSift reverse image search and Deploy inference platform. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/hive-imagesift site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Hive: crawler operator profile | Agent Ready Crawlers > Hive: crawler operator profile covering ImageSift reverse image search and Deploy inference platform. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/hive-imagesift Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Hive In short Hive changed its name to Deploy on 26 October 2026 and now sells hosted AI models. Its ImageSift unit crawls public images for a reverse image search index, keeping each picture's alt text along with the surrounding text, and accepts opt-outs by email only; no IP list or signature keys exist to confirm a visit. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [deploy.com](https://deploy.com) In one line AI model company, renamed from Hive to Deploy in October 2026, whose ImageSift unit runs ImagesiftBot to collect public images and page text for a reverse image search index. Last read 2026-11-05 ## What Hive does for this layer ### From the vendor's own pages ImageSift's about page says ImageSiftBot scrapes the internet for publicly available images to support Hive's web intelligence products and sends the user agent "Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)". ([imagesift.com](https://imagesift.com/about)) ImageSift respects robots.txt groups addressed to ImagesiftBot, and where no group names ImagesiftBot but one names Googlebot, ImagesiftBot follows the Googlebot rules instead of the \* group. ([imagesift.com](https://imagesift.com/about)) ImagesiftBot reads Crawl-delay as the minimum seconds between the starts of consecutive requests, splitting each day into intervals of that length and sending at most one request per interval. ([imagesift.com](https://imagesift.com/about)) Along with each image, ImageSiftBot saves the host URL, the text on the page and the image's alt text, which go into an index used for similar-image search. ([imagesift.com](https://imagesift.com/about)) ImageSift's page offers opt-out by email to ImageSift support and names no form, IP list or verification method. ([imagesift.com](https://imagesift.com/about)) imagesift.com itself serves a reverse image search box that takes an uploaded file or an image URL. ([imagesift.com](https://imagesift.com/)) Hive announced on 2026-10-26 that it is now called Deploy and is broadening into an enterprise AI inference platform serving open-source and proprietary models. ([thehive.ai](https://thehive.ai/blog/hive-is-now-deploy), dated 2026-10-26) Deploy's pricing page offers pay-as-you-go billing for developers and custom Enterprise pricing, with hosted models such as DeepSeek-V4.1-Flash-1017 at USD 0.10 per million input tokens. ([deploy.com](https://deploy.com/pricing)) ### Claims the vendor makes about itself Hive's rename post says its proprietary portfolio has over 20 models, which grew from content moderation into detecting IP-protected and AI-generated content. ([thehive.ai](https://thehive.ai/blog/hive-is-now-deploy), dated 2026-10-26) Hive says hundreds of organisations send billions of pieces of content a day through its APIs and that it operates its own datacenter infrastructure. ([thehive.ai](https://thehive.ai/blog/hive-is-now-deploy), dated 2026-10-26) ### What our crawl saw A GET for imagesift.com/.well-known/http-message-signatures-directory on 2026-11-05 returned HTTP 200 with an HTML page shell, not a key directory, so ImagesiftBot has no Web Bot Auth keys at that host. ([imagesift.com](https://imagesift.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | ImageSift reverse image search | Search for similar images across the index ImagesiftBot builds. | No price recorded | | Deploy inference platform | Hosted open-source and proprietary models, including Hive's moderation and AI-content detection models. | usage-based Per million input tokens for DeepSeek-V4.1-Flash-1017; Enterprise is custom. | USD 0.10 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Directives addressed to ImagesiftBot are respected, Crawl-delay included; Googlebot rules apply when ImagesiftBot is not named. (imagesift.com) | | Web Bot Auth | No statement | No key directory at imagesift.com (2026-11-05). | | Published IP list | No statement | No IP list on the ImageSift page (2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, text/html page shell, no keys | GET https://imagesift.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Hive does not do - ImageSift publishes no IP list or signature keys, so ImagesiftBot can be identified only by its user agent string (checked 2026-11-05). - ImageSift's page does not say whether images it collects are used to train Hive's or Deploy's models. ## Reading notes A site that allows Googlebot and blocks everything else through the \* group still admits [ImagesiftBot](https://agentreadycrawlers.com/crawlers/imagesiftbot/), because ImageSift applies its Googlebot rule when no group names its bot. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [ImagesiftBot](https://agentreadycrawlers.com/crawlers/imagesiftbot/) token ImagesiftBot ## Sources - [deploy.com](https://deploy.com) read 2026-11-05 - [Imagesift by Hive (ImageSift Bot)](https://imagesift.com/about) read 2026-11-05 - [Imagesift by Hive](https://imagesift.com/) read 2026-11-05 - [Hive is now Deploy](https://thehive.ai/blog/hive-is-now-deploy) read 2026-11-05 - [Pricing | Deploy](https://deploy.com/pricing) read 2026-11-05 - [ImageSift key directory probe](https://imagesift.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Huawei: crawler operator profile | Agent Ready Crawlers" description: "Huawei: crawler operator profile covering PetalBot and Verify PetalBot. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/huawei site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Huawei: crawler operator profile | Agent Ready Crawlers > Huawei: crawler operator profile covering PetalBot and Verify PetalBot. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/huawei Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Huawei In short A site that blocks PetalBot drops out of Petal search entirely, and pages already indexed can linger for several months after the visits stop. The crawl rate follows server capacity, site quality and how often a site changes, and Huawei's webmaster site offers a lookup tool for confirming an address, since no range file is published. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [huawei.com](https://www.huawei.com) In one line Huawei runs PetalBot for the Petal search engine, whose index also feeds Huawei Assistant and AI Search; PetalBot follows robots.txt and is verified by reverse DNS under aspiegel.com or an IP lookup tool. Last read 2026-11-05 ## What Huawei does for this layer ### From the vendor's own pages Huawei says PetalBot indexes PC and mobile sites for the Petal search engine and for content recommendations in Huawei Assistant and AI Search services. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) Huawei publishes two PetalBot strings: a desktop string, Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot), and an Android mobile string with the same PetalBot token. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) Huawei says PetalBot complies with the robots protocol and that blocking it makes a site's pages unsearchable in the Petal search engine. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) Huawei tells site owners to verify PetalBot by reverse DNS, where genuine hostnames follow the pattern petalbot.\*.aspiegel.com. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) Huawei says PetalBot adjusts its crawl volume according to server capacity, website quality and website updates. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) Huawei says that after PetalBot stops visiting a site, clearing pages already in the Petal index may take several months. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) The Petal webmaster site offers a Verify PetalBot tool that checks whether a given IP address is a verified PetalBot address. ([webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | PetalBot | Crawler for Petal Search, Huawei Assistant and AI Search. | No price recorded | | Verify PetalBot | Lookup tool on the Petal webmaster site that checks an IP address. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | PetalBot complies with the robots protocol. (webmaster.petalsearch.com) | | Web Bot Auth | No statement | Not on the PetalBot page (read 2026-11-05). | | Published IP list | No statement | No range file published; an IP lookup tool instead (read 2026-11-05). | | Reverse DNS check | Supports | Hostnames petalbot.*.aspiegel.com. (webmaster.petalsearch.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Huawei does not do - Huawei publishes no PetalBot IP range file; checks run through reverse DNS or the webmaster site's lookup tool. - Huawei's PetalBot page names no separate token for AI uses, and blocking PetalBot removes a site from Petal search. ## Reading notes Huawei's PetalBot page names aspiegel.com as the reverse DNS domain, while the [PetalBot row](https://agentreadycrawlers.com/crawlers/petalbot/) in the Crawler Index, read on 4 November 2026, gave petalsearch.com. The vendor's own page is the primary source. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [PetalBot](https://agentreadycrawlers.com/crawlers/petalbot/) token PetalBot ## Sources - [huawei.com](https://www.huawei.com) read 2026-11-05 - [Petal Search webmaster site: PetalBot](https://webmaster.petalsearch.com/site/petalbot) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "HUMAN Security: bot management profile | Agent Ready Crawlers" description: "HUMAN Security: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources." canonical: https://agentreadycrawlers.com/vendors/human-security site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # HUMAN Security: bot management profile | Agent Ready Crawlers > HUMAN Security: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Source: https://agentreadycrawlers.com/vendors/human-security Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 5 of 15 # HUMAN Security In short HUMAN Security sells bot and agent controls that grade yes on agent verification, per-purpose policy and reporting and partial on charging, for 7 of 10 points. Cryptographic verification is missing at the medium trust level, which HUMAN itself flags as a source of false negatives. The price is not public. Bot and fraud defense vendor whose AgenticTrust module rates AI agents High, Medium or Low trust and lets sites allow or deny agent actions such as browsing, login, account creation and checkout. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/alternatives/) · [humansecurity.com](https://www.humansecurity.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [humansecurity.com](https://www.humansecurity.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | per-request cryptographic keys (e.g. ChatGPT's published key), plus behavior and private signals; Web Bot Auth not named on pages read | High Trust requires verification via cryptographic key on each request. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Yes | per-action categories (content, engage, account creation, checkout, rate limit) by trust level | Category permissions listed in Trust Levels blog. Source, read 2026-11-04. | | Charging for access | Partial | through scraping monetization partners (TollBit logo on product page); no native charging found | Ecosystem section of AgenticTrust page. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned on AgenticTrust page or Trust Levels blog. Source, read 2026-11-04. | | Cost | Not graded | no public price; sales-led | Package plans page shows no prices. Source, read 2026-11-04. | | Reporting | Yes | user and session insights on agent actions, monthly State of Agentic Traffic benchmark | Visibility section of AgenticTrust page. Source, read 2026-11-04. | | Where it runs | Not graded | platform: Sightline integrates with the customer's stack; exact deployment points not confirmed on pages read | AgenticTrust page describes integration into HUMAN's broader platform. Source, read 2026-11-04. | ## Cost no public price; sales-led (checked 2026-11-04) [Source](https://www.humansecurity.com/platform/package-plans), read 2026-11-04. ## What HUMAN Security does not do - Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02). - No native pay-per-crawl; monetization is through partner tools (AgenticTrust page). - No published price list (package plans page). - HUMAN does not charge AI crawlers itself; the AgenticTrust page points to scraping monetization partners. - HUMAN's own docs say agents without cryptographic verification can be misread as human users, so they stay at Medium trust. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source AgenticTrust is a module of HUMAN Sightline Cyberfraud Defense; Confused.com integrated it for its insurance journeys. [Source](https://www.humansecurity.com/newsroom/confused-dot-com-partners-with-human-to-protect-digital-customer-journeys-in-agentic-era/), read 2026-11-04. - Primary source HUMAN assigns observed agents a High, Medium or Low Trust Level; all agents start at Medium, and High goes to agents verifiable by a cryptographic key on each request. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04, dated 2026-09-02. - Primary source Trust Levels weigh evidence of abuse or bypassing controls, cryptographic identity, behavioral consistency and discoverability, and a High level can be lowered if an agent starts misbehaving. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04, dated 2026-09-02. - Primary source Customers set agent permissions per category such as Content and Products, Engage, Account Creation, Checkout and Rate Limit, keyed to trust level. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04, dated 2026-09-02. - Vendor claim The AgenticTrust page says sites can deny or allow browsing, login or checkout per agent and verify agents with cryptographic signatures. [Source](https://www.humansecurity.com/applications/agentic-ai/), read 2026-11-04. - Vendor claim HUMAN's Satori team found 5.7% of traffic presenting an AI crawler or scraper user agent was spoofed. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04, dated 2026-09-02. - Vendor claim HUMAN reports 7,851% year-over-year growth in agentic AI traffic since 2025. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04, dated 2026-09-02. - Vendor claim In August 2026 agentic traffic grew 27% month over month with a 21.9% blocking rate; Comet Browser was 40.7% of agentic traffic, Claude Chrome Extension 22% and ChatGPT Agent 19.8%. [Source](https://www.humansecurity.com/learn/blog/state-of-agentic-traffic-august-2026-agentic-traffic-grows-27-reaches-new-high-as-codex-debuts-strongly/), read 2026-11-04, dated 2026-09-03. - Primary source HUMAN offers cryptographic verification of the Amazon Bedrock AgentCore Browser so agents built on AWS can be told apart from spoofed ones. [Source](https://www.humansecurity.com/newsroom/human-delivers-cryptographic-verification-amazon-bedrock-agentcore-browser/), read 2026-11-04. - Vendor claim The AgenticTrust page lists integrations with AI developers, chargeback protection vendors, scraping monetization tools, identity providers, OWASP and AGNTCY.org. [Source](https://www.humansecurity.com/applications/agentic-ai/), read 2026-11-04. - Vendor claim HUMAN says it was named a Leader in The Forrester Wave: Bot and Agent Trust Management Software, Q2 2026. [Source](https://www.humansecurity.com/newsroom/forrester-wave-bot-agent-trust-management-software-q2-2026/), read 2026-11-04, dated 2026-06-15. - Primary source HUMAN's package plans page lists Sightline AgenticTrust without any published price. [Source](https://www.humansecurity.com/platform/package-plans), read 2026-11-04. ## Dated news - 2026-09-02**HUMAN publishes Agent Trust Levels for AgenticTrust**. HUMAN described its High, Medium and Low Trust Levels for AI agents. Agents start at Medium, move to High with per-request cryptographic verification and good behavior, and drop to Low for spoofable identity or abuse. Levels drive default permissions for checkout, account creation and similar actions. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/). - 2026-09-03**HUMAN State of Agentic Traffic, August 2026**. HUMAN's monthly benchmark reported agentic traffic up 27% month over month with 21.9% blocked. Comet Browser led at 40.7% of agentic traffic, Claude Chrome Extension held 22%, and ChatGPT Agent rose to 19.8%, with Codex appearing for the first time. [Source](https://www.humansecurity.com/learn/blog/state-of-agentic-traffic-august-2026-agentic-traffic-grows-27-reaches-new-high-as-codex-debuts-strongly/). ## Compared with - [DataDome vs HUMAN Security](https://agentreadycrawlers.com/compare/datadome-vs-human-security/): DataDome publishes plan prices (from USD 3,830 a month); HUMAN publishes none and rates agents High, Medium or Low trust, with High reserved for agents that carry a cryptographic key on each request. - [HUMAN Security vs Kasada](https://agentreadycrawlers.com/compare/human-security-vs-kasada/): Kasada launched AI Agent Trust on 2026-01-21 with a verified agent directory that supports Web Bot Auth; HUMAN's AgenticTrust grades agents by trust level and per-action permissions, without naming Web Bot Auth on the pages read. - [Akamai vs HUMAN Security](https://agentreadycrawlers.com/compare/akamai-vs-human-security/): Akamai is listed by OpenAI as recognising ChatGPT's Cloud browser under the bot name "ChatGPT Agent"; HUMAN lists the same agent in Known Bots & Crawlers and AgenticTrust permissions. - [Cloudflare vs HUMAN Security](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security/): Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. ## What HUMAN Security does for this layer ### From the vendor's own pages HUMAN introduced AgenticTrust on 30 Jul 2025 as a new module of HUMAN Sightline Cyberfraud Defense covering websites, applications and mobile apps. ([humansecurity.com](https://www.humansecurity.com/learn/blog/agentictrust-govern-ai-agents/), dated 2025-07-30) HUMAN's AgenticTrust launch post sorts traffic into three classes, humans, bots and AI agents, and lists permissions, rate limits and the most targeted routes among what customers can review per agent. ([humansecurity.com](https://www.humansecurity.com/learn/blog/agentictrust-govern-ai-agents/), dated 2025-07-30) HUMAN's documentation cites ChatGPT as an agent that exposes its key so services can verify it, and points implementers to the IETF Web Bot Auth working group. ([docs.humansecurity.com](https://docs.humansecurity.com/applications/about-agent-trust-levels)) HUMAN assigns a Low Trust Level to agents that are unverified or easily spoofed, or that never or inconsistently declare their identity through a private signal to HUMAN. ([docs.humansecurity.com](https://docs.humansecurity.com/applications/about-agent-trust-levels)) HUMAN publishes HUMAN Verified AI Agent on GitHub, an open-source showcase of A2A-based agents that sign requests with HTTP Message Signatures (RFC 9421) following the Web Bot Auth architecture draft. ([github.com](https://github.com/humansecurity/human-verified-ai-agent)) HUMAN's Muse post says HUMAN supports the web-bot-auth standard but that most AI providers do not yet use it. ([humansecurity.com](https://www.humansecurity.com/learn/blog/meta-muse-agentic-traffic/), dated 2026-09-28) ### Claims the vendor makes about itself HUMAN's 28 Sep 2026 post says AgenticTrust identifies and verifies Meta's Muse and more than 30 other AI agents using proprietary research or digital signatures. ([humansecurity.com](https://www.humansecurity.com/learn/blog/meta-muse-agentic-traffic/), dated 2026-09-28) HUMAN reported in the same post that Muse made up an average of 72% of daily AI-agent requests it observed, 5.2 times the daily volume of ChatGPT Agent. ([humansecurity.com](https://www.humansecurity.com/learn/blog/meta-muse-agentic-traffic/), dated 2026-09-28) HUMAN's AgenticTrust product page offers verification of agents by cryptographic digital signatures and AI-generated insight into agent activity at the user level. ([humansecurity.com](https://www.humansecurity.com/applications/agentic-ai/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Sightline Cyberfraud Defense with AgenticTrust | Agent identification, trust levels and per-category permissions for browsing, login, account creation and checkout. | sales-led No price on the AgenticTrust page. | Not published | 2026-11-05 | | HUMAN Verified AI Agent (open source) | Reference code for agents that sign requests with HTTP Message Signatures. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | HUMAN says it supports web-bot-auth; signed agents get a High Trust Level. (humansecurity.com) | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol statement on the HUMAN pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No Agent Pay support statement on the HUMAN pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to HUMAN Security (Sightline, AgenticTrust)](https://agentreadycrawlers.com/vendors/human-security/alternatives/) graded on the rubric ## Sources - [humansecurity.com: humansecurity.com](https://www.humansecurity.com) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [humansecurity.com: agentic ai](https://www.humansecurity.com/applications/agentic-ai/) read 2026-11-04 - [humansecurity.com: confused dot com partners with human to protect digital customer journeys in agentic era](https://www.humansecurity.com/newsroom/confused-dot-com-partners-with-human-to-protect-digital-customer-journeys-in-agentic-era/) read 2026-11-04 - [humansecurity.com: state of agentic traffic august 2026 agentic traffic grows 27 reaches new high as codex debuts strongly](https://www.humansecurity.com/learn/blog/state-of-agentic-traffic-august-2026-agentic-traffic-grows-27-reaches-new-high-as-codex-debuts-strongly/) read 2026-11-04 - [humansecurity.com: human delivers cryptographic verification amazon bedrock agentcore browser](https://www.humansecurity.com/newsroom/human-delivers-cryptographic-verification-amazon-bedrock-agentcore-browser/) read 2026-11-04 - [humansecurity.com: forrester wave bot agent trust management software q2 2026](https://www.humansecurity.com/newsroom/forrester-wave-bot-agent-trust-management-software-q2-2026/) read 2026-11-04 - [AgenticTrust: The Trust Layer for Agentic Commerce in the AI Era - HUMAN Security](https://www.humansecurity.com/learn/blog/agentictrust-govern-ai-agents/) read 2026-11-05 - [About Agent Trust Levels | HUMAN Documentation](https://docs.humansecurity.com/applications/about-agent-trust-levels) read 2026-11-05 - [HumanSecurity/human-verified-ai-agent (GitHub)](https://github.com/humansecurity/human-verified-ai-agent) read 2026-11-05 - [Is Meta's Muse the first Mass-Market AI Agent? What Our Traffic Data Shows](https://www.humansecurity.com/learn/blog/meta-muse-agentic-traffic/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to HUMAN Security | Agent Ready Crawlers" description: "Alternatives to HUMAN Security: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head." canonical: https://agentreadycrawlers.com/vendors/human-security/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to HUMAN Security | Agent Ready Crawlers > Alternatives to HUMAN Security: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Source: https://agentreadycrawlers.com/vendors/human-security/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to HUMAN Security In short HUMAN Security ranks fifth of 15 on 7 of 10 points, just below the four-way tie at the top, with Akamai the nearest on points. Four head-to-head pages compare it with DataDome, Kasada, Akamai and Cloudflare. Its price is not public, and medium-trust agents lack cryptographic verification. ## Head-to-head, with the sourced difference - [**HUMAN Security vs DataDome**](https://agentreadycrawlers.com/compare/datadome-vs-human-security/). DataDome publishes plan prices (from USD 3,830 a month); HUMAN publishes none and rates agents High, Medium or Low trust, with High reserved for agents that carry a cryptographic key on each request. [Source](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/), read 2026-11-04. - [**HUMAN Security vs Kasada**](https://agentreadycrawlers.com/compare/human-security-vs-kasada/). Kasada launched AI Agent Trust on 2026-01-21 with a verified agent directory that supports Web Bot Auth; HUMAN's AgenticTrust grades agents by trust level and per-action permissions, without naming Web Bot Auth on the pages read. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04. - [**HUMAN Security vs Akamai**](https://agentreadycrawlers.com/compare/akamai-vs-human-security/). Akamai is listed by OpenAI as recognising ChatGPT's Cloud browser under the bot name "ChatGPT Agent"; HUMAN lists the same agent in Known Bots & Crawlers and AgenticTrust permissions. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. - [**HUMAN Security vs Cloudflare**](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security/). Cloudflare identifies ChatGPT's Cloud browser by bot tag chatgpt-agent and detection ID 129220581; HUMAN identifies it as "ChatGPT Agent" and adds AgenticTrust permissions per action. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with HUMAN Security's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than HUMAN Security on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | commerce agents | per-purpose policy | No public price; sales-led | | AWS WAF Bot Control | 8 | charging | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | commerce agents | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | charging, commerce agents | per-purpose policy | From USD 3,830/month | | Skyfire | 6 | charging, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | charging | verifies agents, per-purpose policy | Free for publishers to start | | Vercel | 6 | nothing | charging | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | nothing | per-purpose policy, charging | No public price; sales-led | | Fastly | 4 | commerce agents | verifies agents, per-purpose policy, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, per-purpose policy, charging, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | commerce agents | verifies agents, per-purpose policy, charging, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy, charging | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, charging, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, per-purpose policy, charging, reporting | Platform plans from USD 0; extension price not stated | ## Where HUMAN Security is not the fit - Medium-trust agents lack cryptographic verification, so HUMAN itself warns of false negatives (Trust Levels blog, 2026-09-02). - No native pay-per-crawl; monetization is through partner tools (AgenticTrust page). - No published price list (package plans page). ## Sources - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 - [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Hyperbrowser: crawler operator profile | Agent Ready Crawlers" description: "Hyperbrowser: crawler operator profile covering Hyperbrowser sessions and Hyperbrowser Web API (fetch, search, extract). Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/hyperbrowser site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Hyperbrowser: crawler operator profile | Agent Ready Crawlers > Hyperbrowser: crawler operator profile covering Hyperbrowser sessions and Hyperbrowser Web API (fetch, search, extract). Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/hyperbrowser Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Hyperbrowser In short Hyperbrowser lets a team restrict signing to as many as 50 hostnames, and keeps the signing identity fixed at www.hyperbrowser.ai. Its Web API endpoints for fetch, search and extract cannot sign at all, and the free account's 5,000 credits come before a USD 30 Startup plan. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [hyperbrowser.ai](https://www.hyperbrowser.ai) In one line Cloud browser platform for AI agents whose sessions can sign requests with Web Bot Auth under the fixed Signature-Agent www.hyperbrowser.ai, on request per team, alongside stealth, proxies and CAPTCHA solving. Last read 2026-11-05 ## What Hyperbrowser does for this layer ### From the vendor's own pages Hyperbrowser sessions can sign their requests with Web Bot Auth following the IETF httpsig protocol draft, and the feature is enabled for teams on request. ([hyperbrowser.ai](https://www.hyperbrowser.ai/docs/sessions/web-bot-auth.md)) Hyperbrowser's Signature-Agent is always "https://www.hyperbrowser.ai", and its key directory is a JSON Web Key Set of Ed25519 keys whose kid is the RFC 7638 thumbprint, with every key also signing the directory response. ([hyperbrowser.ai](https://www.hyperbrowser.ai/docs/sessions/web-bot-auth.md)) Hyperbrowser tells site owners that signed requests come from real browsers and usually carry a standard browser User-Agent, so they should identify them by signature rather than by User-Agent. ([hyperbrowser.ai](https://www.hyperbrowser.ai/docs/sessions/web-bot-auth.md)) A Hyperbrowser session that asks for Web Bot Auth is either signed or refused and never starts unsigned; the option cannot be combined with useUltraStealth, and scrape, crawl and extract requests do not accept it. ([hyperbrowser.ai](https://www.hyperbrowser.ai/docs/sessions/web-bot-auth.md)) Hyperbrowser customers can limit signing to up to 50 hostnames, subdomains included, with the onlyDomains option. ([hyperbrowser.ai](https://www.hyperbrowser.ai/docs/sessions/web-bot-auth.md)) Hyperbrowser's pricing page gives a free account 5,000 credits, prices Startup at USD 30 a month and Scale at USD 100 a month, and bills browsers at USD 0.10 an hour and proxy data at USD 10 per GB, where 1,000 credits equal USD 1. ([hyperbrowser.ai](https://www.hyperbrowser.ai/pricing)) Hyperbrowser charges 9 extra credits per page for a fetch with ultra stealth or a chosen location, on top of 1 credit per fetched page. ([hyperbrowser.ai](https://www.hyperbrowser.ai/pricing)) Hyperbrowser's docs index lists Stealth Mode anti-detection settings and CAPTCHA solving, and an x402 endpoint that sells one browser session at 0.10 USDC per hour with no account or API key. ([hyperbrowser.ai](https://www.hyperbrowser.ai/docs/llms.txt)) ### What our crawl saw Hyperbrowser's key directory at www.hyperbrowser.ai/.well-known/http-message-signatures-directory returned HTTP 200, content type application/http-message-signatures-directory+json, with one Ed25519 key on 2026-11-05. ([hyperbrowser.ai](https://www.hyperbrowser.ai/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Hyperbrowser sessions | Cloud browsers for agents with stealth, proxies, CAPTCHA solving and optional Web Bot Auth signing. | credits Startup a month; free account 5,000 credits; browser USD 0.10 an hour. | USD 30 | 2026-11-05 | | Hyperbrowser Web API (fetch, search, extract) | Page fetch and search endpoints; these cannot sign with Web Bot Auth. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | The docs carry only a troubleshooting tip to check robots.txt; no crawl policy (2026-11-05). | | Web Bot Auth | Supports | Per-team opt-in signing with a fixed Signature-Agent and published key directory. (hyperbrowser.ai) | | Published IP list | No statement | No egress IP list published (2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key | GET https://www.hyperbrowser.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Hyperbrowser does not do - Hyperbrowser does not sign by default; teams must ask for Web Bot Auth, and unsigned sessions carry an ordinary browser User-Agent. ## Reading notes Hyperbrowser has no row in the [Crawler Index](https://agentreadycrawlers.com/crawlers/) yet. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [hyperbrowser.ai](https://www.hyperbrowser.ai) read 2026-11-05 - [Web Bot Auth | Hyperbrowser docs](https://www.hyperbrowser.ai/docs/sessions/web-bot-auth.md) read 2026-11-05 - [Hyperbrowser key directory](https://www.hyperbrowser.ai/.well-known/http-message-signatures-directory) read 2026-11-05 - [Pricing | Hyperbrowser](https://www.hyperbrowser.ai/pricing) read 2026-11-05 - [Hyperbrowser documentation index (llms.txt)](https://www.hyperbrowser.ai/docs/llms.txt) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "IAB Tech Lab: licensing intermediary profile | Agent Ready Crawlers" description: "IAB Tech Lab: licensing intermediary profile covering CoMP 1.0, Spiders and Bots List and Agent Registry. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/iab-tech-lab site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # IAB Tech Lab: licensing intermediary profile | Agent Ready Crawlers > IAB Tech Lab: licensing intermediary profile covering CoMP 1.0, Spiders and Bots List and Agent Registry. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/iab-tech-lab Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Licensing intermediary # IAB Tech Lab In short IAB Tech Lab's CoMP 1.0, settled in April 2026, names five price types and six ways for a licensed AI system to authenticate, from a user agent to Web Bot Auth, and leaves issuing tokens and moving money to others. Its Spiders and Bots List is sold by subscription; the Agent Registry is free to any registered user. Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/) In one line Ad-industry standards body whose Content Monetization Protocols (CoMP) 1.0 define how a licensed AI crawler asks a publisher for content and gets a token, alongside a paid Spiders and Bots List and a free Agent Registry. Last read 2026-11-05 ## What IAB Tech Lab does for this layer ### From the vendor's own pages IAB Tech Lab opened CoMP v1 for public comment until 9 Apr 2026 and finalised it on 28 Apr 2026. ([iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/), dated 2026-04-28) IAB Tech Lab renamed its LLM Content Ingest API initiative to Content Monetization Protocols (CoMP) for AI after a workshop with more than 80 executives, including Cloudflare, Fastly and AWS, where TollBit, Dappier, Cloudflare and ProRata presented. ([iabtechlab.com](https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/), dated 2025-08-19) IAB Tech Lab says CoMP covers the APIs between an AI system and a content owner once the bot holds a license, and states that CoMP is not a bot blocking system, not a licensing marketplace and not a definition of economic models. ([iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/), dated 2026-04-28) In IAB Tech Lab's CoMP workflow a bot without an agreement is denied and sent to a Licensing URL; with an agreement it sends an access request and its reason and receives an access token for the content. ([iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/), dated 2026-04-28) The CoMP 1.0 specification lists six ways an AI system can authenticate: UserAgent, IP range, Token, WebBot Auth, Agent ID from the Tech Lab Registry, and Other. ([raw.githubusercontent.com](https://raw.githubusercontent.com/IABTechLab/CoMP/main/CoMP-1.0.md)) CoMP 1.0 defines the price types Per-Use, Per-Query, Per-Token, Flat and Tiered Rate, and leaves token issuance, counting and payment out of scope as clearing-house functions. ([raw.githubusercontent.com](https://raw.githubusercontent.com/IABTechLab/CoMP/main/CoMP-1.0.md)) IAB Tech Lab licenses the CoMP specification under Creative Commons Attribution 3.0, and only IAB Tech Lab members may take part in the working group. ([github.com](https://github.com/IABTechLab/CoMP)) IAB Tech Lab's Bot and Crawler Management Guidance tells publishers to verify crawlers beyond the user agent with ASN lookup, reverse DNS and IP reputation, and to review robots.txt and middleware rules quarterly. ([raw.githubusercontent.com](https://raw.githubusercontent.com/IABTechLab/CoMP/main/Bot%20and%20Crawler%20Management%20Guidance.md)) The IAB Tech Lab Spiders and Bots List costs USD 5,000 a year for IAB Tech Lab members, USD 7,500 for IAB associate members and USD 15,000 for non-members, and is updated monthly with the Alliance for Audited Media. ([iabtechlab.com](https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/)) IAB Tech Lab's Agent Registry went live on 3 Mar 2026 and is free to anyone who registers in the Tech Lab Tools Portal. ([iabtechlab.com](https://iabtechlab.com/navigating-the-agentic-frontier-the-iab-tech-lab-2026-roadmap/), dated 2026-03-03) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | CoMP 1.0 | Request and response objects for a licensed AI system asking a content owner or marketplace for content, with license URL, scope, allowed use, price type and retrieval method. | open specification CC BY 3.0 | USD 0 | 2026-11-05 | | Spiders and Bots List | Paid list of known robots and valid browsers for filtering invalid traffic in ad measurement. | annual subscription USD 15,000 a year for non-members | USD 5000 | 2026-11-05 | | Agent Registry | Registry of agents in the Tech Lab Tools Portal; CoMP accepts its Agent ID as an authentication method. | free with Tools Portal registration live since 2026-03-03 | USD 0 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | The guidance tells publishers to publish a machine-readable robots.txt and update it quarterly. (raw.githubusercontent.com) | | Web Bot Auth | Supports | CoMP 1.0 lists WebBot Auth (value 3) among accepted AI authentication methods. (raw.githubusercontent.com) | | Published IP list | Partial | CoMP 1.0 accepts IP range as an authentication method; IAB Tech Lab publishes no crawler IP lists of its own. (raw.githubusercontent.com) | | Reverse DNS check | Supports | The Bot and Crawler Management Guidance recommends reverse DNS on the IP for major search engines. (raw.githubusercontent.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What IAB Tech Lab does not do - IAB Tech Lab does not issue tokens, count usage or move money under CoMP; those clearing-house functions sit with marketplaces or the content owner. - IAB Tech Lab does not block bots; CoMP assumes the publisher already runs its own blocking strategy. ## Reading notes The RSL 1.0 announcement of 10 December 2025 lists IAB Tech Lab among the supporters of RSL, and CoMP's bot guidance tells publishers to watch RSL developments; see [RSL Collective](https://agentreadycrawlers.com/vendors/rsl-collective/). ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [TollBit](https://agentreadycrawlers.com/vendors/tollbit/) licensing intermediary - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management - [Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/) bot management ## Sources - [CoMP (Content Monetization Protocols) Initiative](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/) read 2026-11-05 - [Announcing the Content Monetization Protocols (CoMP) for AI Working Group](https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/) read 2026-11-05 - [CoMP 1.0 specification (IABTechLab/CoMP on GitHub)](https://raw.githubusercontent.com/IABTechLab/CoMP/main/CoMP-1.0.md) read 2026-11-05 - [IABTechLab/CoMP repository](https://github.com/IABTechLab/CoMP) read 2026-11-05 - [IAB Tech Lab Bot and Crawler Management Guidance](https://raw.githubusercontent.com/IABTechLab/CoMP/main/Bot%20and%20Crawler%20Management%20Guidance.md) read 2026-11-05 - [IAB Tech Lab Spiders and Bots List](https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/) read 2026-11-05 - [Navigating the Agentic Frontier: The IAB Tech Lab 2026 Roadmap](https://iabtechlab.com/navigating-the-agentic-frontier-the-iab-tech-lab-2026-roadmap/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Imperva: bot management profile | Agent Ready Crawlers" description: "Imperva: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 11 of 15." canonical: https://agentreadycrawlers.com/vendors/imperva site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Imperva: bot management profile | Agent Ready Crawlers > Imperva: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 11 of 15. Source: https://agentreadycrawlers.com/vendors/imperva Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 11 (tied) of 15 # Imperva In short Imperva, part of Thales, sells Advanced Bot Protection, which grades yes on reporting and partial on agent verification, for 3 of 10 points; per-purpose policy, charging and commerce agents are not published. Neither the Imperva nor the Thales pages read mention verifying a signed agent, and the price is not public. Application security vendor owned by Thales whose Advanced Bot Protection separates human, good bot and bad bot traffic; agent-specific verified-trust controls exist so far only in a controlled-availability Google Cloud offering. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Imperva](https://agentreadycrawlers.com/vendors/imperva/alternatives/) · [imperva.com](https://www.imperva.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Operator Thales Homepage [imperva.com](https://www.imperva.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Partial | verified trust for verified AI agents in Imperva for Google Cloud (controlled availability); no Web Bot Auth support found (checked 2026-11-04) | Thales-Google blog lists Verified Trust; ABP page names no verification method for agents. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Not published | per-purpose AI crawler policy (checked 2026-11-04) | ABP page has no AI crawler or training controls. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | No monetization feature found on ABP page. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned on ABP page or Thales-Google blog. Source, read 2026-11-04. | | Cost | Not graded | no public price; sales-led | ABP page lists no pricing. Source, read 2026-11-04. | | Reporting | Yes | customizable dashboards and reports | ABP page claims hundreds of dimensions for dashboards. Source, read 2026-11-04. | | Where it runs | Not graded | edge or cloud: Imperva cloud WAF stack, Imperva for AWS and Imperva for Google Cloud | AWS and Google offerings described in Thales sources. Source, read 2026-11-04. | ## Cost no public price; sales-led (checked 2026-11-04) [Source](https://www.imperva.com/products/advanced-bot-protection-management/), read 2026-11-04. ## What Imperva does not do - No Web Bot Auth or signed-agent verification found on Imperva or Thales pages read (checked 2026-11-04). - No AI crawler monetization or pay-per-crawl feature found (ABP page). - Agent verified-trust controls are limited to Imperva for Google Cloud and are in Controlled Availability (Thales blog, 2026-05-04). - Imperva's Advanced Bot Protection page names no Web Bot Auth, IP-list or reverse DNS method for verifying AI agents (read 2026-11-05). - Imperva publishes no charging or licensing option for AI crawlers. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Vendor claim The Advanced Bot Protection page says the product separates human, good bot and bad bot traffic and includes good bot management. [Source](https://www.imperva.com/products/advanced-bot-protection-management/), read 2026-11-04, dated 2026-10-15. - Primary source The Advanced Bot Protection page describes agentic AI as a new class of automated traffic that behaves like legitimate users but names no AI crawler, agent or Web Bot Auth controls. [Source](https://www.imperva.com/products/advanced-bot-protection-management/), read 2026-11-04, dated 2026-10-15. - Vendor claim Imperva says ABP offers customizable dashboards and reports built from hundreds of traffic dimensions. [Source](https://www.imperva.com/products/advanced-bot-protection-management/), read 2026-11-04, dated 2026-10-15. - Primary source Thales and Google partnered on 4 May 2026 to protect traffic through Google's AI Agent Gateway; the agentic controls include real-time visibility, a verified trust option for verified AI agents and agentic browsers, and threat blocking. [Source](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), read 2026-11-04, dated 2026-05-04. - Primary source Those agentic AI controls ship in Imperva for Google Cloud (IGC) and are in Controlled Availability. [Source](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), read 2026-11-04, dated 2026-05-04. - Vendor claim Thales's 2026 Bad Bot Report says bots are 53% of internet traffic and bad bots alone 40%. [Source](https://cpl.thalesgroup.com/blog/application-security/bad-bots-in-the-agentic-age), read 2026-11-04, dated 2026-04-30. - Vendor claim Thales says it blocked 17.2 trillion bad bot requests in 2025. [Source](https://cpl.thalesgroup.com/blog/application-security/bad-bots-in-the-agentic-age), read 2026-11-04, dated 2026-04-30. - Vendor claim Thales observed daily AI-driven bot attacks rise from 2 million to 25 million between 2024 and 2025. [Source](https://cpl.thalesgroup.com/blog/application-security/bad-bots-in-the-agentic-age), read 2026-11-04, dated 2026-04-30. - Secondary source In early August 2026 Thales launched Imperva for AWS on AWS Marketplace, a SaaS bundle that includes Advanced Bot Protection. [Source](https://softprom.com/thales-launches-imperva-for-aws-to-protect-applications-and-apis-in-the-cloud), read 2026-11-04, dated 2026-08-03. ## Compared with - [Imperva vs Cloudflare](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare/): Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. - [Imperva vs DataDome](https://agentreadycrawlers.com/compare/imperva-vs-datadome/): Imperva's product page names agentic AI as a traffic class but lists no AI crawler or agent controls; DataDome documents Web Bot Auth, KYA, IP lists and reverse DNS for agent identification. ## What Imperva does for this layer ### From the vendor's own pages Imperva offers a free trial of Advanced Bot Protection from its product page and publishes no price. ([imperva.com](https://www.imperva.com/products/advanced-bot-protection-management/)) The 2026 Bad Bot Report describes a new class of AI-driven fetcher bots that take actions rather than crawl and interact directly with APIs. ([cpl.thalesgroup.com](https://cpl.thalesgroup.com/blog/application-security/bad-bots-in-the-agentic-age), dated 2026-04-30) Thales says Imperva for Google Cloud runs natively inside the Google Cloud environment, so web and API traffic is inspected without leaving Google's infrastructure. ([cpl.thalesgroup.com](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), dated 2026-05-04) Thales lists agent-to-MCP inspection, covering unauthorized tool execution, anomalous tool calls and malicious parameter injection, among the capabilities it plans to deliver in Google's agent gateway ecosystem. ([cpl.thalesgroup.com](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), dated 2026-05-04) ### Claims the vendor makes about itself Imperva's Advanced Bot Protection page says the product covers all OWASP 21 Automated Threats across websites, mobile apps and APIs. ([imperva.com](https://www.imperva.com/products/advanced-bot-protection-management/)) Imperva says it was named a Security Leader in the SecureIQLab 2026 WAAP Report. ([imperva.com](https://www.imperva.com/products/advanced-bot-protection-management/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Advanced Bot Protection | Good bot management, bad bot mitigation, dashboards and reports for websites, mobile apps and APIs. | sales-led, free trial Free trial offered on the product page. | Not published | 2026-11-05 | | Imperva for Google Cloud (IGC) | Application security inside Google Cloud with agentic AI controls (visibility, verified trust, threat blocking); Controlled Availability. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | No Web Bot Auth or signed-agent statement on the Imperva and Thales pages read 2026-11-05. | | Visa Trusted Agent Protocol | No statement | No statement on the pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Imperva, a Thales company (Advanced Bot Protection)](https://agentreadycrawlers.com/vendors/imperva/alternatives/) graded on the rubric ## Sources - [imperva.com: imperva.com](https://www.imperva.com) read 2026-11-04 - [imperva.com: advanced bot protection management](https://www.imperva.com/products/advanced-bot-protection-management/) read 2026-11-04 - [cpl.thalesgroup.com: thales google protect ai agent ecosystem](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem) read 2026-11-04 - [cpl.thalesgroup.com: bad bots in the agentic age](https://cpl.thalesgroup.com/blog/application-security/bad-bots-in-the-agentic-age) read 2026-11-04 - [softprom.com: thales launches imperva for aws to protect applications and apis in the cloud](https://softprom.com/thales-launches-imperva-for-aws-to-protect-applications-and-apis-in-the-cloud) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Alternatives to Imperva | Agent Ready Crawlers" description: "Alternatives to Imperva: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/imperva/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Imperva | Agent Ready Crawlers > Alternatives to Imperva: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/imperva/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Imperva In short Imperva shares 11th place on 3 of 10 points with Arcjet and Auth0 for AI Agents, and two head-to-head pages compare it with Cloudflare and DataDome. Imperva grades yes only on reporting, its Imperva and Thales pages say nothing on verifying signed agents, and the price is not public. ## Head-to-head, with the sourced difference - [**Imperva vs Cloudflare**](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare/). Imperva's verified-trust control for AI agents exists only in Imperva for Google Cloud in Controlled Availability (2026-05-04); Cloudflare's verified bots and signed agents work on every zone. [Source](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem), read 2026-11-04. - [**Imperva vs DataDome**](https://agentreadycrawlers.com/compare/imperva-vs-datadome/). Imperva's product page names agentic AI as a traffic class but lists no AI crawler or agent controls; DataDome documents Web Bot Auth, KYA, IP lists and reverse DNS for agent identification. [Source](https://docs.datadome.co/docs/agentic-trust-getting-started), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Imperva's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Imperva on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, per-purpose policy, charging, commerce agents | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy, charging | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, charging, commerce agents | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, per-purpose policy, charging, commerce agents | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy, charging | nothing | No public price; sales-led | | Skyfire | 6 | verifies agents, charging, commerce agents | reporting | Not published (checked 2026-11-04) | | TollBit | 6 | per-purpose policy, charging | nothing | Free for publishers to start | | Vercel | 6 | verifies agents, per-purpose policy | nothing | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents, per-purpose policy | nothing | No public price; sales-led | | Fastly | 4 | per-purpose policy, charging, commerce agents | reporting | No public price; sales-led | | Arcjet | 3 | per-purpose policy | reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | per-purpose policy, commerce agents | verifies agents, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Stytch IsAgent | 2 | nothing | reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | per-purpose policy | verifies agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Imperva is not the fit - No Web Bot Auth or signed-agent verification found on Imperva or Thales pages read (checked 2026-11-04). - No AI crawler monetization or pay-per-crawl feature found (ABP page). - Agent verified-trust controls are limited to Imperva for Google Cloud and are in Controlled Availability (Thales blog, 2026-05-04). ## Sources - [imperva.com: advanced bot protection management](https://www.imperva.com/products/advanced-bot-protection-management/) read 2026-11-04 - [cpl.thalesgroup.com: thales google protect ai agent ecosystem](https://cpl.thalesgroup.com/blog/cybersecurity/thales-google-protect-ai-agent-ecosystem) read 2026-11-04 - [docs.datadome.co: agentic trust getting started](https://docs.datadome.co/docs/agentic-trust-getting-started) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Kasada: bot management profile | Agent Ready Crawlers" description: "Kasada: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 9 of 15." canonical: https://agentreadycrawlers.com/vendors/kasada site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Kasada: bot management profile | Agent Ready Crawlers > Kasada: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 9 of 15. Source: https://agentreadycrawlers.com/vendors/kasada Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 9 of 15 # Kasada In short Kasada's AI Agent Trust, launched on 21 January 2026, grades yes on agent verification and reporting and partial on per-purpose policy, for 5 of 10 points. It has no monetization or pay-per-crawl feature, commerce agents are not published, and the price is available only through sales. Bot defense vendor that added AI Agent Trust in January 2026: a verified agent directory with Web Bot Auth support, trust-based access policies enforced at the edge, and agent reporting in the Kasada Portal. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Kasada](https://agentreadycrawlers.com/vendors/kasada/alternatives/) · [kasada.io](https://www.kasada.io) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Based in Australia Homepage [kasada.io](https://www.kasada.io) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | Web Bot Auth plus Kasada's verified agent directory | Launch post names Web Bot Auth support in the directory. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | endpoint-level access policies per verified agent; no published search versus training split (checked 2026-11-04) | Homepage describes endpoint-level policies. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | Launch post has no monetization feature. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned in launch post or homepage. Source, read 2026-11-04. | | Cost | Not graded | no public price; sales-led | Homepage shows no prices. Source, read 2026-11-04. | | Reporting | Yes | Kasada Portal and Custom Reporting for verified agent activity | Launch post reporting section. Source, read 2026-11-04. | | Where it runs | Not graded | edge: enforcement upstream of origin | Launch post: real-time enforcement at the edge. Source, read 2026-11-04. | ## Cost no public price; sales-led (checked 2026-11-04). Indirectly, Vercel BotID Deep Analysis (powered by Kasada) costs USD 1 per 1,000 checkBotId() calls on Vercel Pro. [Source](https://www.kasada.io/), read 2026-11-04. ## What Kasada does not do - No monetization or pay-per-crawl feature in AI Agent Trust (launch post, 2026-01-21). - No public price list (homepage read 2026-11-04). - No published support for Visa TAP or Mastercard Agent Pay (pages read 2026-11-04). - Kasada publishes no price; its product pages lead to a demo request (read 2026-11-05). - Kasada's AI Agent Trust offers no paid access or licensing route for AI crawlers. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Kasada launched AI Agent Trust on 21 Jan 2026 to give enterprises verified, policy-based control over AI agents and automated traffic. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04, dated 2026-01-21. - Primary source AI Agent Trust has four parts: a verified bots and agent directory, policy-based agent access controls, real-time enforcement at the edge, and reporting. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04, dated 2026-01-21. - Primary source The agent directory supports emerging standards such as Web Bot Auth. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04, dated 2026-01-21. - Vendor claim Trust decisions are enforced upstream, before automated traffic reaches downstream systems, analytics or revenue. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04, dated 2026-01-21. - Primary source Verified agent activity, including verification results, request patterns and endpoint interactions, shows in the Kasada Portal and in Custom Reporting. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04, dated 2026-01-21. - Vendor claim Kasada's homepage says it verifies agent identity, detects impersonators and applies endpoint-level access policies without reducing every decision to allow or block. [Source](https://www.kasada.io/), read 2026-11-04. - Vendor claim Kasada says it can be deployed behind an incumbent bot-management provider. [Source](https://www.kasada.io/), read 2026-11-04. - Primary source Kasada powers the Deep Analysis tier of Vercel BotID. [Source](https://vercel.com/docs/botid), read 2026-11-04. - Vendor claim Kasada says it was named a Leader in The Forrester Wave: Bot and Agent Trust Management Software, Q2 2026. [Source](https://www.businesswire.com/news/home/20260615262564/en/), read 2026-11-04, dated 2026-06-15. ## Compared with - [HUMAN Security vs Kasada](https://agentreadycrawlers.com/compare/human-security-vs-kasada/): Kasada launched AI Agent Trust on 2026-01-21 with a verified agent directory that supports Web Bot Auth; HUMAN's AgenticTrust grades agents by trust level and per-action permissions, without naming Web Bot Auth on the pages read. - [Kasada vs Vercel](https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid/): Kasada powers Vercel BotID Deep Analysis, sold by Vercel at USD 1 per 1,000 checkBotId() calls on Pro; Kasada itself is sales-led and adds an AI agent directory and edge policies. - [DataDome vs Kasada](https://agentreadycrawlers.com/compare/datadome-vs-kasada/): DataDome publishes prices and monetization tiers; Kasada publishes none but says it can deploy behind an incumbent bot manager. - [Kasada vs Akamai](https://agentreadycrawlers.com/compare/kasada-vs-akamai/): Kasada says it can be deployed behind an incumbent bot manager; Akamai's Bot & Agent Control runs on Akamai's own edge with partner integrations for identity and charging. ## What Kasada does for this layer ### From the vendor's own pages Kasada says it was founded in 2015 and is headquartered in Sydney, with hubs in New York, Boston, Melbourne and San Francisco. ([kasada.io](https://www.kasada.io/about-us)) Kasada's AI Agent Trust launch post describes its directory as covering AI crawlers, assistants and agents, enriched with vendor identity and category data. ([kasada.io](https://www.kasada.io/blog/kasada-launches-ai-agent-trust-to-secure-agentic-commerce), dated 2026-01-21) Kasada's homepage says its scraping protection verifies legitimate agents and gives each one access only to the endpoints the site owner approves. ([kasada.io](https://www.kasada.io/)) Kasada publishes comparison pages against Akamai, Cloudflare and HUMAN Security from its homepage. ([kasada.io](https://www.kasada.io/)) ### Claims the vendor makes about itself Kasada says its client-side code runs a full virtual machine with custom bytecode instead of obfuscated open-source JavaScript. ([kasada.io](https://www.kasada.io/)) Kasada reports that in one deployment behind Akamai Bot Manager it classed 90% of requests as bad bots and saw a false-negative rate above 75% on traffic Akamai had allowed. ([kasada.io](https://www.kasada.io/)) Kasada says it protects USD 150 billion of eCommerce and 250 million people daily. ([kasada.io](https://www.kasada.io/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bot Defense | Edge bot mitigation with obfuscated client-side checks. | sales-led Homepage leads to a demo request. | Not published | 2026-11-05 | | AI Agent Trust | Verified agent directory, trust-based access policies, edge enforcement and agent reporting in the Kasada Portal. | No price recorded | | Account Intelligence | Human fraud detection at the edge. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | AI Agent Trust's directory supports emerging standards such as Web Bot Auth. (kasada.io) | | Visa Trusted Agent Protocol | No statement | No statement on the Kasada pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the Kasada pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Vercel (BotID, Firewall bot management)](https://agentreadycrawlers.com/vendors/vercel-botid/) bot management - [Alternatives to Kasada (Bot Defense, AI Agent Trust)](https://agentreadycrawlers.com/vendors/kasada/alternatives/) graded on the rubric ## Sources - [kasada.io: kasada.io](https://www.kasada.io) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [businesswire.com: en](https://www.businesswire.com/news/home/20260615262564/en/) read 2026-11-04 - [About Us - Learn More About Our Company | Kasada](https://www.kasada.io/about-us) read 2026-11-05 - [Kasada Launches AI Agent Trust to Secure Agentic Commerce - Kasada](https://www.kasada.io/blog/kasada-launches-ai-agent-trust-to-secure-agentic-commerce) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Kasada | Agent Ready Crawlers" description: "Alternatives to Kasada: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/kasada/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Kasada | Agent Ready Crawlers > Alternatives to Kasada: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/kasada/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Kasada In short Kasada ranks ninth of 15 on 5 of 10 points, one place above Fastly, which is the nearest on points. Four head-to-head pages compare it with HUMAN Security, Vercel, DataDome and Akamai. Kasada's agent product cannot charge crawlers, and its price comes only from a sales conversation. ## Head-to-head, with the sourced difference - [**Kasada vs HUMAN Security**](https://agentreadycrawlers.com/compare/human-security-vs-kasada/). Kasada launched AI Agent Trust on 2026-01-21 with a verified agent directory that supports Web Bot Auth; HUMAN's AgenticTrust grades agents by trust level and per-action permissions, without naming Web Bot Auth on the pages read. [Source](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/), read 2026-11-04. - [**Kasada vs Vercel**](https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid/). Kasada powers Vercel BotID Deep Analysis, sold by Vercel at USD 1 per 1,000 checkBotId() calls on Pro; Kasada itself is sales-led and adds an AI agent directory and edge policies. [Source](https://vercel.com/docs/botid), read 2026-11-04. - [**Kasada vs DataDome**](https://agentreadycrawlers.com/compare/datadome-vs-kasada/). DataDome publishes prices and monetization tiers; Kasada publishes none but says it can deploy behind an incumbent bot manager. [Source](https://www.kasada.io/), read 2026-11-04. - [**Kasada vs Akamai**](https://agentreadycrawlers.com/compare/kasada-vs-akamai/). Kasada says it can be deployed behind an incumbent bot manager; Akamai's Bot & Agent Control runs on Akamai's own edge with partner integrations for identity and charging. [Source](https://www.kasada.io/), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Kasada's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Kasada on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | charging, commerce agents | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | per-purpose policy, charging | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | per-purpose policy, charging, commerce agents | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | charging, commerce agents | nothing | From USD 3,830/month | | HUMAN Security | 7 | per-purpose policy, charging | nothing | No public price; sales-led | | Skyfire | 6 | charging, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | charging | verifies agents | Free for publishers to start | | Vercel | 6 | per-purpose policy | nothing | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Fastly | 4 | charging, commerce agents | verifies agents, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | commerce agents | verifies agents, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Kasada is not the fit - No monetization or pay-per-crawl feature in AI Agent Trust (launch post, 2026-01-21). - No public price list (homepage read 2026-11-04). - No published support for Visa TAP or Mastercard Agent Pay (pages read 2026-11-04). ## Sources - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 - [kasada.io: kasada launches ai agent trust to secure agentic commerce](https://www.kasada.io/kasada-launches-ai-agent-trust-to-secure-agentic-commerce/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Kernel: crawler operator profile | Agent Ready Crawlers" description: "Kernel: crawler operator profile covering Kernel cloud browsers and Kernel Search (KernelSearchBot). Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/kernel site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Kernel: crawler operator profile | Agent Ready Crawlers > Kernel: crawler operator profile covering Kernel cloud browsers and Kernel Search (KernelSearchBot). Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/kernel Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Kernel In short Kernel treats signing as a complement to its stealth mode and residential proxies, not a guarantee of access, and grants it by request to Startup and Enterprise customers. Its two signed identities, Kernel Agent and KernelSearchBot, each have a separate key directory, and the free tier carries USD 5 of monthly credits. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [kernel.sh](https://www.kernel.sh) In one line Cloud browser company for AI agents that runs two signed identities, Kernel Agent for user-directed browsing and KernelSearchBot for search crawling, each with its own Web Bot Auth key directory so sites can treat them separately. Last read 2026-11-05 ## What Kernel does for this layer ### From the vendor's own pages Kernel Search identifies itself with the KernelSearchBot user agent token and follows robots.txt directives for that token, including crawl-delay. ([kernel.sh](https://www.kernel.sh/docs/bots.md)) Kernel says Kernel Agent is not an automatic crawler and signs its requests with Web Bot Auth instead of sending a dedicated crawler user agent token. ([kernel.sh](https://www.kernel.sh/docs/bots.md)) Kernel serves a separate key directory for each identity so that site owners can allow search indexing while blocking user-directed agents, or the reverse. ([kernel.sh](https://www.kernel.sh/docs/bots.md)) Kernel's Web Bot Auth token is an opt-in feature enabled selectively on request for Startup and Enterprise plan customers, and being on an eligible plan does not switch it on. ([kernel.sh](https://www.kernel.sh/docs/browsers/bot-detection/web-bot-auth.md)) Kernel's docs say Web Bot Auth complements its stealth mode and proxies, which deal with browser and network signals, and does not guarantee access to every site. ([kernel.sh](https://www.kernel.sh/docs/browsers/bot-detection/web-bot-auth.md)) Kernel's pricing page lists Free at USD 0 with USD 5 of monthly credits and 5 concurrent browsers, Hobbyist at USD 30 a month and Start-up at USD 200 a month with 150 concurrent browsers, plus usage at USD 0.0000166667 per GB-second. ([kernel.sh](https://www.kernel.sh/pricing)) Kernel says its usage rates include proxy traffic and that free tiers include stealth mode, which attaches a Kernel-managed residential proxy and a captcha auto-solver to each browser. ([kernel.sh](https://www.kernel.sh/pricing)) ### Claims the vendor makes about itself Kernel says it is listed in Cloudflare's bots and agents directory and in Vercel's public directory at bots.fyi. ([kernel.sh](https://www.kernel.sh/docs/bots.md)) ### What our crawl saw Both Kernel key directories, www.kernel.sh and search.bot.kernel.sh, returned HTTP 200 with content type application/http-message-signatures-directory+json and one Ed25519 key each on 2026-11-05. ([kernel.sh](https://www.kernel.sh/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Kernel cloud browsers | Isolated Chromium for agents with stealth, proxies, managed auth and opt-in Web Bot Auth. | subscription plus usage Free with USD 5 monthly credits; Hobbyist USD 30, Start-up USD 200 a month; usage per GB-second. | USD 0 | 2026-11-05 | | Kernel Search (KernelSearchBot) | Crawler that builds search indexes and retrieval databases; signed and robots.txt-compliant. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | KernelSearchBot follows robots.txt for its token, crawl-delay included. (kernel.sh) | | Web Bot Auth | Supports | Kernel Agent and Kernel Search sign with Web Bot Auth; key directories published per identity. (kernel.sh) | | Published IP list | No statement | No IP list on the bots page (2026-11-05). | | Cloudflare verified bots | Supports | Kernel says it is listed in Cloudflare's bots and agents directory. (kernel.sh) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key | GET https://www.kernel.sh/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key | GET https://search.bot.kernel.sh/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Kernel does not do - Kernel does not sign customer sessions with its identity unless it approves a request on an eligible plan; other sessions rely on stealth mode and residential proxies. - Kernel publishes no IP list for KernelSearchBot or Kernel Agent (bots page read 2026-11-05). ## Reading notes KernelSearchBot and Kernel Agent are not rows in the [Crawler Index](https://agentreadycrawlers.com/crawlers/) yet. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [kernel.sh](https://www.kernel.sh) read 2026-11-05 - [KERNEL Bots and Agents | Kernel docs](https://www.kernel.sh/docs/bots.md) read 2026-11-05 - [Web Bot Auth (WBA) | Kernel docs](https://www.kernel.sh/docs/browsers/bot-detection/web-bot-auth.md) read 2026-11-05 - [Kernel Agent key directory](https://www.kernel.sh/.well-known/http-message-signatures-directory) read 2026-11-05 - [Pricing | Kernel](https://www.kernel.sh/pricing) read 2026-11-05 - [Kernel Search key directory](https://search.bot.kernel.sh/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Known Agents: bot management vendor profile | Agent Ready Crawlers" description: "Known Agents: bot management vendor profile covering Agent Analytics, Automatic Robots.txt, Bad Bot Blocking and 1 more. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/known-agents site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Known Agents: bot management vendor profile | Agent Ready Crawlers > Known Agents: bot management vendor profile covering Agent Analytics, Automatic Robots.txt, Bad Bot Blocking and 1 more. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/known-agents Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Known Agents In short Known Agents began in 2023 as Dark Visitors, a side project listing eight AI bots, and now serves a public agent directory that anyone can submit to. Customers feed it traffic from server-side hooks in CDNs and clouds; a free tier of one project suits small sites, and Growth costs USD 299 a month. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [knownagents.com](https://knownagents.com) In one line Agent analytics and access-control service, formerly Dark Visitors, that identifies AI agents and crawlers from server-side data, keeps an automatic robots.txt current, and blocks bots that ignore it or spoof trusted identities. Last read 2026-11-05 ## What Known Agents does for this layer ### From the vendor's own pages Known Agents says it began in 2023 as Dark Visitors, a side project documenting the eight known AI bots then on the web, and was founded in New York City. ([knownagents.com](https://knownagents.com/about)) Known Agents' Automatic Robots.txt keeps crawler rules current as new agents appear, and its Bad Bot Blocking enforces them against bots that ignore robots.txt or spoof trusted identities. ([knownagents.com](https://knownagents.com/about)) Known Agents collects traffic server-side through WordPress, Cloudflare, Fastly, Akamai, AWS, Google Cloud, Vercel, Node.js and REST API integrations. ([knownagents.com](https://knownagents.com/about)) Known Agents' Agent Identification returns fields such as agent\_token, agent\_type\_name and operator\_name with a verification result for each request. ([knownagents.com](https://knownagents.com/)) Known Agents prices Free at USD 0 for one project and 100,000 events, Startup at USD 29 a month for 1 million events, and Growth at USD 299 a month for 10 million events. ([knownagents.com](https://knownagents.com/pricing)) Known Agents runs a public agent directory with a Submit an Agent form, listing entries such as Google-Agent, ElevenlabsBot and QualifiedBot. ([knownagents.com](https://knownagents.com/agents)) ### Claims the vendor makes about itself Known Agents says more than 5,000 websites use the platform. ([knownagents.com](https://knownagents.com/about)) Known Agents' about page states "We are not an anti-bot company." ([knownagents.com](https://knownagents.com/about)) ### What our crawl saw darkvisitors.com redirected to knownagents.com when fetched with the crawl user agent on 2026-11-05. ([darkvisitors.com](https://darkvisitors.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Agent Analytics | Server-side analytics of AI agents, crawlers and scrapers, including robots.txt violations. | monthly by events Startup USD 29, Growth USD 299 a month; Enterprise custom. | USD 0 | 2026-11-05 | | Automatic Robots.txt | Robots.txt rules that update as new bots are found. | No price recorded | | Bad Bot Blocking | Request-time blocking of bots that ignore robots.txt or spoof identities. | No price recorded | | Agent Identification | API that names and verifies agents from incoming requests. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Automatic Robots.txt publishes and updates robots.txt rules for compliant crawlers. (knownagents.com) | | Web Bot Auth | No statement | Known Agents pages read 2026-11-05 do not name the verification method. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Known Agents does not do - Known Agents' pages read do not say how it verifies agents; Web Bot Auth, IP lists and reverse DNS are not named (read 2026-11-05). - Known Agents does not charge AI crawlers for access on a site owner's behalf. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) no token ## Sources - [knownagents.com](https://knownagents.com) read 2026-11-05 - [Dark Visitors (redirects to Known Agents)](https://darkvisitors.com/) read 2026-11-05 - [About Known Agents | A Toolkit for the Agentic Web](https://knownagents.com/about) read 2026-11-05 - [Track, control, and optimize for AI agents and bots | Known Agents](https://knownagents.com/) read 2026-11-05 - [Pricing | Known Agents](https://knownagents.com/pricing) read 2026-11-05 - [Agent Directory | Known Agents](https://knownagents.com/agents) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Link11: bot management vendor profile | Agent Ready Crawlers" description: "Link11: bot management vendor profile covering Link11 WAAP (WAF, API Security, Bot Management, Web DDoS). Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/link11 site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Link11: bot management vendor profile | Agent Ready Crawlers > Link11: bot management vendor profile covering Link11 WAAP (WAF, API Security, Bot Management, Web DDoS). Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/link11 Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Link11 In short Link11 is a German provider that sells bot controls as one module of a wider WAAP platform. Customers choose whether AI crawlers are watched or blocked, and our crawl was served an empty body with a non-standard status from the Bot Management page. No price is published, and the platform also protects its own DNS against DDoS. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Based in Germany Homepage [link11.com](https://www.link11.com) In one line German security provider whose WAAP platform joins WAF, API security, Bot Management and Web DDoS protection, with a setting to monitor or block AI crawlers; the reblaze.com domain now redirects to Link11. Last read 2026-11-05 ## What Link11 does for this layer ### From the vendor's own pages Link11's homepage says customers decide whether AI crawlers are monitored or blocked in its Bot Management. ([link11.com](https://www.link11.com/en/)) Link11 runs WAF, API Security, Bot Management and Web DDoS Protection as four modules under one interface, one policy model and one operating team. ([link11.com](https://www.link11.com/en/)) Link11 lists ISO 27001 and BSI C5 on its homepage. ([link11.com](https://www.link11.com/en/)) Link11 serves static content from the edge and offers secure DNS that is itself protected against DDoS. ([link11.com](https://www.link11.com/en/)) ### Claims the vendor makes about itself Link11 describes its security as European-developed and cites more than 20 years of experience and patented DDoS protection. ([link11.com](https://www.link11.com/en/)) ### What our crawl saw A GET to www.reblaze.com with the crawl user agent on 2026-11-05 redirected to www.link11.com/en/. ([reblaze.com](https://www.reblaze.com/)) Link11's Bot Management product page returned HTTP status 247 with an empty body to the crawl user agent on 2026-11-05. ([link11.com](https://www.link11.com/en/products/bot-management/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Link11 WAAP (WAF, API Security, Bot Management, Web DDoS) | Four protection modules under one policy model, with AI crawler monitor or block. | sales-led No price on the homepage. | Not published | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | Link11 pages read 2026-11-05 name no verification method for AI crawlers. | | Visa Trusted Agent Protocol | No statement | No statement on the Link11 pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the Link11 pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Link11 does not do - Link11's pages read offer monitor or block for AI crawlers and name no way to verify a crawler's identity (read 2026-11-05). - Link11 publishes no price. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [link11.com](https://www.link11.com) read 2026-11-05 - [Link11: Cybersecurity & DDoS Protection from Europe](https://www.link11.com/en/) read 2026-11-05 - [Reblaze (redirects to Link11)](https://www.reblaze.com/) read 2026-11-05 - [Link11 Bot Management (challenge response to the crawl user agent)](https://www.link11.com/en/products/bot-management/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Manus: crawler operator profile | Agent Ready Crawlers" description: "Manus: crawler operator profile covering Manus Cloud Browser and Manus Browser Operator. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/manus site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Manus: crawler operator profile | Agent Ready Crawlers > Manus: crawler operator profile covering Manus Cloud Browser and Manus Browser Operator. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/manus Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Manus In short Manus told users on 29 December 2025 that it was joining Meta, while staying based in Singapore. It asks the user to take over when a CAPTCHA or an SMS code stops the agent, keeps each user's sessions in a separate isolated browser, and works in logged-in tabs only after the user authorises a session. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Operator Meta Based in Singapore Homepage [manus.im](https://manus.im) In one line General-purpose AI agent, owned by Meta since December 2025, that browses either in an isolated cloud browser or through an extension inside the user's own browser, and publishes no identifier, robots.txt policy or signing keys for site owners. Last read 2026-11-05 ## What Manus does for this layer ### From the vendor's own pages Manus announced on 2025-12-29 that it is joining Meta and said the company would continue to operate from Singapore and keep selling its subscription through its own app and website. ([manus.im](https://manus.im/blog/manus-joins-meta-for-next-era-of-innovation), dated 2025-12-29) Manus browses in two modes: Cloud Browser, an isolated sandboxed browser in the cloud, and Browser Operator, an extension that works inside the user's local browser with existing logins and sessions. ([manus.im](https://manus.im/docs/features/browser-operator)) Manus's Browser Operator docs say using the local browser "avoids CAPTCHAs and security checks" on sites where the user is already logged in. ([manus.im](https://manus.im/docs/features/browser-operator)) When Cloud Browser meets SMS codes, CAPTCHAs or multi-factor prompts, Manus asks the user to "Take Over" the browser, complete the check and hand control back. ([manus.im](https://manus.im/docs/features/cloud-browser)) Manus lets users log into personal or work accounts inside Cloud Browser so the agent can act with those sessions, and gives each user a separate isolated browser instance. ([manus.im](https://manus.im/docs/features/cloud-browser)) Manus's Browser Operator requires the user to authorise every session, and the user can stop the agent by closing its browser tab. ([manus.im](https://manus.im/docs/features/browser-operator)) ### Claims the vendor makes about itself Manus says its agent had processed more than 147 trillion tokens and powered over 80 million virtual computers by the time of the Meta announcement. ([manus.im](https://manus.im/blog/manus-joins-meta-for-next-era-of-innovation), dated 2025-12-29) Manus's homepage, read on 2026-11-05, introduces Manus 2.0 and lists slides, websites, games, video and design among the tasks the agent takes on. ([manus.im](https://manus.im/)) ### What our crawl saw A GET for manus.im/.well-known/http-message-signatures-directory returned HTTP 404 on 2026-11-05. ([manus.im](https://manus.im/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Manus Cloud Browser | Isolated cloud browser the agent operates, with user takeover for verification steps. | No price recorded | | Manus Browser Operator | Browser extension that lets the agent work in the user's own browser and logged-in tabs. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | No robots.txt statement on the docs pages read (2026-11-05). | | Web Bot Auth | No statement | No key directory at manus.im (HTTP 404, 2026-11-05). | | Published IP list | No statement | No IP list published for Cloud Browser traffic (2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://manus.im/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Manus does not do - Manus publishes no user agent string, robots.txt token, IP list or signature key directory for either browser mode (checked 2026-11-05). - Manus Browser Operator works inside the user's own browser with the user's logins, so its requests carry that browser's identity rather than a Manus identifier. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Manus cloud browser](https://agentreadycrawlers.com/crawlers/manus/) no token ## Sources - [manus.im](https://manus.im) read 2026-11-05 - [Manus Joins Meta for Next Era of Innovation](https://manus.im/blog/manus-joins-meta-for-next-era-of-innovation) read 2026-11-05 - [Browser Operator - Manus Documentation](https://manus.im/docs/features/browser-operator) read 2026-11-05 - [Cloud browser - Manus Documentation](https://manus.im/docs/features/cloud-browser) read 2026-11-05 - [Manus: Hands On AI](https://manus.im/) read 2026-11-05 - [Manus key directory probe](https://manus.im/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Mastercard: vendor profile | Agent Ready Crawlers" description: "Mastercard: agent verification provider profile covering Mastercard Agent Pay and Developer Playbook on Preparing for Agentic Commerce. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/mastercard site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Mastercard: vendor profile | Agent Ready Crawlers > Mastercard: agent verification provider profile covering Mastercard Agent Pay and Developer Playbook on Preparing for Agentic Commerce. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/mastercard Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Mastercard In short Mastercard tells merchants not to block consumer-owned agents wholesale, comparing that with blocking mobile browsers in 2010, and to add friction at checkout instead. In the simplest integration the merchant sees ordinary card fields filled with tokenized credentials and can check verification data from request headers. The site refused our crawl, and the playbook's example key directory does not resolve. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/) In one line Card network whose Agent Pay registers AI agents and has them sign requests with Web Bot Auth under the tag Agent-pay-auth, so a merchant or its CDN can check the agent's key in a network directory before letting it buy. Last read 2026-11-05 ## What Mastercard does for this layer ### From the vendor's own pages Mastercard's Developer Playbook on Preparing for Agentic Commerce names three network frameworks a merchant will meet: Mastercard Agent Pay, Visa Intelligent Commerce with the Trusted Agent Protocol, and the American Express ACE Developer Kit with Agent Purchase Protection. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/)) Mastercard's playbook lists Agent-pay-auth as the Signature-Input tag for Mastercard Agent Pay, next to Visa's agent-browser-auth for browsing and agent-payer-auth for checkout. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/)) The example signed request in Mastercard's playbook carries the header Signature-Agent: "https://agentpay-key-directory.mastercard.com/" with an Ed25519 keyid, created and expires timestamps and a nonce. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/)) Mastercard's playbook says agents that implement Web Bot Auth on their own, such as ChatGPT, use the generic web-bot-auth tag, and tells merchants to accept both the scheme tags and the generic one. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/)) In Mastercard's registration model an agent creates a key pair at onboarding and registers its public key in a payment network key directory; the merchant's edge provider or backend fetches that key to validate each signature. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/)) Mastercard's playbook names Cloudflare, Akamai and Fastly as edge providers that support Web Bot Auth verification at the network edge, as the alternative to verifying in application code. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/)) At Mastercard's Level 1, an agent using Agent Pay types tokenized payment credentials into the merchant's normal checkout fields while hash verification data arrives in HTTP headers that the merchant backend may validate. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/)) Mastercard's playbook calls indiscriminate blocking of consumer-owned agents such as OpenClaw, Agent Zero and Hermes Agent "the equivalent of blocking mobile browsers in 2010" and recommends open browsing with progressive friction at checkout. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/)) Mastercard also describes merchant registration: the acquirer or PSP submits the merchant to the network's agentic commerce registry, and once verified the merchant appears in a trusted merchant directory that certified agents query at checkout. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/22/)) ### Reported by a third party Cloudflare wrote on 24 Oct 2025 that, to start, Visa and Mastercard host their own key directories for Visa-registered and Mastercard-registered agents. ([blog.cloudflare.com](https://blog.cloudflare.com/secure-agentic-commerce/), dated 2025-10-24) ### What our crawl saw On 2026-11-05 a GET of https://www.mastercard.com/robots.txt with the crawl user agent returned HTTP 403 with an Akamai Access Denied page, so Mastercard's Agent Pay product pages on that host were not read for this profile. ([mastercard.com](https://www.mastercard.com/robots.txt)) On 2026-11-05 the host agentpay-key-directory.mastercard.com, used in the playbook's example Signature-Agent header, did not resolve in DNS, so no Mastercard agent key directory could be fetched. ([developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Mastercard Agent Pay | Agent registration and Web Bot Auth signing with the Agent-pay-auth tag, with tokenized credentials entered at checkout. | No price recorded | | Developer Playbook on Preparing for Agentic Commerce | Public merchant guide on developer.mastercard.com covering three readiness levels, Web Bot Auth verification and merchant registration. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | The playbook builds agent identification on Web Bot Auth and RFC 9421 and accepts the generic web-bot-auth tag. (developer.mastercard.com) | | Visa Trusted Agent Protocol | Partial | The playbook documents Visa's tags so merchants can verify both schemes; Mastercard does not run TAP. (developer.mastercard.com) | | Mastercard Agent Pay | Supports | Mastercard defines the Agent Pay tag Agent-pay-auth and the agent key directory model in its merchant playbook. (developer.mastercard.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Crawl access to www.mastercard.com | HTTP 403, Akamai Access Denied, robots.txt included | GET https://www.mastercard.com/robots.txt with the crawl user agent | 2026-11-05 | | Agent Pay key directory host | agentpay-key-directory.mastercard.com does not resolve | DNS lookup and GET of /.well-known/http-message-signatures-directory | 2026-11-05 | ## What Mastercard does not do - Mastercard does not verify signatures for the merchant; the merchant's CDN or backend does the check against the network's key directory. - Mastercard publishes no public list of registered agents or a reachable key directory URL that a site can test against (checked 2026-11-05). - Agent Pay covers agents with payment intent; it gives no signal for training or search crawlers. ## Reading notes www.mastercard.com and businesswire.com refused our crawl user agent (a 403, and a robots.txt disallow), so press releases were not read; the facts come from developer.mastercard.com, which allows the crawl and serves markdown. ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management - [Akamai (Bot & Agent Control)](https://agentreadycrawlers.com/vendors/akamai/) bot management - [Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/) bot management - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token ## Sources - [Developer Playbook on Preparing for Agentic Commerce (Mastercard Developers)](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/) read 2026-11-05 - [Mastercard agentic commerce guide, Level 1: identifying agent traffic with Web Bot Auth](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/21/) read 2026-11-05 - [Mastercard agentic commerce guide, Level 2: merchant registration and identity](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/22/) read 2026-11-05 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard | Cloudflare Blog](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-05 - [www.mastercard.com robots.txt (403 to the crawl user agent)](https://www.mastercard.com/robots.txt) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Meta: crawler operator profile | Agent Ready Crawlers" description: "Meta: crawler operator profile covering Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher and 2 more. Sources, standards and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/meta site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Meta: crawler operator profile | Agent Ready Crawlers > Meta: crawler operator profile covering Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher and 2 more. Sources, standards and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/meta Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Meta In short Meta lists five crawlers, for model training, Meta AI search, user-driven fetching, advertising and link previews. It adds that two of them may skip robots.txt, and that changes can take up to 24 hours to be seen. Five Web Bot Auth keys are served from www.meta.com, but no Meta page says which agent uses them. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [meta.com](https://www.meta.com) In one line Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt. Last read 2026-11-05 ## What Meta does for this layer ### From the vendor's own pages Meta says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) Meta says Meta-WebIndexer crawls to improve Meta AI search results and that allowing it helps Meta cite and link to a site in Meta AI responses. ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) Meta says Meta-ExternalFetcher fetches links at a user's request, including to help AI complete tasks on websites, and "may bypass robots.txt rules". ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) Meta says FacebookExternalHit might bypass robots.txt when it runs security or integrity checks, such as checking for malware. ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) Meta asks site owners to allow up to 24 hours for robots.txt changes, because its crawlers may cache the file for that long. ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) Meta's page says it prefers robots.txt over non-standard formats such as NoAI tags for publishers to state their preferences. ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) Meta says FacebookExternalHit needs Open Graph properties within the first 1 MB of a page and servers that support gzip and deflate encodings. ([developers.facebook.com](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)) ### Reported by a third party Verisoul reported on 2026-11-01 that Meta publishes no user agent token, signature or IP list for its Muse agent, which browses from Meta's cloud as desktop Chrome on Linux. ([verisoul.ai](https://www.verisoul.ai/agents/meta-muse), dated 2026-11-01) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Meta-ExternalAgent | Crawler for AI model training and direct indexing. | No price recorded | | Meta-WebIndexer | Crawler for Meta AI search results and citations. | No price recorded | | Meta-ExternalFetcher | User-requested fetcher that may bypass robots.txt. | No price recorded | | Meta-ExternalAds | Crawler for advertising and business products. | No price recorded | | FacebookExternalHit | Link preview crawler for content shared in Meta's apps. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | Meta asks publishers to use robots.txt; Meta-ExternalFetcher and FacebookExternalHit may bypass it. (developers.facebook.com) | | Web Bot Auth | No statement | Meta's crawler page says nothing about signatures, although www.meta.com served a key directory on 2026-11-05 (observed). | | Published IP list | No statement | Meta's crawler page links no IP range list (read 2026-11-05). | | Reverse DNS check | No statement | No reverse DNS pattern on the crawler page (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | www.meta.com: 200, application/http-message-signatures-directory+json, 5 keys (undocumented) | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | ## What Meta does not do - Meta's crawler page links no IP range list, so its documented tokens cannot be checked against addresses from that page. - Meta's crawler page does not say which Meta agent, if any, signs with the keys served at www.meta.com. - Meta's current crawler page no longer lists FacebookBot (read 2026-11-05). ## Reading notes Meta's robots.txt at www.facebook.com disallows our crawl token for the .well-known path, so we did not probe it for a key directory; the five keys at www.meta.com were found instead. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Meta in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/meta/) 7 rows - [Meta-ExternalAgent](https://agentreadycrawlers.com/crawlers/meta-externalagent/) token meta-externalagent - [Meta-ExternalFetcher](https://agentreadycrawlers.com/crawlers/meta-externalfetcher/) token meta-externalfetcher - [Meta-WebIndexer](https://agentreadycrawlers.com/crawlers/meta-webindexer/) token meta-webindexer - [Meta-ExternalAds](https://agentreadycrawlers.com/crawlers/meta-externalads/) token meta-externalads - [FacebookExternalHit](https://agentreadycrawlers.com/crawlers/facebookexternalhit/) token facebookexternalhit - [FacebookBot (legacy token)](https://agentreadycrawlers.com/crawlers/facebookbot/) token FacebookBot - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) no token ## Sources - [meta.com](https://www.meta.com) read 2026-11-05 - [Meta Web Crawlers](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) read 2026-11-05 - [www.meta.com Web Bot Auth key directory path (GET probe)](https://www.meta.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [How to detect Meta Muse (Verisoul Agent Atlas)](https://www.verisoul.ai/agents/meta-muse) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Microsoft: crawler operator profile | Agent Ready Crawlers" description: "Microsoft: crawler operator profile covering Bingbot, AI Training and AI Grounding preferences and Publisher Content Marketplace. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/microsoft site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Microsoft: crawler operator profile | Agent Ready Crawlers > Microsoft: crawler operator profile covering Bingbot, AI Training and AI Grounding preferences and Publisher Content Marketplace. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/microsoft Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Microsoft In short From 1 February 2027 Bingbot will honour a shared training preference written into robots.txt, while a page-level grounding preference, set with a meta tag or header, can keep content out of services sold to third parties. Neither signal governs crawl access or sets up payment, and bingbot.json lists 29 IPv4 prefixes. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [microsoft.com](https://www.microsoft.com) In one line Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding. Last read 2026-11-05 ## What Microsoft does for this layer ### From the vendor's own pages Microsoft announced on 2026-10-30 that from 2027-02-01 it will honour the shared IETF AI Training preference in robots.txt, written as a group for User-Agent: \_\_usage\_ai-training. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), dated 2026-10-30) Microsoft adds a page-level AI Grounding preference with three values, Allow, DisallowExtended and Disallow, where DisallowExtended keeps grounding to Microsoft-run experiences such as Copilot and Microsoft 365 and excludes grounding services sold to third parties. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), dated 2026-10-30) Microsoft says NoArchive and NoCache stop being read as training or grounding preferences after 2027-03-01, and asks sites that rely on them to publish the new preferences before then. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), dated 2027-03-01) Microsoft says both preferences default to Allow, do not change eligibility for traditional Bing results, and do not set up licensing or payment. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)) Microsoft says the Bing search syndication API returns links for traditional search and is not affected by the grounding preference. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)) Microsoft's page-level syntax is a meta tag named Bingbot or an X-Robots-Tag header, for example content="ai-training:disallow" or ai-grounding:disallowextended. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)) Microsoft says its training and grounding settings govern use of content, not crawl access, so an Allow does not override a robots.txt rule that blocks Bingbot. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)) Microsoft opens a temporary grounding preference in Bing Webmaster Tools on 2026-12-15 for sites still using NoArchive or NoCache, and any such temporary preference expires on 2027-09-30. ([blogs.bing.com](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), dated 2026-12-15) Microsoft launched the Publisher Content Marketplace on 2026-02-03, a pilot in which publishers set licensing and usage terms, AI builders license content for grounding, and publishers are "paid on delivered value". ([about.ads.microsoft.com](https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web), dated 2026-02-03) ### Claims the vendor makes about itself Microsoft says it co-designed the marketplace with publishers including The Associated Press, Condé Nast, Hearst Magazines, People Inc, USA TODAY Co. and Vox Media, began with grounding in Microsoft Copilot, and is onboarding Yahoo as a demand partner. ([about.ads.microsoft.com](https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web), dated 2026-02-03) ### What our crawl saw Microsoft's bingbot.json listed 29 IPv4 prefixes with a creationTime of 2024-01-03 when fetched on 2026-11-05. ([bing.com](https://www.bing.com/toolbox/bingbot.json), dated 2024-01-03) Neither www.bing.com nor www.microsoft.com served a Web Bot Auth key directory on 2026-11-05; both returned HTTP 404. ([bing.com](https://www.bing.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Bingbot | Crawler for Bing search and the web grounding behind Copilot and Microsoft 365. | No price recorded | | AI Training and AI Grounding preferences | Robots.txt and page-level controls read from 2027-02-01 that separate training use from grounding use. | No price recorded | | Publisher Content Marketplace | Pilot marketplace where publishers set terms and AI builders license content for grounding, with usage-based reporting. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Microsoft says an Allow preference does not override a robots.txt rule blocking Bingbot, and reads the AI Training preference from robots.txt groups from 2027-02-01. (blogs.bing.com) | | Web Bot Auth | No statement | No key directory on www.bing.com or www.microsoft.com (observed 404, 2026-11-05). | | Content Signals | No statement | Microsoft's October 2026 post cites the IETF AI Preferences draft, not Content-Signals. | | Published IP list | Supports | bingbot.json published at bing.com/toolbox (29 IPv4 prefixes observed 2026-11-05). (bing.com) | | Reverse DNS check | No statement | Bing's crawler help pages are JavaScript-only and could not be read with the crawl user agent (2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 on www.bing.com and www.microsoft.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | bingbot.json | 29 IPv4 prefixes, creationTime 2024-01-03 | GET and count prefixes | 2026-11-05 | ## What Microsoft does not do - Microsoft accepts the AI Grounding preference only at page level; it cannot be set in robots.txt. - Microsoft's October 2026 post names no separate Copilot user agent, so Copilot grounding is controlled through the preferences rather than by blocking a token. - Microsoft's preferences take effect on 2027-02-01, so a robots.txt training rule written today is not yet read by Bingbot as a training preference. ## Reading notes Bing's webmaster crawler help pages render only with JavaScript, so Bingbot's reverse DNS and token details were not re-read from them; see the [Bingbot row](https://agentreadycrawlers.com/crawlers/bingbot/). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Microsoft in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/microsoft/) 3 rows - [Bingbot](https://agentreadycrawlers.com/crawlers/bingbot/) token bingbot - [MicrosoftPreview](https://agentreadycrawlers.com/crawlers/microsoftpreview/) token MicrosoftPreview - [Browse with Copilot (formerly Copilot Actions)](https://agentreadycrawlers.com/crawlers/copilot-actions/) no token ## Sources - [microsoft.com](https://www.microsoft.com) read 2026-11-05 - [Your content, your choice: clearer preferences for training and grounding (Bing blog)](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) read 2026-11-05 - [Building Toward a Sustainable Content Economy for the Agentic Web (Microsoft Advertising)](https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web) read 2026-11-05 - [bingbot.json IP ranges](https://www.bing.com/toolbox/bingbot.json) read 2026-11-05 - [www.bing.com Web Bot Auth key directory path (GET probe)](https://www.bing.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Microsoft Entra Agent ID: vendor profile | Agent Ready Crawlers" description: "Microsoft Entra Agent ID: agent verification provider profile covering Microsoft Entra Agent ID. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/microsoft-entra-agent-id site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Microsoft Entra Agent ID: vendor profile | Agent Ready Crawlers > Microsoft Entra Agent ID: agent verification provider profile covering Microsoft Entra Agent ID. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/microsoft-entra-agent-id Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Microsoft Entra Agent ID In short Microsoft Entra Agent ID treats an agent as a service principal that holds no credentials of its own: a blueprint issues the federated credentials it authenticates with, and passwords, SMS codes and passkeys play no part. A sponsor, a person or group, answers when something goes wrong. Tokens are valid only in the tenant that made the identity. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Operator Microsoft Homepage [learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities) In one line Microsoft's identity service for AI agents inside an Entra tenant: each agent gets its own service principal created from a blueprint, with tokens, a human sponsor and audit, for agents an organisation runs rather than agents visiting a website. Last read 2026-11-05 ## What Microsoft Entra Agent ID does for this layer ### From the vendor's own pages Microsoft defines an Entra agent identity as a special service principal in Microsoft Entra ID that an agent identity blueprint creates and is authorised to impersonate. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) Microsoft Entra agent identities hold no credentials of their own and authenticate only through federated identity credentials issued by the blueprint, which holds certificates, keys, client secrets or federated credentials. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) Microsoft states that AI agents in Entra do not use passwords, SMS, passkeys or authenticator apps to authenticate. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) An Entra agent identity can request tokens where it is the subject, receive tokens where it is the audience, and request user tokens where a user is the subject and the agent is the actor. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) Each Entra agent identity can carry a sponsor, the human user or group accountable for the agent, who is contacted in a security incident. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) Entra agent identities can receive tokens only in the tenant where they were created, while a blueprint can be multitenant and create tenant-local agent identities in each tenant that adds it. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) Some Entra agents get a second, optional agent's user account, a tenant user account marked as an AI agent and tied one-to-one to the agent identity, for systems that require a user account. ([learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Microsoft Entra Agent ID | Agent identities, blueprints, sponsors and agent user accounts in Microsoft Entra, managed in the admin center or through Microsoft Graph. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The agent identity overview covers Entra tokens only and does not mention signing HTTP requests to third-party websites (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Microsoft Entra Agent ID does not do - Microsoft Entra Agent ID does not let an outside website verify a visiting agent; its tokens are issued and accepted inside Entra tenants. - Microsoft Entra Agent ID gives an agent no way to sign requests to third-party websites; the overview lists only requesting tokens, receiving tokens and on-behalf-of user tokens. ## Reading notes Microsoft as a crawler operator, with Bingbot, MicrosoftPreview and Copilot Actions, has [its own profile](https://agentreadycrawlers.com/vendors/microsoft/); this one covers the Entra identity product only. ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Bingbot](https://agentreadycrawlers.com/crawlers/bingbot/) token bingbot - [Browse with Copilot (formerly Copilot Actions)](https://agentreadycrawlers.com/crawlers/copilot-actions/) no token ## Sources - [Overview of agent identities in Microsoft Entra (Microsoft Learn)](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Mistral AI: crawler operator profile | Agent Ready Crawlers" description: "Mistral AI: crawler operator profile covering MistralAI-User, MistralAI-Index and MistralAI-Training. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/mistral-ai site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Mistral AI: crawler operator profile | Agent Ready Crawlers > Mistral AI: crawler operator profile covering MistralAI-User, MistralAI-Index and MistralAI-Training. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/mistral-ai Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Mistral AI In short Mistral AI documents full user agent strings for all three of its crawlers and IP files for two of them, listing four prefixes for the user fetcher and two for the index crawler. Nothing but its name identifies the training crawler. Mistral says the data its index crawler gathers never goes into generative training. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [mistral.ai](https://mistral.ai) In one line Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two. Last read 2026-11-05 ## What Mistral AI does for this layer ### From the vendor's own pages Mistral's crawler page lists three user agents, MistralAI-User, MistralAI-Index and MistralAI-Training, each with its own robots.txt tag. ([docs.mistral.ai](https://docs.mistral.ai/robots)) Mistral says MistralAI-User acts on user requests in Vibe and is not used for automatic crawling or to collect content for generative AI training. ([docs.mistral.ai](https://docs.mistral.ai/robots)) Mistral says content crawled by MistralAI-Index, which feeds Mistral search in Vibe, is not used for generative AI training of any kind. ([docs.mistral.ai](https://docs.mistral.ai/robots)) Mistral says MistralAI-Training builds datasets for training its generative models, is not used for search or live answers, and can be disallowed in robots.txt. ([docs.mistral.ai](https://docs.mistral.ai/robots)) Mistral publishes full user agent strings, such as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots). ([docs.mistral.ai](https://docs.mistral.ai/robots)) ### Claims the vendor makes about itself Mistral says on its about page that the company was founded in April 2023. ([mistral.ai](https://mistral.ai/about)) ### What our crawl saw Mistral's mistralai-user-ips.json held 4 IPv4 prefixes with a creationTime of 2025-02-19 on 2026-11-05. ([mistral.ai](https://mistral.ai/mistralai-user-ips.json), dated 2025-02-19) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | MistralAI-User | Fetcher for pages a Vibe user's question needs. | No price recorded | | MistralAI-Index | Crawler for the search index behind Vibe. | No price recorded | | MistralAI-Training | Crawler that gathers training data for Mistral's generative models. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Mistral uses a robots.txt tag per crawler; the MistralAI-User tag governs which sites user requests can reach, and MistralAI-Training can be disallowed. (docs.mistral.ai) | | Web Bot Auth | No statement | Not on the crawler page; key directory path 404 (observed 2026-11-05). | | Published IP list | Partial | IP files for MistralAI-User and MistralAI-Index; none listed for MistralAI-Training. (docs.mistral.ai) | | Reverse DNS check | No statement | No reverse DNS pattern on the crawler page (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | mistralai-user-ips.json | 4 IPv4 prefixes, creationTime 2025-02-19 | GET and count prefixes | 2026-11-05 | | mistralai-index-ips.json | 2 IPv4 prefixes, creationTime 2026-04-19 | GET and count prefixes | 2026-11-05 | ## What Mistral AI does not do - Mistral lists no IP file for MistralAI-Training, so the training crawler can be checked by user agent only (read 2026-11-05). - Mistral's crawler page gives no reverse DNS pattern or signing keys for any of its three crawlers (read 2026-11-05). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Mistral AI in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/mistral-ai/) 3 rows - [MistralAI-User](https://agentreadycrawlers.com/crawlers/mistralai-user/) token MistralAI-User - [MistralAI-Index](https://agentreadycrawlers.com/crawlers/mistralai-index/) token MistralAI-Index - [MistralAI-Training](https://agentreadycrawlers.com/crawlers/mistralai-training/) token MistralAI-Training ## Sources - [mistral.ai](https://mistral.ai) read 2026-11-05 - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-05 - [MistralAI-User IP addresses](https://mistral.ai/mistralai-user-ips.json) read 2026-11-05 - [MistralAI-Index IP addresses](https://mistral.ai/mistralai-index-ips.json) read 2026-11-05 - [About Mistral](https://mistral.ai/about) read 2026-11-05 - [mistral.ai Web Bot Auth key directory path (GET probe)](https://mistral.ai/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Netacea: bot management vendor profile | Agent Ready Crawlers" description: "Netacea: bot management vendor profile covering Netacea Bot Protection. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/netacea site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Netacea: bot management vendor profile | Agent Ready Crawlers > Netacea: bot management vendor profile covering Netacea Bot Protection. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/netacea Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Netacea In short Netacea installs at the CDN or load balancer as an edge function, with no client code, and leaves the site running if the service fails. In a case study for a fashion retailer it claims to find 22 times as many bots as Akamai's product, and its pages name no Web Bot Auth or other signature check. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [netacea.com](https://netacea.com) In one line Server-side bot protection vendor that integrates at the CDN or load balancer without client code, classifies humans, good bots, bad bots and AI agents, and enforces access by intent rather than declared identity. Last read 2026-11-05 ## What Netacea does for this layer ### From the vendor's own pages Netacea runs as an Akamai EdgeWorker with DataStream 2 logs, a Cloudflare Worker, an AWS Lambda@Edge function, Fastly VCL with real-time log streaming, or an F5 LX iRule. ([netacea.com](https://netacea.com/platform/integrations/)) Netacea describes its integration as agentless and says the service fails open rather than halting the site. ([netacea.com](https://netacea.com/)) ### Claims the vendor makes about itself Netacea's platform page says it governs bots and AI agents by "enforcing access based on intent, not identity." ([netacea.com](https://netacea.com/platform/)) Netacea says its detection models are trained on hundreds of billions of weekly requests. ([netacea.com](https://netacea.com/platform/)) Netacea claims a 0.001% false positive rate and bot blocking 33 times more effective than competitors. ([netacea.com](https://netacea.com/)) Netacea's case studies claim 22 times more bots detected than Akamai Bot Manager for a fashion retailer and 33 times more than HUMAN for an education platform. ([netacea.com](https://netacea.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Netacea Bot Protection | Server-side, agentless bot detection and response for web, app and API traffic. | sales-led Homepage leads to a demo booking. | Not published | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | Netacea pages read 2026-11-05 name no cryptographic agent verification. | | Visa Trusted Agent Protocol | No statement | No statement on the Netacea pages read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement on the Netacea pages read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Netacea does not do - Netacea's pages read name no Web Bot Auth or other signature-based agent verification (read 2026-11-05). - Netacea publishes no price and offers no AI crawler licensing route. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [netacea.com](https://netacea.com) read 2026-11-05 - [Platform | Netacea](https://netacea.com/platform/) read 2026-11-05 - [Integrations | Netacea](https://netacea.com/platform/integrations/) read 2026-11-05 - [Home | Netacea](https://netacea.com/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Netlify: bot management profile | Agent Ready Crawlers" description: "Netlify: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 15 of 15." canonical: https://agentreadycrawlers.com/vendors/netlify site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Netlify: bot management profile | Agent Ready Crawlers > Netlify: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 15 of 15. Source: https://agentreadycrawlers.com/vendors/netlify Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 15 of 15 # Netlify In short Netlify's bot controls are a User Agent Blocker extension and firewall traffic rules; only per-purpose policy grades partial, for 1 of 10 points, and every other question is not published. Platform plans start at USD 0 but no price is stated for the extension, and there is no verified-bot list, reverse DNS or Web Bot Auth check. Web hosting platform whose AI bot control is a User Agent Blocker extension running as an Edge Function, alongside IP and geo firewall traffic rules and rate limiting; it has no verified-bot or signature checking. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Netlify](https://agentreadycrawlers.com/vendors/netlify/alternatives/) · [netlify.com](https://www.netlify.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [netlify.com](https://www.netlify.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Not published | support (checked 2026-11-04): blocking matches user-agent strings only | Block AI crawlers docs mention no verified bots or Web Bot Auth. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | choose AI crawlers versus SEO or search crawlers from a preset list | User Agent Blocker settings. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | No monetization feature in docs read. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | support (checked 2026-11-04) | Not mentioned in docs read. Source, read 2026-11-04. | | Cost | Not graded | published: platform plans from USD 0; extension price not stated | Pricing page. Source, read 2026-11-04. | | Reporting | Not published | no AI bot reporting found (checked 2026-11-04) | Docs read describe blocking only. Source, read 2026-11-04. | | Where it runs | Not graded | edge: Netlify Edge Function and edge firewall rules | Extension uses an Edge Function. Source, read 2026-11-04. | ## Cost Free USD 0 (300 credits), Personal USD 9/month (1,000 credits), Pro USD 20/month (3,000 credits), Enterprise custom; Firewall Traffic Rules and basic rate limiting listed from Free; User Agent Blocker price not stated. [Source](https://www.netlify.com/pricing/), read 2026-11-04. ## What Netlify does not do - No verified-bot list, reverse DNS or Web Bot Auth checks; user agents can be spoofed (block AI crawlers docs). - Preset list is incomplete by Netlify's own statement (block AI crawlers docs, 2026-09-18). - No AI crawler monetization (docs read 2026-11-04). - Netlify does not check that a request using a crawler's user agent comes from that crawler's operator. - Netlify offers no paid access or licensing route for AI crawlers. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Netlify's User Agent Blocker extension blocks requests from a preset list of common AI crawlers and SEO or search crawlers that the site owner picks, using a Netlify Edge Function. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04, dated 2026-09-18. - Primary source Netlify notes the preset list does not include all possible AI crawlers and AI bots. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04, dated 2026-09-18. - Primary source The User Agent Blocker extension can be activated for only one project at a time. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04, dated 2026-09-18. - Primary source Netlify's guide pairs robots.txt with a custom Edge Function that matches user-agent strings for bots that ignore robots rules. [Source](https://developers.netlify.com/guides/blocking-ai-bots-and-controlling-crawlers/), read 2026-11-04. - Primary source Firewall Traffic Rules restrict access by IP address and geography, per site or team, with separate rules for published and unpublished deploys. [Source](https://www.netlify.com/blog/general-availability-web-application-firewall-traffic-rules), read 2026-11-04. - Primary source The general availability post said Firewall Traffic Rules were for Enterprise customers with Netlify High-Performance Edge. [Source](https://www.netlify.com/blog/general-availability-web-application-firewall-traffic-rules), read 2026-11-04. - Primary source The current pricing page lists Firewall Traffic Rules and basic rate limiting from the Free plan; Free is USD 0 with 300 credits, Personal USD 9/month with 1,000 credits, Pro USD 20/month with 3,000 credits. [Source](https://www.netlify.com/pricing/), read 2026-11-04. - Primary source Netlify evaluates Firewall Traffic Rules first, then WAF rules, then rate limiting rules. [Source](https://docs.netlify.com/manage/security/secure-access-to-sites/overview/), read 2026-11-04. ## Compared with - [Vercel vs Netlify](https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify/): Vercel verifies bots by IP, reverse DNS and Web Bot Auth; Netlify's AI bot control is a User Agent Blocker extension that matches strings only and can be active on one project at a time. - [Cloudflare vs Netlify](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify/): Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. ## What Netlify does for this layer ### From the vendor's own pages Netlify rate limiting rules can match by path, client IP ranges, geolocation, cookie or HTTP header, and count requests per visitor over a 60-second window by default. ([docs.netlify.com](https://docs.netlify.com/manage/security/secure-access-to-sites/rate-limiting/)) Netlify rate limiting either blocks with HTTP 429 or rewrites the request to another path such as /rate\_limit\_page.html. ([docs.netlify.com](https://docs.netlify.com/manage/security/secure-access-to-sites/rate-limiting/)) Netlify's basic rate limiting is on all plans, while full functionality needs an Enterprise plan with High-Performance Edge. ([docs.netlify.com](https://docs.netlify.com/manage/security/secure-access-to-sites/rate-limiting/)) Netlify's rate limiting docs list preventing excessive scraping and bot traffic as a use case, next to mitigating DDoS and shielding backend services such as AI inference calls. ([docs.netlify.com](https://docs.netlify.com/manage/security/secure-access-to-sites/rate-limiting/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | User Agent Blocker extension | Edge Function that blocks a preset list of AI, SEO and search crawler user agents. | No price recorded | | Firewall Traffic Rules and rate limiting | IP, geography and header rules plus per-visitor or per-domain rate limits. | plan-based Free USD 0, Personal USD 9, Pro USD 20 a month (pricing page read 2026-11-04). | USD 0 | 2026-11-04 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | No signature verification in the Netlify docs read 2026-11-05. | | Published IP list | No statement | Rules match client IP ranges the customer enters; no operator IP lists are consumed. | | Reverse DNS check | No statement | No reverse DNS verification in the Netlify docs read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Netlify (User Agent Blocker, Firewall Traffic Rules, rate limiting)](https://agentreadycrawlers.com/vendors/netlify/alternatives/) graded on the rubric ## Sources - [netlify.com: netlify.com](https://www.netlify.com) read 2026-11-04 - [netlify.com: pricing](https://www.netlify.com/pricing/) read 2026-11-04 - [docs.netlify.com: block ai crawlers](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers) read 2026-11-04 - [developers.netlify.com: blocking ai bots and controlling crawlers](https://developers.netlify.com/guides/blocking-ai-bots-and-controlling-crawlers/) read 2026-11-04 - [netlify.com: general availability web application firewall traffic rules](https://www.netlify.com/blog/general-availability-web-application-firewall-traffic-rules) read 2026-11-04 - [docs.netlify.com: overview](https://docs.netlify.com/manage/security/secure-access-to-sites/overview/) read 2026-11-04 - [Block AI crawlers and bots | Netlify Docs](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers/) read 2026-11-05 - [Rate limiting | Netlify Docs](https://docs.netlify.com/manage/security/secure-access-to-sites/rate-limiting/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Netlify | Agent Ready Crawlers" description: "Alternatives to Netlify: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/netlify/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Netlify | Agent Ready Crawlers > Alternatives to Netlify: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/netlify/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Netlify In short Netlify ranks last of 15 with 1 of 10 points, and Stytch IsAgent is the nearest on points. Head-to-head pages set it against Vercel and Cloudflare, the two other platforms that run a site's edge. Platform plans start at USD 0, but no price is stated for the extension. ## Head-to-head, with the sourced difference - [**Netlify vs Vercel**](https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify/). Vercel verifies bots by IP, reverse DNS and Web Bot Auth; Netlify's AI bot control is a User Agent Blocker extension that matches strings only and can be active on one project at a time. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04. - [**Netlify vs Cloudflare**](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify/). Cloudflare classifies crawlers by behaviour and verifies them by signature or IP; Netlify blocks a preset list of user-agent strings that it says does not cover all AI crawlers. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Netlify's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Netlify on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, charging, commerce agents, reporting | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy, charging, reporting | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, charging, commerce agents, reporting | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, charging, commerce agents, reporting | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy, charging, reporting | nothing | No public price; sales-led | | Skyfire | 6 | verifies agents, charging, commerce agents | per-purpose policy | Not published (checked 2026-11-04) | | TollBit | 6 | verifies agents, charging, reporting | nothing | Free for publishers to start | | Vercel | 6 | verifies agents, per-purpose policy, reporting | nothing | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents, reporting | nothing | No public price; sales-led | | Fastly | 4 | verifies agents, charging, commerce agents | nothing | No public price; sales-led | | Arcjet | 3 | verifies agents, reporting | nothing | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | commerce agents, reporting | nothing | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | verifies agents, reporting | per-purpose policy | No public price; sales-led | | Stytch IsAgent | 2 | verifies agents, reporting | per-purpose policy | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | ## Where Netlify is not the fit - No verified-bot list, reverse DNS or Web Bot Auth checks; user agents can be spoofed (block AI crawlers docs). - Preset list is incomplete by Netlify's own statement (block AI crawlers docs, 2026-09-18). - No AI crawler monetization (docs read 2026-11-04). ## Sources - [netlify.com: pricing](https://www.netlify.com/pricing/) read 2026-11-04 - [docs.netlify.com: block ai crawlers](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Notte: crawler operator profile | Agent Ready Crawlers" description: "Notte: crawler operator profile covering Notte browser sessions and agents. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/notte site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Notte: crawler operator profile | Agent Ready Crawlers > Notte: crawler operator profile covering Notte browser sessions and agents. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/notte Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Notte In short Notte offers browser sessions, web agents and serverless browser functions, with automatic CAPTCHA solving on by default through its API and a per-session user agent override, and prices sessions from USD 0.01 an hour. Its CLI leaves Web Bot Auth switched off unless a customer sets it, and the docs never name an address for the signatures. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [notte.cc](https://www.notte.cc) In one line Browser platform for AI agents (sessions, agents and serverless browser functions) that says it is registered as a verified bot with Cloudflare and signs session requests with Web Bot Auth when a customer turns it on. Last read 2026-11-05 ## What Notte does for this layer ### From the vendor's own pages Notte's docs describe the flow as signing with HTTP Message Signatures (RFC 9421), after which Cloudflare checks the signature against its verified bots list. ([docs.notte.cc](https://docs.notte.cc/features/sessions/cloudflare-web-bot-auth)) Notte's docs suggest pointing a signed session at webbotauth.io/test to confirm that requests carry valid signatures. ([docs.notte.cc](https://docs.notte.cc/features/sessions/cloudflare-web-bot-auth)) Notte's CLI reference lists automatic CAPTCHA solving with an API default of true, a per-session user agent override, and an advanced stealth mode available to approved workspaces. ([docs.notte.cc](https://docs.notte.cc/llms-full.txt)) Notte prices browser sessions at USD 0.01 an hour, browser functions at USD 0.05 an hour and residential proxies at USD 10 per GB, with bring-your-own proxy at USD 0.20 per GB on the Startup plan. ([notte.cc](https://www.notte.cc/pricing)) Notte's plans are Free with USD 10 of one-time credits, paid plans at USD 20 and USD 100 a month with matching monthly credits, and custom terms. ([notte.cc](https://www.notte.cc/pricing)) ### Claims the vendor makes about itself Notte says it is "registered as a verified bot" in Cloudflare's registry and that, when web\_bot\_auth is set, it signs every outgoing HTTP request in the session with its registered identity. ([docs.notte.cc](https://docs.notte.cc/features/sessions/cloudflare-web-bot-auth)) Notte describes itself as a full-stack browser platform for AI agents, backed by Y Combinator (W25) and SOC 2 Type II certified. ([notte.cc](https://www.notte.cc/)) ### What our crawl saw GETs for the Web Bot Auth key directory at notte.cc, www.notte.cc and api.notte.cc returned HTTP 404 on 2026-11-05, and Notte's docs name no Signature-Agent URL. ([notte.cc](https://www.notte.cc/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Notte browser sessions and agents | Cloud browser sessions, AI web agents and browser functions with proxies, CAPTCHA solving and optional Web Bot Auth. | subscription plus usage Per browser-session hour; plans USD 20 or USD 100 a month. | USD 0.01 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | No robots.txt policy in the docs read (2026-11-05). | | Web Bot Auth | Supports | web_bot_auth session flag signs every request; default off. (docs.notte.cc) | | Cloudflare verified bots | Supports | Notte says it is registered as a verified bot with Cloudflare. (docs.notte.cc) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404 | GET https://www.notte.cc/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404 | GET https://notte.cc/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Notte does not do - Notte does not publish the Signature-Agent URL or key directory it signs with, so outside Cloudflare a site has nothing to verify against (checked 2026-11-05). - Notte sessions are unsigned unless the customer sets web\_bot\_auth, and the CLI exposes a user agent override. ## Reading notes The [Crawler Index](https://agentreadycrawlers.com/crawlers/) has no row for Notte. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [notte.cc](https://www.notte.cc) read 2026-11-05 - [Cloudflare Web Bot Auth - Notte](https://docs.notte.cc/features/sessions/cloudflare-web-bot-auth) read 2026-11-05 - [Notte full documentation (llms-full.txt)](https://docs.notte.cc/llms-full.txt) read 2026-11-05 - [Notte Pricing | Notte](https://www.notte.cc/pricing) read 2026-11-05 - [Notte (site summary served as text)](https://www.notte.cc/) read 2026-11-05 - [Notte key directory probe](https://www.notte.cc/.well-known/http-message-signatures-directory) read 2026-11-05 - [notte.cc key directory probe](https://notte.cc/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "OpenAI: crawler operator profile | Agent Ready Crawlers" description: "OpenAI: crawler operator profile covering GPTBot, OAI-SearchBot, OAI-AdsBot and 2 more. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/openai site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # OpenAI: crawler operator profile | Agent Ready Crawlers > OpenAI: crawler operator profile covering GPTBot, OAI-SearchBot, OAI-AdsBot and 2 more. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/openai Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # OpenAI In short Each of OpenAI's four robots has a robots.txt setting of its own, and a change reaches ChatGPT search after roughly a day. Only its agent browser is checkable by signature, with a key at chatgpt.com; the other robots rely on IP files, and opting out of OAI-SearchBot keeps a site out of ChatGPT answers. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [openai.com](https://openai.com) In one line OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. Last read 2026-11-05 ## What OpenAI does for this layer ### From the vendor's own pages OpenAI's crawler page lists four user agents, OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, and says each robots.txt setting works independently of the others. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says that when a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both uses to avoid duplicate crawling. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says a change to a site's robots.txt can take about 24 hours to show in its search results. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI states that OAI-AdsBot only visits landing pages submitted as ads on ChatGPT and that the data it collects is not used to train generative AI foundation models. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says GPTBot and OAI-SearchBot may add a robots.txt marker to the user agent when fetching robots.txt files, so logs without paths can still separate those requests. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) ### What our crawl saw OpenAI's chatgpt-user.json listed 235 IPv4 prefixes with a creationTime of 2026-10-27 when fetched on 2026-11-05, the largest of OpenAI's four range files. ([openai.com](https://openai.com/chatgpt-user.json), dated 2026-10-27) A GET to chatgpt.com/.well-known/http-message-signatures-directory on 2026-11-05 returned HTTP 200, content type application/http-message-signatures-directory+json, with one Ed25519 key. ([chatgpt.com](https://chatgpt.com/.well-known/http-message-signatures-directory)) The same well-known key path on openai.com returned HTTP 404 on 2026-11-05, so OpenAI's signing identity lives on chatgpt.com rather than its corporate domain. ([openai.com](https://openai.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | GPTBot | Crawler whose content may be used to train OpenAI's generative foundation models; disallowing it is the training opt-out. | No price recorded | | OAI-SearchBot | Crawler for the index behind ChatGPT search answers. | No price recorded | | OAI-AdsBot | Fetcher that checks landing pages submitted as ChatGPT ads. | No price recorded | | ChatGPT-User | User-triggered fetcher for ChatGPT, Custom GPTs and GPT Actions. | No price recorded | | ChatGPT agent cloud browser | Hosted browser for ChatGPT's agent mode whose requests are signed with keys published at chatgpt.com. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | GPTBot and OAI-SearchBot follow robots.txt tokens; OpenAI says robots.txt rules may not apply to ChatGPT-User. (developers.openai.com) | | Web Bot Auth | Supports | chatgpt.com serves a key directory with the Web Bot Auth media type (observed 2026-11-05). (chatgpt.com) | | Content Signals | No statement | Not mentioned on OpenAI's crawler page (read 2026-11-05). | | Published IP list | Supports | Four JSON range files: gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json. (developers.openai.com) | | Reverse DNS check | No statement | OpenAI's crawler page gives no reverse DNS pattern (read 2026-11-05). | | Cloudflare verified bots | No statement | OpenAI's crawler page does not mention Cloudflare's verified bot list (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | chatgpt.com: 200, application/http-message-signatures-directory+json, 1 key; openai.com: 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | gptbot.json | 18 IPv4 prefixes, creationTime 2026-09-22 | GET and count prefixes | 2026-11-05 | | searchbot.json | 39 IPv4 prefixes, creationTime 2026-01-02 | GET and count prefixes | 2026-11-05 | | chatgpt-user.json | 235 IPv4 prefixes, creationTime 2026-10-27 | GET and count prefixes | 2026-11-05 | | adsbot.json | 2 IPv4 prefixes, creationTime 2026-05-12 | GET and count prefixes | 2026-11-05 | ## What OpenAI does not do - OpenAI publishes no reverse DNS pattern for its robots, so verification rests on the IP files or, for the agent browser, on signatures. - Blocking ChatGPT-User in robots.txt is not a guaranteed stop, because OpenAI says robots.txt rules may not apply to user-started fetches. - OpenAI's crawler page names no token for ChatGPT Atlas browsing (read 2026-11-05). ## Reading notes [OpenAI's agent allowlisting help article](https://help.openai.com/en/articles/11845367) returned HTTP 403 to our crawl user agent on 5 November 2026, so what this profile says about agent signing rests on the key directory we observed at chatgpt.com. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [OpenAI in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/openai/) 6 rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) token GPTBot - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) no token ## Sources - [openai.com](https://openai.com) read 2026-11-05 - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) read 2026-11-05 - [OpenAI chatgpt-user.json IP ranges](https://openai.com/chatgpt-user.json) read 2026-11-05 - [OpenAI gptbot.json IP ranges](https://openai.com/gptbot.json) read 2026-11-05 - [chatgpt.com Web Bot Auth key directory path (GET probe)](https://chatgpt.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [openai.com Web Bot Auth key directory path (GET probe)](https://openai.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [OpenAI searchbot.json IP ranges](https://openai.com/searchbot.json) read 2026-11-05 - [OpenAI adsbot.json IP ranges](https://openai.com/adsbot.json) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Parallel Web Systems: crawler operator profile | Agent Ready Crawlers" description: "Parallel Web Systems: crawler operator profile covering Parallel Search, Extract and Task APIs. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/parallel site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Parallel Web Systems: crawler operator profile | Agent Ready Crawlers > Parallel Web Systems: crawler operator profile covering Parallel Search, Extract and Task APIs. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/parallel Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Parallel Web Systems In short Parallel Web Systems feeds its search and extraction APIs from two signed identities, documented from 24 October 2026. ShapBot builds the index and has a robots.txt token and an IP list; Shap-User fetches pages at a user's direction and offers no robots.txt control. Search requests are priced from USD 0.001. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [parallel.ai](https://parallel.ai) In one line Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only. Last read 2026-11-05 ## What Parallel Web Systems does for this layer ### From the vendor's own pages Parallel Web Systems documents two identities on a bots page effective 2026-10-24: ShapBot, which builds Parallel's search index, and Shap-User, which fetches pages at users' direction. ([parallel.ai](https://parallel.ai/parallel-web-systems-bots), dated 2026-10-24) ShapBot's full user agent is "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0", and Parallel tells site owners to manage it with a User-agent: ShapBot group in robots.txt. ([parallel.ai](https://parallel.ai/parallel-web-systems-bots)) Shap-User sends a user agent ending "compatible; Shap-User/0.1.0", and Parallel says Shap-User "is intended to provide visibility rather than to be managed through automated controls". ([parallel.ai](https://parallel.ai/parallel-web-systems-bots)) Parallel signs both ShapBot and Shap-User requests with Web Bot Auth and publishes keys at shapbot.parallel.ai and shap-user.parallel.ai under /.well-known/http-message-signatures-directory. ([parallel.ai](https://parallel.ai/parallel-web-systems-bots)) Parallel warns that restricting ShapBot "prevents our system from indexing your content", which it says may reduce a site's discoverability through its web APIs and search features. ([parallel.ai](https://parallel.ai/parallel-web-systems-bots)) Parallel invites content owners with questions about its index to explore joining "Index by Parallel" through a dedicated contact address. ([parallel.ai](https://parallel.ai/parallel-web-systems-bots)) Parallel's pricing page allows up to 5,000 requests a month free and prices the Search API at USD 0.001 to USD 0.005 for 10 results and the Extract API at USD 0.001 per request. ([parallel.ai](https://parallel.ai/pricing)) ### What our crawl saw Parallel's ShapBot IP list at docs.parallel.ai/resources/shapbot.json had a creationTime of 2026-07-14 and 10 single-address IPv4 prefixes when read on 2026-11-05. ([docs.parallel.ai](https://docs.parallel.ai/resources/shapbot.json)) The ShapBot and Shap-User key directories both returned HTTP 200 with content type application/http-message-signatures-directory+json and two Ed25519 keys each on 2026-11-05. ([shapbot.parallel.ai](https://shapbot.parallel.ai/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Parallel Search, Extract and Task APIs | Web search, page extraction and research tasks for agents, built on the ShapBot index. | pay per request Search API low end per request for 10 results; 5,000 free requests a month. | USD 0.001 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | ShapBot is managed with a User-agent: ShapBot robots.txt group. (parallel.ai) | | Web Bot Auth | Supports | ShapBot and Shap-User sign requests; key directories on two hosts. (parallel.ai) | | Published IP list | Partial | IP list published for ShapBot only, not for Shap-User. (docs.parallel.ai) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 2 Ed25519 keys | GET https://shapbot.parallel.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 2 Ed25519 keys | GET https://shap-user.parallel.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | ShapBot IP list | creationTime 2026-07-14, 10 IPv4 /32 prefixes | GET https://docs.parallel.ai/resources/shapbot.json with the crawl user agent | 2026-11-05 | ## What Parallel Web Systems does not do - Parallel gives Shap-User no robots.txt control; its bots page says user-initiated fetches are for visibility, not automated management. ## Reading notes ShapBot and Shap-User from Parallel are not in the [Crawler Index](https://agentreadycrawlers.com/crawlers/) yet. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [parallel.ai](https://parallel.ai) read 2026-11-05 - [Overview of Parallel Web Systems' Bots | Parallel](https://parallel.ai/parallel-web-systems-bots) read 2026-11-05 - [ShapBot IP list (shapbot.json)](https://docs.parallel.ai/resources/shapbot.json) read 2026-11-05 - [ShapBot key directory](https://shapbot.parallel.ai/.well-known/http-message-signatures-directory) read 2026-11-05 - [Parallel Pricing | Parallel](https://parallel.ai/pricing) read 2026-11-05 - [Shap-User key directory](https://shap-user.parallel.ai/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Perplexity: crawler operator profile | Agent Ready Crawlers" description: "Perplexity: crawler operator profile covering PerplexityBot and Perplexity-User. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/perplexity site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Perplexity: crawler operator profile | Agent Ready Crawlers > Perplexity: crawler operator profile covering PerplexityBot and Perplexity-User. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/perplexity Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Perplexity In short Perplexity splits its traffic between an index crawler and a user-triggered fetcher, ships allow-rule recipes for two WAF products, and publishes JSON range files that sites are told to treat as the source of truth. PerplexityBot is said not to feed foundation model training, while the fetcher generally disregards robots.txt. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [perplexity.ai](https://www.perplexity.ai) In one line Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF. Last read 2026-11-05 ## What Perplexity does for this layer ### From the vendor's own pages Perplexity says PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for AI foundation models. ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) Perplexity says Perplexity-User visits pages when users ask Perplexity a question and that, since a user requested the fetch, it "generally ignores robots.txt rules". ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) Perplexity's Cloudflare recipe is a custom rule that matches each user agent against its own IP ranges and uses the Skip action, placed above any rule that blocks or challenges. ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) Perplexity notes that Cloudflare's Bot Fight Mode cannot be skipped by WAF custom rules and may still challenge its bots. ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) Perplexity points out that Cloudflare's Free plan allows only one custom IP list, which is not enough to hold the ranges of both bots. ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) Perplexity's AWS WAF recipe pairs an IP set match with a User-Agent string match in one AND statement whose action is Allow. ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) Perplexity tells site owners to refresh their WAF rules from its two JSON endpoints automatically and to treat those files as the source of truth. ([docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) ### What our crawl saw Perplexity's perplexitybot.json listed 13 IPv4 prefixes with a creationTime of 2026-10-28 when fetched on 2026-11-05. ([perplexity.ai](https://www.perplexity.ai/perplexitybot.json), dated 2026-10-28) No Web Bot Auth key directory was found at www.perplexity.ai on 2026-11-05; the well-known path returned HTTP 404. ([perplexity.ai](https://www.perplexity.ai/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | PerplexityBot | Crawler for the Perplexity search index; Perplexity says it is not used for foundation model training. | No price recorded | | Perplexity-User | Fetcher for pages a user's question needs; Perplexity says it generally ignores robots.txt. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | PerplexityBot is managed through robots.txt; Perplexity-User generally ignores robots.txt rules. (docs.perplexity.ai) | | Web Bot Auth | No statement | Not on the crawler page; key directory path 404 on www.perplexity.ai (observed 2026-11-05). | | Published IP list | Supports | perplexitybot.json and perplexity-user.json, one file per bot. (docs.perplexity.ai) | | Reverse DNS check | No statement | Not on the crawler page (read 2026-11-05). | | Cloudflare verified bots | No statement | The page gives a Cloudflare WAF Skip rule recipe but says nothing about verified bot status (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | perplexitybot.json | 13 IPv4 prefixes, creationTime 2026-10-28 | GET and count prefixes | 2026-11-05 | | perplexity-user.json | 10 IPv4 prefixes, creationTime 2026-10-28 | GET and count prefixes | 2026-11-05 | ## What Perplexity does not do - Perplexity's crawler page documents no identifier for the Comet browser or for Perplexity Computer (read 2026-11-05). - A robots.txt rule does not reliably stop Perplexity-User, which Perplexity says generally ignores robots.txt. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Perplexity in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/perplexity/) 4 rows - [PerplexityBot](https://agentreadycrawlers.com/crawlers/perplexitybot/) token PerplexityBot - [Perplexity-User](https://agentreadycrawlers.com/crawlers/perplexity-user/) token Perplexity-User - [Comet (browser assistant)](https://agentreadycrawlers.com/crawlers/perplexity-comet/) no token - [Perplexity Computer](https://agentreadycrawlers.com/crawlers/perplexity-computer/) no token ## Sources - [perplexity.ai](https://www.perplexity.ai) read 2026-11-05 - [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) read 2026-11-05 - [Perplexity perplexitybot.json IP ranges](https://www.perplexity.ai/perplexitybot.json) read 2026-11-05 - [Perplexity perplexity-user.json IP ranges](https://www.perplexity.ai/perplexity-user.json) read 2026-11-05 - [www.perplexity.ai Web Bot Auth key directory path (GET probe)](https://www.perplexity.ai/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "ProRata: licensing intermediary profile | Agent Ready Crawlers" description: "ProRata: licensing intermediary profile covering ProRata Attribution, Gist Answers and Gist GEO. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/prorata site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # ProRata: licensing intermediary profile | Agent Ready Crawlers > ProRata: licensing intermediary profile covering ProRata Attribution, Gist Answers and Gist GEO. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/prorata Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Licensing intermediary # ProRata In short ProRata sells attribution: it measures what share of an AI answer each source supplies, a basis for licensing, while saying that payment itself depends on the rights and agreements in place. Its Gist Answers engine falls back on a network of licensed titles that Gist puts above 1,600, and Gist GEO measures brand presence in AI search. Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [prorata.ai](https://prorata.ai) In one line AI attribution company that splits an AI answer into claims and measures each source's share as a basis for licensing and payment, and runs Gist Answers on a network of licensed publisher titles. Last read 2026-11-05 ## What ProRata does for this layer ### From the vendor's own pages ProRata Attribution breaks an AI answer into claims, matches each claim to supporting sources, calculates each source's proportion of the answer and shows it in an attribution bar. ([prorata.ai](https://prorata.ai/)) ProRata states that attribution does not pay rights holders by itself: "Payment depends on the relevant rights, licenses, and commercial agreements." ([prorata.ai](https://prorata.ai/solutions/publishers-and-creators)) ProRata was founded by Bill Gross, who is its CEO, and names its mission as protecting creativity in the era of generative AI through attribution infrastructure. ([prorata.ai](https://prorata.ai/company)) ProRata lists Touring Capital, Mayfield, Prime Movers Lab, Idealab Studio, BOLD Capital Partners, DMG Media, Revolution and SBI Investment among its investors. ([prorata.ai](https://prorata.ai/company)) ProRata's second Gist product, Gist GEO, tracks and measures how brands appear in generative AI search. ([prorata.ai](https://prorata.ai/)) ### Claims the vendor makes about itself Gist Answers is an answer engine a publisher embeds on its own pages, drawing first on that publisher's content and then on a network Gist says holds more than 1,600 licensed titles from more than 150 publishing partners. ([gist.ai](https://gist.ai/products/gist-answers)) Gist states that all content in its network is licensed, that nothing is scraped, and that publisher content is never used to train AI models and can be removed at any time. ([gist.ai](https://gist.ai/products/gist-answers)) Gist names Woman's World, a McClatchy publication, as a publisher running Gist Answers. ([gist.ai](https://gist.ai/products/gist-answers)) ### Reported by a third party IAB Tech Lab reported that ProRata presented AI bot traffic data at its August 2025 publisher workshop, which led to the CoMP working group. ([iabtechlab.com](https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/), dated 2025-08-19) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | ProRata Attribution | Attribution as a Service: claim-level measurement of each source's contribution to an AI answer. | No price recorded | | Gist Answers | Embeddable answer engine for publishers built on a licensed content network. | No price recorded | | Gist GEO | Measurement of brand presence in generative AI search. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | ProRata's homepage, publishers page and company page do not address robots.txt or crawler access (checked 2026-11-05). | | Web Bot Auth | No statement | No Web Bot Auth mention on the pages read (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What ProRata does not do - ProRata does not block, verify or charge crawlers at a publisher's site; it measures contribution inside answers and licenses content into its own Gist network. - ProRata publishes no price or revenue-share figure on the pages read on 2026-11-05. ## Reading notes prorata.ai/publishers returns 404; the publishers page is at /solutions/publishers-and-creators on the same site. ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [prorata.ai](https://prorata.ai) read 2026-11-05 - [ProRata: The trust layer for generative AI](https://prorata.ai/) read 2026-11-05 - [Publishers and Creators: AI attribution (ProRata)](https://prorata.ai/solutions/publishers-and-creators) read 2026-11-05 - [About ProRata](https://prorata.ai/company) read 2026-11-05 - [Gist Answers](https://gist.ai/products/gist-answers) read 2026-11-05 - [Announcing the Content Monetization Protocols (CoMP) for AI Working Group](https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Prove: agent verification provider profile | Agent Ready Crawlers" description: "Prove: agent verification provider profile covering Prove Verified Agent, Prove Agent Pay and Prove Agent Protect. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/prove site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Prove: agent verification provider profile | Agent Ready Crawlers > Prove: agent verification provider profile covering Prove Verified Agent, Prove Agent Pay and Prove Agent Protect. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/prove Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Prove In short Prove builds Verified Agent as a layer between an agent and the businesses it deals with: tokens bind a person, an agent and a payment credential, each action stays within session limits, and a registry of publishers, merchants, networks and CDNs is checked first. Prove says most large US banks trust it, and passive multi-factor checks replace SMS codes. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [prove.com](https://www.prove.com/solutions/agentic-suite) In one line Identity verification company whose Verified Agent product binds an AI agent to a verified person or business with identity-bound tokens, checks participants against a shared trust registry and logs co-signed consent for agent purchases. Last read 2026-11-05 ## What Prove does for this layer ### From the vendor's own pages Prove announced Prove Verified Agent on 23 Oct 2025 from New York as a trust and verification layer for agents acting for consumers and businesses. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce), dated 2025-10-23) Prove Verified Agent launched with initial support for Google's AP2 protocol, and Prove says it will become protocol-agnostic. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce), dated 2025-10-23) Prove Verified Agent provisions identity-bound tokens that link the verified person, the acting agent and the payment credential, and enforces session-level authorization limits on each agent action. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce), dated 2025-10-23) Prove describes a live shared trust registry of agent publishers, relying parties, merchants, payment networks and CDNs, checked for good standing before any delegated action. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce), dated 2025-10-23) Prove says every agent interaction is co-signed by user and merchant keys, producing a consent and outcome log for disputes, chargebacks and regulatory reporting. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce), dated 2025-10-23) Prove payments under Verified Agent use passive multi-factor authentication to avoid SMS one-time passwords at checkout. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce), dated 2025-10-23) The Prove Agentic Suite has three parts: Prove Verified Agent for agent identity and permissions, Prove Agent Pay for credential use within user rules, and Prove Agent Protect for dispute evidence. ([prove.com](https://www.prove.com/solutions/agentic-suite)) Prove's suite page frames the gap as four questions infrastructure cannot answer today: who the agent is, who the person behind it is, what it may do and what happened. ([prove.com](https://www.prove.com/solutions/agentic-suite)) ### Claims the vendor makes about itself Prove says it is trusted by 19 of the top 20 US banks and more than 1,500 brands, while its suite page says 2,000+ companies. ([prove.com](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Prove Verified Agent | Agent identity verification, user linkage and delegated permission enforcement, with a shared trust registry. | sales-led suite page offers "Let's Talk" only | Not published | 2026-11-05 | | Prove Agent Pay | Agent use of a verified user's payment credentials within user-set rules, producing a signed transaction artifact. | No price recorded | | Prove Agent Protect | Dispute evidence and post-transaction validation from Agent Pay records. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The launch post and suite page describe tokens and a trust registry without naming Web Bot Auth (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol mention on the pages read (checked 2026-11-05). | | Mastercard Agent Pay | No statement | The launch post quotes a Mastercard executive but states no Agent Pay integration (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Prove does not do - Prove Verified Agent does not classify crawlers or anonymous bots arriving at a site; it verifies agents that carry Prove-issued credentials. - Prove publishes no public list of the agents or participants in its trust registry (pages read 2026-11-05). ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [Prove Agentic Suite](https://www.prove.com/solutions/agentic-suite) read 2026-11-05 - [Prove Launches Verified Agent Solution to Secure the $1.7 Trillion Agentic Commerce Revolution](https://www.prove.com/blog/prove-verified-agent-secure-agentic-commerce) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Radware: bot management vendor profile | Agent Ready Crawlers" description: "Radware: bot management vendor profile covering Radware Bot Manager and Radware Agent Trust Management. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/radware site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Radware: bot management vendor profile | Agent Ready Crawlers > Radware: bot management vendor profile covering Radware Bot Manager and Radware Agent Trust Management. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/radware Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Radware In short Radware's Agent Trust Management, introduced on 29 September 2026, compares the intent an agent declares in conversation with what it then does, and its sample policy lets an agent browse but not log in or check out. Only Web Bot Auth agents get cryptographic verification. Radware's own site served a browser check, so this profile leans on a wire reprint. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [radware.com](https://www.radware.com) In one line Application security vendor whose Bot Manager and Agent Trust Management identify Web Bot Auth-compliant and non-compliant agents, capture an agent's declared intent in conversation, and restrict actions such as login or checkout. Last read 2026-11-05 ## What Radware does for this layer ### Claims the vendor makes about itself Radware cites its H1 Global Threat Analysis Report: 77% of organizations are deploying or implementing AI agents, and 17.2% report full visibility into them. ([stocktitan.net](https://www.stocktitan.net/news/RDWR/radware-introduces-agent-trust-management-to-help-businesses-mdwk343puaik.html), dated 2026-09-29) ### Reported by a third party Radware introduced Agent Trust Management on 29 Sep 2026 as part of its Cloud Application Protection services, available from that day. ([stocktitan.net](https://www.stocktitan.net/news/RDWR/radware-introduces-agent-trust-management-to-help-businesses-mdwk343puaik.html), dated 2026-09-29) Radware says Agent Trust Management identifies agents from request headers, client-side behavioral signals and direct agent responses, and cryptographically verifies agents that comply with Web Bot Auth. ([stocktitan.net](https://www.stocktitan.net/news/RDWR/radware-introduces-agent-trust-management-to-help-businesses-mdwk343puaik.html), dated 2026-09-29) Radware says the product captures an agent's declared intent through active conversation and keeps checking that the agent's actions stay consistent with it. ([stocktitan.net](https://www.stocktitan.net/news/RDWR/radware-introduces-agent-trust-management-to-help-businesses-mdwk343puaik.html), dated 2026-09-29) Radware's example policy lets an agent browse a site while restricting sensitive actions such as login or checkout. ([stocktitan.net](https://www.stocktitan.net/news/RDWR/radware-introduces-agent-trust-management-to-help-businesses-mdwk343puaik.html), dated 2026-09-29) ### What our crawl saw radware.com answered the crawl user agent with a browser verification page on 2026-11-05, so Radware's own product pages could not be read for this profile. ([radware.com](https://www.radware.com/products/bot-manager/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Radware Bot Manager | Behavior-based bot detection with AI crawler management, delivered in Cloud Application Protection services. | No price recorded | | Radware Agent Trust Management | Agent identification, declared-intent capture, trust assessment and permission-based governance of agent actions. | sales-led No price in the September 2026 release. | Not published | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Radware says Web Bot Auth-compliant agents are cryptographically verified (release of 2026-09-29). (stocktitan.net) | | Visa Trusted Agent Protocol | No statement | The September 2026 release names no Trusted Agent Protocol support. | | Mastercard Agent Pay | No statement | The September 2026 release names no Agent Pay support. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Radware does not do - Radware publishes no price for Bot Manager or Agent Trust Management in the release read on 2026-11-05. - Radware's release offers no paid access or licensing route for AI crawlers. ## Reading notes radware.com served a browser check to our crawl user agent on 5 November 2026, so the facts come from StockTitan's full reprint of the GlobeNewswire release. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [radware.com](https://www.radware.com) read 2026-11-05 - [Radware launches AI agent trust management solution | RDWR Stock News](https://www.stocktitan.net/news/RDWR/radware-introduces-agent-trust-management-to-help-businesses-mdwk343puaik.html) read 2026-11-05 - [Radware Bot Manager (browser verification page served to the crawl user agent)](https://www.radware.com/products/bot-manager/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Vendor ranking and rubric | Agent Ready Crawlers" description: "Ranking of 15 bot management and agent identity vendors on a published rubric: verification, per-purpose policy, charging, commerce agents and reporting." canonical: https://agentreadycrawlers.com/vendors/ranking site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Vendor ranking and rubric | Agent Ready Crawlers > Ranking of 15 bot management and agent identity vendors on a published rubric: verification, per-purpose policy, charging, commerce agents and reporting. Source: https://agentreadycrawlers.com/vendors/ranking Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Vendors at the door In short 15 bot management, agent identity and monetisation vendors, graded on five documented questions and ranked by points out of ten. Akamai, AWS WAF Bot Control, Cloudflare and DataDome tie at 8. Cost and where the product runs are described, not graded. Ora is not ranked because it does not manage bots. These 15 are the vendors graded on the rubric. [The full vendor list](https://agentreadycrawlers.com/vendors/) profiles 73 companies in four types, with their standards, prices and sources. | Rank | Vendor | Points | Verifies agents | Per-purpose policy | Charging | Commerce agents | Reporting | Cost, read 2026-11-04 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1= | Akamai | 8 | Yes | Partial | Partial | Yes | Yes | No public price; sales-led | | 1= | AWS WAF Bot Control | 8 | Yes | Yes | Yes | Not published | Yes | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | 1= | Cloudflare | 8 | Yes | Yes | Partial | Partial | Yes | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | 1= | DataDome | 8 | Yes | Partial | Yes | Partial | Yes | From USD 3,830/month | | 5 | HUMAN Security | 7 | Yes | Yes | Partial | Not published | Yes | No public price; sales-led | | 6= | Skyfire | 6 | Yes | Not its job | Yes | Yes | Not published | Not published (checked 2026-11-04) | | 6= | TollBit | 6 | Partial | Partial | Yes | Not published | Yes | Free for publishers to start | | 6= | Vercel | 6 | Yes | Yes | Not published | Not published | Yes | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | 9 | Kasada | 5 | Yes | Partial | Not published | Not published | Yes | No public price; sales-led | | 10 | Fastly | 4 | Partial | Partial | Partial | Partial | Not published | No public price; sales-led | | 11= | Arcjet | 3 | Partial | Partial | Not published | Not published | Partial | From USD 25/month plus USD 5 per 1M web requests | | 11= | Auth0 for AI Agents | 3 | Not its job | Partial | Not published | Partial | Partial | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | 11= | Imperva | 3 | Partial | Not published | Not published | Not published | Yes | No public price; sales-led | | 14 | Stytch IsAgent | 2 | Partial | Not published | Not published | Not published | Partial | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | 15 | Netlify | 1 | Not published | Partial | Not published | Not published | Not published | Platform plans from USD 0; extension price not stated | ## What the ranking does and does not say Akamai, AWS WAF Bot Control, Cloudflare and DataDome tie at the top with 8 of 10, so the ranking cannot separate them. They differ in what they are strong at: Akamai has 3 yes grades, AWS WAF Bot Control has 4 yes grades, Cloudflare has 3 yes grades, DataDome has 3 yes grades. The profiles list what each does not do. A vendor that does one job well can rank low on a broad rubric. Not its job counts as zero, so identity and monetisation products such as Skyfire, TollBit and Stytch IsAgent sit below the bot management suites. Read the row for the job you need, not the rank. Grades come from what each vendor documents on its own site, read 2026-11-04. We have not tested the products. A vendor's description of itself is a vendor claim, and the profiles label it. Ora does not manage bots, so it has no row here; [the method page says where it fits](https://agentreadycrawlers.com/method/#ora-on-this-layer). ## The rubric Can it tell a verified agent from a spoof? Yes where the vendor documents verification by Web Bot Auth signatures, IP validation or reverse DNS. Partial where it depends on partners, lists only or an unconfirmed method. Not published where nothing is documented. Can it express a policy per purpose (train, search, answer, act)? Yes where separate controls exist for crawler purposes or for named bot categories. Partial where policy is per bot or per category without a purpose split. Does it support pay-per-crawl or any charging? Yes where charging is generally available. Partial where it is in beta or runs through a partner. Not published where the vendor documents none. Does it let signed commerce agents (Visa TAP, Mastercard Agent Pay) through without custom work? Yes where support is documented as shipped or announced with named integrations. Partial where support is built on Web Bot Auth but the rule is not confirmed in documents. Not published otherwise. What does it cost? (not graded) Quoted from the vendor with the date read. Not graded: a free tier is not better or worse than a sales-led price for every reader. What does it report? Yes where AI or bot activity dashboards, analytics or logs are documented. Partial where reporting is limited to general bot analytics. Does it work at the edge or in the app? (not graded) Described as the vendor documents it. Not graded. ## Head-to-heads and alternatives 30 head-to-head pages cover the pairs a buyer is most likely to search; every vendor also has an alternatives page. Start from a profile: [Akamai](https://agentreadycrawlers.com/vendors/akamai/), [AWS WAF Bot Control](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/), [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/), [DataDome](https://agentreadycrawlers.com/vendors/datadome/), [HUMAN Security](https://agentreadycrawlers.com/vendors/human-security/), [Skyfire](https://agentreadycrawlers.com/vendors/skyfire/) and the rest. - [Cloudflare vs DataDome](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome/) - [DataDome vs HUMAN Security](https://agentreadycrawlers.com/compare/datadome-vs-human-security/) - [Akamai vs Cloudflare](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare/) - [Arcjet vs Vercel](https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid/) - [Skyfire vs Stytch IsAgent](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent/) - [TollBit vs Cloudflare](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/) - [Cloudflare vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control/) - [Cloudflare vs Vercel](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid/) - [DataDome vs Akamai](https://agentreadycrawlers.com/compare/datadome-vs-akamai/) - [HUMAN Security vs Kasada](https://agentreadycrawlers.com/compare/human-security-vs-kasada/) - [Fastly vs Cloudflare](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare/) - [AWS WAF Bot Control vs Akamai](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai/) - [Vercel vs Netlify](https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify/) - [Cloudflare vs Netlify](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify/) - [Skyfire vs TollBit](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit/) - [Stytch IsAgent vs Auth0 for AI Agents](https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents/) - [Akamai vs HUMAN Security](https://agentreadycrawlers.com/compare/akamai-vs-human-security/) - [Cloudflare vs HUMAN Security](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security/) - [Kasada vs Vercel](https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid/) - [Arcjet vs Cloudflare](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare/) - [DataDome vs Kasada](https://agentreadycrawlers.com/compare/datadome-vs-kasada/) - [Fastly vs Akamai](https://agentreadycrawlers.com/compare/fastly-vs-akamai/) - [AWS WAF Bot Control vs DataDome](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome/) - [Imperva vs Cloudflare](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare/) - [Imperva vs DataDome](https://agentreadycrawlers.com/compare/imperva-vs-datadome/) - [TollBit vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control/) - [Skyfire vs Cloudflare](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare/) - [Auth0 for AI Agents vs Skyfire](https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire/) - [Vercel vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control/) - [Kasada vs Akamai](https://agentreadycrawlers.com/compare/kasada-vs-akamai/) ## Sources - [Vendor rubric (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [akamai.com: akamai.com](https://www.akamai.com) read 2026-11-04 - [arcjet.com: arcjet.com](https://arcjet.com) read 2026-11-04 - [auth0.com: ai](https://auth0.com/ai) read 2026-11-04 - [aws.amazon.com: bot control](https://aws.amazon.com/waf/features/bot-control/) read 2026-11-04 - [cloudflare.com: cloudflare.com](https://www.cloudflare.com) read 2026-11-04 - [datadome.co: datadome.co](https://datadome.co) read 2026-11-04 - [fastly.com: fastly.com](https://www.fastly.com) read 2026-11-04 - [humansecurity.com: humansecurity.com](https://www.humansecurity.com) read 2026-11-04 - [imperva.com: imperva.com](https://www.imperva.com) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io) read 2026-11-04 - [netlify.com: netlify.com](https://www.netlify.com) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz) read 2026-11-04 - [stytch.com: stytch.com](https://stytch.com) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com) read 2026-11-04 - [vercel.com: vercel.com](https://vercel.com) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "RSL Collective: licensing intermediary profile | Agent Ready Crawlers" description: "RSL Collective: licensing intermediary profile covering RSL 1.0 standard and RSL Collective licensing platform. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/rsl-collective site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # RSL Collective: licensing intermediary profile | Agent Ready Crawlers > RSL Collective: licensing intermediary profile covering RSL 1.0 standard and RSL Collective licensing platform. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/rsl-collective Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Licensing intermediary # RSL Collective In short RSL Collective, a nonprofit that publishers join free, co-developed Really Simple Licensing, published as a Recommendation on 10 December 2025. The standard offers ai-all, ai-input and ai-index uses, so a site can accept search indexing yet decline AI answers, and a licence file can price a crawl through x402 over HTTP 402. Enforcement is left to CDNs and servers. Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [rslcollective.org](https://rslcollective.org) In one line Nonprofit collective rights organisation and co-developer of the RSL standard, a machine-readable license file a site links from robots.txt so crawlers can see, buy and prove a license for training, AI input, search or crawling. Last read 2026-11-05 ## What RSL Collective does for this layer ### From the vendor's own pages The RSL Technical Steering Committee published Really Simple Licensing 1.0 as a Recommendation on 2025-12-10, superseding version 0.9. ([rslstandard.org](https://rslstandard.org/rsl), dated 2025-12-10) The RSL Standard and the RSL Collective were announced together on 10 Sep 2025. ([rslstandard.org](https://rslstandard.org/press), dated 2025-09-10) RSL 1.0 builds on RSS and the Robots Exclusion Protocol (RFC 9309): a site adds a line such as "License: https://example.com/license.xml" to robots.txt, or points to the file from an HTTP Link header, an RSS feed or an HTML link element. ([rslstandard.org](https://rslstandard.org/rsl), dated 2025-12-10) RSL 1.0 defines three optional protocols on top of the core file: the Open License Protocol as an OAuth 2.0 extension for license tokens, the Crawler Authorization Protocol as an HTTP authorization scheme for licensed crawlers, and the Encrypted Media Standard. ([rslstandard.org](https://rslstandard.org/rsl), dated 2025-12-10) RSL clients must revalidate a license document according to its max-age and apply a default of 30 days when no max-age is given. ([rslstandard.org](https://rslstandard.org/rsl), dated 2025-12-10) The RSL 1.0 specification shows a pay-per-crawl license whose accepts element names the x402 protocol (media type application/x402+json), with prices settled through an HTTP 402 Payment Required exchange. ([rslstandard.org](https://rslstandard.org/rsl), dated 2025-12-10) RSL 1.0 added the usage categories ai-all, ai-input and ai-index, so a publisher can allow search indexing while refusing use in AI search answers. ([rslstandard.org](https://rslstandard.org/press/rsl-1-specification-2025), dated 2025-12-10) RSL 1.0 added a "contribution" payment option built with Creative Commons as the first application of its CC signals work, letting noncommercial publishers ask AI systems for monetary or in-kind contributions. ([rslstandard.org](https://rslstandard.org/press/rsl-1-specification-2025), dated 2025-12-10) The RSL Collective's sample collective license points crawlers to the license server https://api.rslcollective.org and pays publishers per AI output under the standard license at rslcollective.org/license. ([rslcollective.org](https://rslcollective.org/)) The RSL Collective describes itself as a nonprofit rights organisation that publishers join for free, with the RSL standard as the basis for collective licensing and bargaining with AI companies. ([rslcollective.org](https://rslcollective.org/about)) ### Claims the vendor makes about itself The RSL 1.0 announcement says Cloudflare, Akamai, Creative Commons and IAB Tech Lab support the standard and that more than 1,500 media organisations, brands and technology companies endorse it. ([rslstandard.org](https://rslstandard.org/press/rsl-1-specification-2025), dated 2025-12-10) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | RSL 1.0 standard | XML license vocabulary plus robots.txt, HTTP header, RSS and HTML discovery, with optional OLP, CAP and EMS protocols. | open standard specification free to read and implement | USD 0 | 2026-11-05 | | RSL Collective licensing platform | Collective license and license server that collects pay-per-output royalties from AI companies for member publishers. | free to join for publishers royalty split not published on the homepage | USD 0 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | RSL extends robots.txt with a License: directive and cites RFC 9309 as its base. (rslstandard.org) | | Web Bot Auth | No statement | Crawler identity in RSL runs through OLP license tokens and the Crawler Authorization Protocol; the spec does not cite Web Bot Auth (checked 2026-11-05). | | Content Signals | No statement | The RSL 1.0 specification and announcement name CC signals, not Cloudflare's Content-Signals policy (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What RSL Collective does not do - RSL does not enforce anything by itself; a crawler that ignores robots.txt also ignores its License: line unless a CDN or server blocks it. - The RSL Collective homepage gives no royalty rates or revenue split for publishers (read 2026-11-05). ## Reading notes rslstandard.org hosts the specification and the press page, while rslcollective.org hosts the nonprofit and its licence server. Fastly's support is cited by the RSL press page, which links a Fastly blog post of 10 September 2025. ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management - [Akamai (Bot & Agent Control)](https://agentreadycrawlers.com/vendors/akamai/) bot management - [Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/) bot management ## Sources - [rslcollective.org](https://rslcollective.org) read 2026-11-05 - [Really Simple Licensing (RSL) 1.0 Specification](https://rslstandard.org/rsl) read 2026-11-05 - [RSL press and announcements](https://rslstandard.org/press) read 2026-11-05 - [RSL AI Licensing 1.0 Now an Official Industry Standard](https://rslstandard.org/press/rsl-1-specification-2025) read 2026-11-05 - [RSL Collective homepage](https://rslcollective.org/) read 2026-11-05 - [About the RSL Collective](https://rslcollective.org/about) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Skyfire: bot management profile | Agent Ready Crawlers" description: "Skyfire: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15." canonical: https://agentreadycrawlers.com/vendors/skyfire site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Skyfire: bot management profile | Agent Ready Crawlers > Skyfire: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15. Source: https://agentreadycrawlers.com/vendors/skyfire Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 6 (tied) of 15 # Skyfire In short Skyfire is an identity and payment layer for agents: its KYA tokens grade yes on verification, charging and commerce agents, for 6 of 10 points. Per-purpose policy is not its job, and reporting is not published. Unsigned bots are outside its reach, since Skyfire only mints and settles tokens. Identity and payments layer for AI agents: it issues signed KYA identity tokens and pay tokens that websites and bot managers verify against Skyfire's public keys, with settlement between Skyfire wallets. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Skyfire](https://agentreadycrawlers.com/vendors/skyfire/alternatives/) · [skyfire.xyz](https://skyfire.xyz) Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [skyfire.xyz](https://skyfire.xyz) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | KYA JWTs that sites verify via Skyfire's JWKS; not a bot manager itself | Verify and Extract Data from Tokens docs. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Not its job | sellers decide access from token claims such as agent and principal identity | Token claims docs. Source, read 2026-11-04. | | Charging for access | Yes | pay and kya-pay tokens charged by sellers, settled between Skyfire wallets | Payments and Settlement docs. Source, read 2026-11-04. | | Lets signed commerce agents through | Yes | KYAPay carries identity plus payment intent; card payment flow documented | KYAPay site and Payment Cards guide. Source, read 2026-11-04. | | Cost | Not graded | not published (checked 2026-11-04) | No fees on homepage or docs index. Source, read 2026-11-04. | | Reporting | Not published | on pages read (checked 2026-11-04) | Docs index lists no analytics section. Source, read 2026-11-04. | | Where it runs | Not graded | either: sellers verify tokens in their backend, in a Cloudflare Worker, or through partner bot managers (F5, Akamai, Fastly) | Docs index and partner releases. Source, read 2026-11-04. | ## Cost no public fee schedule found on homepage or docs index (checked 2026-11-04) [Source](https://skyfire.xyz/), read 2026-11-04. ## What Skyfire does not do - Does not detect or block unsigned bots; it only issues and settles tokens that sites or bot managers must check (docs). - KYA tokens alone cannot be charged; only pay and kya-pay tokens move money (introduction docs). - No published fee schedule (homepage and docs read 2026-11-04). - Skyfire does not sit in the request path; tokens travel straight from buyer to seller and the seller or its bot manager does the verification. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Skyfire issues three token types: kya proves identity, pay authorizes a payment to a seller, and kya-pay combines both. [Source](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md), read 2026-11-04, dated 2026-07-27. - Primary source Skyfire tokens are ES256-signed JWTs sent in a kyapay-token header; the older skyfire-pay-id header is deprecated. [Source](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md), read 2026-11-04, dated 2026-11-03. - Primary source Sellers verify tokens against Skyfire's JWKS at app.skyfire.xyz/.well-known/jwks.json, caching keys up to 60 minutes, then check typ, env, iss, iat, exp and the hid, apd and aid claims. [Source](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md), read 2026-11-04, dated 2026-11-03. - Primary source Creating a pay or kya-pay token commits funds from the buyer's wallet, so a seller charging a valid token is guaranteed payment up to the committed amount. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04, dated 2026-07-13. - Primary source Tokens expire between 10 seconds and 24 hours after creation, and a token validated before expiry can be charged for up to 24 hours after it expires. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04, dated 2026-07-13. - Primary source Full settlement can take up to about 51 hours, but charges settle within a 3-hour window once a token's cumulative charges pass USD 1.00. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04, dated 2026-07-13. - Primary source The KYA protocol lets agent developers state who they are, the platform they run on and the users they represent; KYAPay adds user intent and tokenized payment credentials. [Source](https://kyapay.org/), read 2026-11-04. - Primary source Skyfire's docs include a guide for enforcing KYA tokens on a site served through Cloudflare using a Worker and optional WAF rules. [Source](https://docs.skyfire.xyz/llms.txt), read 2026-11-04. - Primary source F5 and Skyfire announced on 18 Mar 2026 an integration of Skyfire's KYA protocol into F5's bot defense, letting agents complete standard checkout with tokenized payment credentials. [Source](https://www.f5.com/company/news/press-releases/f5-skyfire-secure-agentic-commerce), read 2026-11-04, dated 2026-03-18. - Primary source Fastly and Skyfire partnered on 10 Jun 2026 to bring Skyfire identity and payment-backed credentials to Fastly's edge. [Source](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce), read 2026-11-04, dated 2026-06-10. - Primary source Akamai's June 2026 agentic framework names Skyfire and Experian Know Your Agent among its integrations. [Source](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce), read 2026-11-04, dated 2026-06-15. - Vendor claim Skyfire's homepage presents KYA as a portable verified identity token accepted by the majority of the commercial internet and shows Akamai, Fastly, Experian and Visa among partners. [Source](https://skyfire.xyz/), read 2026-11-04. ## Compared with - [Skyfire vs Stytch IsAgent](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent/): Skyfire issues signed KYA and pay tokens that sellers verify against its JWKS; Stytch IsAgent is a front-end flag that Stytch says must not be used as a security control. - [Skyfire vs TollBit](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit/): Skyfire is the payer side: agents carry kya-pay tokens with committed funds; TollBit is the publisher side: a paywall with publisher-set rates that AI companies access with TollBit tokens. - [Skyfire vs Cloudflare](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare/): Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. - [Auth0 for AI Agents vs Skyfire](https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire/): Auth0 issues identities and short-lived tokens to agents a company builds for its own users; Skyfire issues KYA tokens that any third-party site can verify. ## What Skyfire does for this layer ### From the vendor's own pages Skyfire tells sellers to answer 401 Unauthorized when the kyapay-token header is missing, 403 Forbidden when the token is expired or fails verification, and 402 Payment Required when a pay token's balance is too low. ([docs.skyfire.xyz](https://docs.skyfire.xyz/docs/handling-missing-or-invalid-tokens.md), dated 2026-09-24) Skyfire's security brief for sellers and bot managers says Skyfire runs KYB checks on the buy-side agent platforms it onboards and verifies, directly or through those platforms, the people or businesses behind the agents. ([docs.skyfire.xyz](https://docs.skyfire.xyz/docs/security-brief-for-sellers.md), dated 2026-11-03) Skyfire kya tokens are RFC 7515 and RFC 7519 signed JWTs whose aud, ssi, srl and sdm claims tie each token to one seller, so a copied token fails at any other site. ([docs.skyfire.xyz](https://docs.skyfire.xyz/docs/security-brief-for-sellers.md), dated 2026-11-03) Skyfire lets a seller set the maximum expiry of tokens created for it, trading replay risk against friction for the buying agent. ([docs.skyfire.xyz](https://docs.skyfire.xyz/docs/security-brief-for-sellers.md), dated 2026-11-03) Know Your Agent verification at Skyfire needs a paid subscription on an Individual or Organization plan, and the verified identity then passes to every buyer and seller agent on that account. ([docs.skyfire.xyz](https://docs.skyfire.xyz/docs/kya.md), dated 2026-07-14) Skyfire tokens disclose only the identity fields a seller requires, and token creation fails when the buyer is not verified to the level the service demands. ([docs.skyfire.xyz](https://docs.skyfire.xyz/docs/kya.md), dated 2026-07-14) Skyfire's docs show sellers how to declare token requirements in an OpenAPI spec and in MCP server tool descriptions, and list services in a Skyfire Directory with a Verified Service Guarantee. ([docs.skyfire.xyz](https://docs.skyfire.xyz/llms.txt)) ### Claims the vendor makes about itself Skyfire's documentation index describes Skyfire as "the open identity and payments layer for AI agents" that lets developers monetize APIs, tools and content to agents with token-based authentication and stablecoin payments. ([docs.skyfire.xyz](https://docs.skyfire.xyz/llms.txt)) Skyfire's homepage claims KYA coverage of "60%" of the web through infrastructure partners and shows HUMAN Security, Akamai, DataDome, F5, Fastly, Imperva and Cequence among them. ([skyfire.xyz](https://skyfire.xyz/)) Skyfire's homepage lists Visa, Mastercard, Discover and USDC for its Agentic Wallet, which enrols user cards, holds stablecoins and collects user mandates for checkout. ([skyfire.xyz](https://skyfire.xyz/)) ### Reported by a third party Visa named Skyfire on 18 Dec 2025 among the US agent-enabling partners in Visa Intelligent Commerce pilots, enabling Consumer Reports' recommendation agent to demonstrate a purchase of Bose headphones by browser automation. ([usa.visa.com](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21961.html), dated 2025-12-18) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Know Your Agent (KYA) | Signed identity token binding platform, agent and human principal, verified through a paid subscription. | subscription for verification (Individual or Organization) docs name the plans without prices | Not published | 2026-11-05 | | KYAPay tokens (pay, kya-pay) | Payment-backed tokens a seller charges per request, settled between Skyfire wallets. | No price recorded | | Agentic Wallet | Wallet that enrols user cards and stablecoins and collects user mandates for agent checkout. | No price recorded | | Skyfire Directory | Discovery listing of seller services, tools and agents with a Verified Service Guarantee. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | Skyfire identifies agents with its own JWTs in the kyapay-token header; the docs index has no Web Bot Auth page (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | Visa names Skyfire as a Visa Intelligent Commerce pilot partner; Skyfire's docs do not describe TAP signing (checked 2026-11-05). | | Mastercard Agent Pay | No statement | Mastercard appears in the homepage partner strip; no Agent Pay integration is documented (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Reading notes Skyfire's docs disagree with themselves: the security brief updated on 3 November 2026 still gives skyfire-pay-id as the example header, while the token verification page marks that header deprecated in favour of kyapay-token. Both are recorded. ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Akamai (Bot & Agent Control)](https://agentreadycrawlers.com/vendors/akamai/) bot management - [DataDome (Bot Protect, Agentic Trust)](https://agentreadycrawlers.com/vendors/datadome/) bot management - [Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/) bot management - [HUMAN Security (Sightline, AgenticTrust)](https://agentreadycrawlers.com/vendors/human-security/) bot management - [Imperva, a Thales company (Advanced Bot Protection)](https://agentreadycrawlers.com/vendors/imperva/) bot management - [Auth0 for AI Agents (Okta)](https://agentreadycrawlers.com/vendors/auth0-ai-agents/) verification provider - [Apify Website Content Crawler](https://agentreadycrawlers.com/crawlers/apify/) no token - [Alternatives to Skyfire (KYA, KYAPay, Agentic Wallet)](https://agentreadycrawlers.com/vendors/skyfire/alternatives/) graded on the rubric ## Sources - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [kyapay.org: kyapay.org](https://kyapay.org/) read 2026-11-04 - [docs.skyfire.xyz: llms.txt](https://docs.skyfire.xyz/llms.txt) read 2026-11-04 - [docs.skyfire.xyz: introduction to skyfire.md](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md) read 2026-11-04 - [f5.com: f5 skyfire secure agentic commerce](https://www.f5.com/company/news/press-releases/f5-skyfire-secure-agentic-commerce) read 2026-11-04 - [investors.fastly.com: fastly and skyfire partner enable trusted agentic commerce](https://investors.fastly.com/news-releases/news-release-details/fastly-and-skyfire-partner-enable-trusted-agentic-commerce) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [Skyfire docs: Handling Missing or Invalid Tokens](https://docs.skyfire.xyz/docs/handling-missing-or-invalid-tokens.md) read 2026-11-05 - [Skyfire docs: Security Brief for Sellers and Bot Managers](https://docs.skyfire.xyz/docs/security-brief-for-sellers.md) read 2026-11-05 - [Skyfire docs: Know Your Agent (KYA)](https://docs.skyfire.xyz/docs/kya.md) read 2026-11-05 - [Visa and Partners Complete Secure AI Transactions, Setting the Stage for Mainstream Adoption in 2026](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21961.html) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Skyfire | Agent Ready Crawlers" description: "Alternatives to Skyfire: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/skyfire/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Skyfire | Agent Ready Crawlers > Alternatives to Skyfire: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/skyfire/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Skyfire In short Skyfire shares sixth place on 6 of 10 points with TollBit and Vercel, and four head-to-head pages compare it with Stytch IsAgent, TollBit, Cloudflare and Auth0 for AI Agents. It is an identity and payment layer rather than a gate, so the alternatives that block or classify bots do different work. Its price is not published. ## Head-to-head, with the sourced difference - [**Skyfire vs Stytch IsAgent**](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent/). Skyfire issues signed KYA and pay tokens that sellers verify against its JWKS; Stytch IsAgent is a front-end flag that Stytch says must not be used as a security control. [Source](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0), read 2026-11-04. - [**Skyfire vs TollBit**](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit/). Skyfire is the payer side: agents carry kya-pay tokens with committed funds; TollBit is the publisher side: a paywall with publisher-set rates that AI companies access with TollBit tokens. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04. - [**Skyfire vs Cloudflare**](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare/). Skyfire tokens ride in a kyapay-token header and are verified against Skyfire's JWKS; Cloudflare verifies Web Bot Auth signatures from a Signature-Agent key directory. Skyfire documents enforcing its tokens on Cloudflare with a Worker. [Source](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md), read 2026-11-04. - [**Skyfire vs Auth0 for AI Agents**](https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire/). Auth0 issues identities and short-lived tokens to agents a company builds for its own users; Skyfire issues KYA tokens that any third-party site can verify. [Source](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Skyfire's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Skyfire on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | per-purpose policy, reporting | charging | No public price; sales-led | | AWS WAF Bot Control | 8 | per-purpose policy, reporting | commerce agents | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | per-purpose policy, reporting | charging, commerce agents | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | per-purpose policy, reporting | commerce agents | From USD 3,830/month | | HUMAN Security | 7 | per-purpose policy, reporting | charging, commerce agents | No public price; sales-led | | TollBit | 6 | per-purpose policy, reporting | verifies agents, commerce agents | Free for publishers to start | | Vercel | 6 | per-purpose policy, reporting | charging, commerce agents | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | per-purpose policy, reporting | charging, commerce agents | No public price; sales-led | | Fastly | 4 | per-purpose policy | verifies agents, charging, commerce agents | No public price; sales-led | | Arcjet | 3 | per-purpose policy, reporting | verifies agents, charging, commerce agents | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | per-purpose policy, reporting | verifies agents, charging, commerce agents | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | reporting | verifies agents, charging, commerce agents | No public price; sales-led | | Stytch IsAgent | 2 | reporting | verifies agents, charging, commerce agents | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | per-purpose policy | verifies agents, charging, commerce agents | Platform plans from USD 0; extension price not stated | ## Where Skyfire is not the fit - Does not detect or block unsigned bots; it only issues and settles tokens that sites or bot managers must check (docs). - KYA tokens alone cannot be charged; only pay and kya-pay tokens move money (introduction docs). - No published fee schedule (homepage and docs read 2026-11-04). ## Sources - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [changelog.stytch.com: ann\_iPdXgiqlNrBk0](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [docs.skyfire.xyz: verify and extract data from tokens.md](https://docs.skyfire.xyz/docs/verify-and-extract-data-from-tokens.md) read 2026-11-04 - [docs.skyfire.xyz: introduction to skyfire.md](https://docs.skyfire.xyz/docs/introduction-to-skyfire.md) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Stytch IsAgent: bot management profile | Agent Ready Crawlers" description: "Stytch IsAgent: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources." canonical: https://agentreadycrawlers.com/vendors/stytch-isagent site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Stytch IsAgent: bot management profile | Agent Ready Crawlers > Stytch IsAgent: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Source: https://agentreadycrawlers.com/vendors/stytch-isagent Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 14 of 15 # Stytch IsAgent In short Stytch IsAgent flags agent traffic at a site's front end, and earns partial grades for agent verification and reporting only, for 2 of 10 points. IsAgent calls are free up to 10,000 a month, and device fingerprints beyond 10,000 cost half a cent each. Stytch's guidance is that IsAgent informs a decision but is not itself a security control. Identity platform whose free IsAgent front-end component flags likely AI agents for analytics and agent-friendly flows, and whose fingerprinting product checks Web Bot Auth signatures; IsAgent is explicitly not a security control. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Stytch IsAgent](https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives/) · [stytch.com](https://stytch.com) Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Operator Twilio Based in United States Homepage [stytch.com](https://stytch.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Partial | Web Bot Auth signatures plus heuristic detection (user agent, TLS fingerprint, network traits) | Changelog 2025-09-19 and IsAgent FAQ. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Not published | per-purpose policy (checked 2026-11-04) | IsAgent returns a hint, not a policy engine. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | No monetization in IsAgent docs. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned in changelog or FAQ. Source, read 2026-11-04. | | Cost | Not graded | published: IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | FAQ and pricing page. Source, read 2026-11-04. | | Reporting | Partial | client hint with agent flag and identity, meant for analytics | IsAgent FAQ: use for analytics and steering. Source, read 2026-11-04. | | Where it runs | Not graded | app: client-side SDK calling Stytch's API, plus backend Lookup API for fingerprinting | IsAgent FAQ. Source, read 2026-11-04. | ## Cost IsAgent free up to 10,000 calls/month (contact Stytch above that); Device Fingerprinting USD 0.005 per fingerprint beyond 10,000 included; free tier 10,000 MAUs and AI agents. [Source](https://stytch.com/pricing), read 2026-11-04. ## What Stytch IsAgent does not do - IsAgent is not a security control; its client-side result can be tampered with (Stytch changelog, 2025-08-01). - Detection is heuristic and not 100% accurate (IsAgent FAQ). - No crawler monetization or per-purpose crawler policy (pages read 2026-11-04). - Stytch does not run a crawler licensing or pay-per-crawl product; its agent work covers detection, signature checks and OAuth for agents. - Stytch's Device Fingerprinting docs for agents sit behind a login, so the per-agent rule names a customer sees are not public (checked 2026-11-05). ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source Stytch launched isAgent on 1 Aug 2025 as a lightweight front-end component to detect agentic and programmatic traffic and steer it to agent-friendly experiences. [Source](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0), read 2026-11-04, dated 2025-08-01. - Primary source Stytch says isAgent should not be used as a security measure because attackers can modify the response in their browsers. [Source](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0), read 2026-11-04, dated 2025-08-01. - Primary source IsAgent is free up to 10,000 calls a month; higher volumes go through Stytch directly. [Source](https://isagent-web.vercel.app/faq), read 2026-11-04. - Primary source IsAgent weighs the user agent string, TLS fingerprint and network traits, and Stytch states detection is not 100% accurate. [Source](https://isagent-web.vercel.app/faq), read 2026-11-04. - Primary source Stytch added Web Bot Auth support on 19 Sep 2025, and agents sending Web Bot Auth headers need no registration with Stytch. [Source](https://changelog.stytch.com/announcements/ann_uSpDBNlbdXL8X), read 2026-11-04, dated 2025-09-19. - Primary source Stytch Device Fingerprinting is the product meant for blocking bad bots, with a frontend script and a backend Lookup API. [Source](https://isagent-web.vercel.app/faq), read 2026-11-04. - Primary source Stytch's free tier covers 10,000 monthly active users and AI agents plus 1,000 M2M tokens. [Source](https://stytch.com/pricing), read 2026-11-04. - Primary source Device fingerprints beyond the 10,000 included cost USD 0.005 each. [Source](https://stytch.com/pricing), read 2026-11-04. - Secondary source Stytch announced it has joined Twilio to build an identity layer that tells humans, trusted agents and rogue agents apart. [Source](https://www.plushcap.com/content/stytch/blog/stytch-stytch-has-joined-twilio-to-build-the-intelligent-identity-layer-for-the-internet), read 2026-11-04. ## Compared with - [Skyfire vs Stytch IsAgent](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent/): Skyfire issues signed KYA and pay tokens that sellers verify against its JWKS; Stytch IsAgent is a front-end flag that Stytch says must not be used as a security control. - [Stytch IsAgent vs Auth0 for AI Agents](https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents/): Stytch IsAgent flags inbound agent traffic on a site; Auth0 for AI Agents gives identities and scoped tokens to agents a company builds, and does not police inbound bots. ## What Stytch IsAgent does for this layer ### From the vendor's own pages Stytch's Web Bot Auth announcement of 23 Sep 2025 named Browserbase, Anchor Browser, Cloudflare and Akamai as launch partners. ([stytch.com](https://stytch.com/blog/stytch-supports-web-bot-auth/), dated 2025-09-23) Stytch reads the Signature-Agent header to find an agent's public key directory, so an agent that sends a valid directory URL can be verified without any account at Stytch. ([stytch.com](https://stytch.com/blog/stytch-supports-web-bot-auth/), dated 2025-09-23) Customers already running IsAgent or Device Fingerprinting received Web Bot Auth verification metadata in their existing API responses, with no integration change. ([changelog.stytch.com](https://changelog.stytch.com/announcements/ann_uSpDBNlbdXL8X), dated 2025-09-19) Stytch's signing guide for agent operators generates an Ed25519 key pair with the jose library, serves it at /.well-known/http-message-signatures-directory with the content type application/http-message-signatures-directory+json, and signs requests with tag="web-bot-auth" plus created and expires timestamps. ([stytch.com](https://stytch.com/blog/how-to-implement-web-bot-auth-signing/), dated 2025-09-23) Stytch publishes the signing walkthrough as runnable code in the GitHub repository stytchauth/web-bot-auth-example. ([stytch.com](https://stytch.com/blog/how-to-implement-web-bot-auth-signing/), dated 2025-09-23) Twilio agreed to acquire Stytch on 30 Oct 2025 and completed the deal on 14 Nov 2025; Stytch, Inc. stays the contracted service provider and Twilio Inc. is the parent company. ([changelog.stytch.com](https://changelog.stytch.com/announcements/2025-11-14-a-new-chapter-begins-stytch-joins-twilio), dated 2025-11-14) Twilio describes Stytch as "an identity platform for AI agents that's built for developers" and says it will anchor an identity layer across the Twilio Platform. ([twilio.com](https://www.twilio.com/en-us/blog/company/news/twilio-to-acquire-stytch), dated 2025-11-14) Stytch Connected Apps turns a customer's application into an OAuth 2.0 and OIDC identity provider, so AI agents and MCP clients receive scoped tokens that inherit only the permissions the user already holds. ([stytch.com](https://stytch.com/connected-apps)) Connected Apps lets an organisation define allowlists of the third-party apps and AI agents its members may connect, and revoke any token from the dashboard or the API. ([stytch.com](https://stytch.com/connected-apps)) ### Claims the vendor makes about itself Stytch's homepage claims "99.99% bot detection accuracy" for its bot and fraud protection. ([stytch.com](https://stytch.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | IsAgent | Front-end SDK that labels likely agent and programmatic traffic and reports Web Bot Auth verification results. | free tier, then contact sales free up to 10,000 calls a month per the IsAgent FAQ | USD 0 | 2026-11-05 | | Device Fingerprinting | Script plus Lookup API that returns a verdict per device, used for blocking bots, now carrying Web Bot Auth metadata. | per fingerprint above the included volume per fingerprint beyond the 10,000 included | USD 0.005 | 2026-11-05 | | Connected Apps | OAuth 2.0 and OIDC provider for AI agents, MCP servers and cross-app logins, with consent and org allowlists. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Verifies Web Bot Auth signatures through IsAgent and Device Fingerprinting since September 2025 and reads Signature-Agent for key discovery. (stytch.com) | | Published IP list | No statement | Stytch's Web Bot Auth post names reverse DNS and IP tracking only as the old way to identify Googlebot; no IP-list verification product is described. | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol mention on the Web Bot Auth announcement, changelog or Connected Apps page (checked 2026-11-05). | | Mastercard Agent Pay | No statement | No Agent Pay mention on the pages read (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Browserbase cloud browsers](https://agentreadycrawlers.com/crawlers/browserbase/) no token - [Anchor Browser](https://agentreadycrawlers.com/crawlers/anchor-browser/) no token - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token - [Alternatives to Stytch (IsAgent, Device Fingerprinting, Connected Apps)](https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives/) graded on the rubric ## Sources - [stytch.com: stytch.com](https://stytch.com) read 2026-11-04 - [stytch.com: pricing](https://stytch.com/pricing) read 2026-11-04 - [changelog.stytch.com: ann\_uSpDBNlbdXL8X](https://changelog.stytch.com/announcements/ann_uSpDBNlbdXL8X) read 2026-11-04 - [isagent-web.vercel.app: faq](https://isagent-web.vercel.app/faq) read 2026-11-04 - [changelog.stytch.com: ann\_iPdXgiqlNrBk0](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0) read 2026-11-04 - [plushcap.com: stytch stytch has joined twilio to build the intelligent identity layer for the internet](https://www.plushcap.com/content/stytch/blog/stytch-stytch-has-joined-twilio-to-build-the-intelligent-identity-layer-for-the-internet) read 2026-11-04 - [Stytch supports Web Bot Auth for agent and bot verification](https://stytch.com/blog/stytch-supports-web-bot-auth/) read 2026-11-05 - [How to implement Web Bot Auth for your AI agent, bot, or crawler](https://stytch.com/blog/how-to-implement-web-bot-auth-signing/) read 2026-11-05 - [Stytch changelog: A new chapter begins, Stytch joins Twilio](https://changelog.stytch.com/announcements/2025-11-14-a-new-chapter-begins-stytch-joins-twilio) read 2026-11-05 - [Twilio acquired Stytch, an identity platform for AI agents](https://www.twilio.com/en-us/blog/company/news/twilio-to-acquire-stytch) read 2026-11-05 - [Connected Apps | Stytch](https://stytch.com/connected-apps) read 2026-11-05 - [Stytch homepage](https://stytch.com/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Stytch IsAgent | Agent Ready Crawlers" description: "Alternatives to Stytch IsAgent: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head." canonical: https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Stytch IsAgent | Agent Ready Crawlers > Alternatives to Stytch IsAgent: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Source: https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Stytch IsAgent In short Stytch IsAgent ranks 14th of 15 with 2 of 10 points, ahead of Netlify only, and Arcjet is nearest on points. Head-to-head pages cover Skyfire and Auth0 for AI Agents. IsAgent offers a free allowance of 10,000 calls a month and, by Stytch's own description, is not a security control. ## Head-to-head, with the sourced difference - [**Stytch IsAgent vs Skyfire**](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent/). Skyfire issues signed KYA and pay tokens that sellers verify against its JWKS; Stytch IsAgent is a front-end flag that Stytch says must not be used as a security control. [Source](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0), read 2026-11-04. - [**Stytch IsAgent vs Auth0 for AI Agents**](https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents/). Stytch IsAgent flags inbound agent traffic on a site; Auth0 for AI Agents gives identities and scoped tokens to agents a company builds, and does not police inbound bots. [Source](https://auth0.com/ai), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Stytch IsAgent's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Stytch IsAgent on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, per-purpose policy, charging, commerce agents, reporting | nothing | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy, charging, reporting | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, charging, commerce agents, reporting | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, per-purpose policy, charging, commerce agents, reporting | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy, charging, reporting | nothing | No public price; sales-led | | Skyfire | 6 | verifies agents, charging, commerce agents | reporting | Not published (checked 2026-11-04) | | TollBit | 6 | per-purpose policy, charging, reporting | nothing | Free for publishers to start | | Vercel | 6 | verifies agents, per-purpose policy, reporting | nothing | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents, per-purpose policy, reporting | nothing | No public price; sales-led | | Fastly | 4 | per-purpose policy, charging, commerce agents | reporting | No public price; sales-led | | Arcjet | 3 | per-purpose policy | nothing | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | per-purpose policy, commerce agents | verifies agents | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | reporting | nothing | No public price; sales-led | | Netlify | 1 | per-purpose policy | verifies agents, reporting | Platform plans from USD 0; extension price not stated | ## Where Stytch IsAgent is not the fit - IsAgent is not a security control; its client-side result can be tampered with (Stytch changelog, 2025-08-01). - Detection is heuristic and not 100% accurate (IsAgent FAQ). - No crawler monetization or per-purpose crawler policy (pages read 2026-11-04). ## Sources - [stytch.com: pricing](https://stytch.com/pricing) read 2026-11-04 - [changelog.stytch.com: ann\_iPdXgiqlNrBk0](https://changelog.stytch.com/announcements/ann_iPdXgiqlNrBk0) read 2026-11-04 - [auth0.com: ai](https://auth0.com/ai) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Supertab: licensing intermediary profile | Agent Ready Crawlers" description: "Supertab: licensing intermediary profile covering Supertab Connect and Supertab Pass. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/supertab site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Supertab: licensing intermediary profile | Agent Ready Crawlers > Supertab: licensing intermediary profile covering Supertab Connect and Supertab Pass. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/supertab Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Licensing intermediary # Supertab In short Supertab pairs Supertab Pass, a pay-as-you-go tab for human readers, with Connect, which classifies and licenses non-human traffic on Fastly, CloudFront, Cloudflare Workers or WordPress, from USD 99 a month. A crawler that never asks for a token is only logged or blocked, and Supertab does not publish its revenue share. Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [supertab.co](https://www.supertab.co) In one line Managed RSL licensing service: Supertab Connect hosts a publisher's license.xml and RSL license server, issues license tokens to crawlers, checks them at the CDN with its SDK and bills the use, priced from USD 99 a month. Last read 2026-11-05 ## What Supertab does for this layer ### From the vendor's own pages Supertab Connect's Starter plan costs USD 99 a month for publishers with up to 2.5 million monthly visitors, 5 million bot classifications a month, one active licensee and a 30-day free trial. ([supertab.co](https://www.supertab.co/pricing)) Supertab's Enterprise plan adds payments infrastructure, disbursement and merchant-of-record services, and Supertab says it meters, prices and settles licensed access and pays publishers monthly. ([supertab.co](https://www.supertab.co/pricing)) Supertab Connect's SDK enforces the Crawler Authentication Protocol at the CDN or in the application: it detects bot traffic, validates the license token signature and claims, allows or blocks, and logs the event for billing, while human traffic passes untouched. ([connect-docs.supertab.co](https://connect-docs.supertab.co/introduction/overview)) In Supertab Connect a crawler finds the publisher's terms through a License: directive in robots.txt, reads the RSL license.xml with the usage values search, ai-input, ai-index and ai-train, and requests a token if licensed access is offered. ([connect-docs.supertab.co](https://connect-docs.supertab.co/introduction/overview)) Supertab Connect documents deployments on Fastly Compute or VCL, AWS CloudFront, Cloudflare Workers and WordPress, with SDKs in TypeScript, PHP and Python for other CDNs. ([connect-docs.supertab.co](https://connect-docs.supertab.co/llms.txt)) Supertab describes itself as infrastructure for consumption-based commerce, pairing Supertab Pass for human readers with Supertab Connect for classifying and licensing non-human traffic. ([supertab.co](https://www.supertab.co/about)) Every Supertab Connect plan includes opt-in participation in licensing demand sources, which Supertab lists as marketplaces, collectives and Supertab-sourced demand. ([supertab.co](https://www.supertab.co/pricing)) ### Claims the vendor makes about itself Supertab says it runs the full OAuth 2.0 token endpoint for RSL, with client credential flows, license payload validation, token issuance, revocation and error handling, and calls the server "Fully compliant with RSL 1.0". ([supertab.co](https://www.supertab.co/)) Supertab's homepage claims that more than 50% of web crawl traffic now comes from AI bots and automated agents and that going live with AI licensing takes hours. ([supertab.co](https://www.supertab.co/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Supertab Connect | Hosted RSL license, license server (OLP), edge token checks (CAP), bot classification and billing for AI access. | monthly subscription by visitors and licensees Pro USD 249 a month; Enterprise custom | USD 99 | 2026-11-05 | | Supertab Pass | Pay-as-you-go tab for human readers. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Crawlers discover the publisher's RSL license through a License: directive in robots.txt. (connect-docs.supertab.co) | | Web Bot Auth | No statement | Crawler identity runs through RSL license tokens checked by CAP; the docs index does not mention Web Bot Auth (checked 2026-11-05). | | Content Signals | No statement | No Content-Signals mention on the homepage, pricing page or docs index (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Supertab does not do - Supertab Connect cannot make an unlicensed crawler pay; unlicensed requests are logged or blocked at the edge, and the crawler has to choose to request a token. - Supertab does not publish the share it keeps from licensing revenue settled through Connect (pricing page read 2026-11-05). ## Reading notes Supertab is the former Laterpay business, judging by a careers link that points to laterpay.jobs.personio.de, but no Supertab page says so in words, so it is not stated as a fact. ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/) bot management - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management ## Sources - [supertab.co](https://www.supertab.co) read 2026-11-05 - [Supertab Pricing](https://www.supertab.co/pricing) read 2026-11-05 - [Supertab Connect: Managed Infrastructure for AI Content Licensing](https://www.supertab.co/) read 2026-11-05 - [Supertab Connect docs: overview](https://connect-docs.supertab.co/introduction/overview) read 2026-11-05 - [Supertab Connect docs index (llms.txt)](https://connect-docs.supertab.co/llms.txt) read 2026-11-05 - [About Supertab](https://www.supertab.co/about) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Timpi: crawler operator profile | Agent Ready Crawlers" description: "Timpi: crawler operator profile covering Timpi Search, Timpi Data and Timpi DePIN nodes. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/timpi site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Timpi: crawler operator profile | Agent Ready Crawlers > Timpi: crawler operator profile covering Timpi Search, Timpi Data and Timpi DePIN nodes. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/timpi Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Timpi In short Timpi's crawling is done by contributors' own machines, nodes that earn $NTMPI tokens for collecting data, and the company says its index holds more than 2.2 billion pages. Site owners meet the crawler only through other sites' robots.txt files, where it appears as Timpibot. Timpi gives no opt-out route, and the crawler page paths we tried returned 404. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [timpi.io](https://timpi.io) In one line Operator of an independent search index built by a decentralised network of contributor nodes; its crawler appears in robots.txt files as Timpibot, but Timpi publishes no crawler documentation for site owners. Last read 2026-11-05 ## What Timpi does for this layer ### From the vendor's own pages Timpi sells four products on its own index: Timpi Data (APIs for developers), ORCA (workflow orchestration), Timpi Ads ("Advertising for humans and AI agents") and Timpi Search. ([timpi.io](https://timpi.io/)) Timpi's DePIN page tells contributors to buy a Node Access NFT, assemble the recommended hardware and register a node to earn rewards in $NTMPI tokens. ([timpi.io](https://timpi.io/depin)) ### Claims the vendor makes about itself Timpi's About page claims an index of more than 2.2 billion web pages and more than 100 billion indexed data items. ([timpi.io](https://timpi.io/about)) Timpi says its infrastructure runs on more than 1,700 decentralised nodes with more than 1 PB of distributed storage and 16,000 TFLOPS of compute. ([timpi.io](https://timpi.io/about)) Timpi's comparison post names Collector, Guardian and GeoCore nodes and says a component it calls the Timpi Autonomous Protocol coordinates crawling and indexing. ([timpi.io](https://timpi.io/timpi-vs-other-search-engines)) Timpi's blog says its node network once indexed over 5 billion pages in 48 hours during tests, and that its nodes run on machines already in use by contributors rather than in data centers. ([timpi.io](https://timpi.io/timpi-vs-other-search-engines)) ### What our crawl saw Requests for timpi.io/timpibot, /crawler and /bot returned HTTP 404 on 2026-11-05, and the timpi.io sitemap read the same day lists no crawler or bot page. ([timpi.io](https://timpi.io/sitemap.xml)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Timpi Search | Search engine on Timpi's own index. | No price recorded | | Timpi Data | APIs that give developers and AI systems access to the index. | No price recorded | | Timpi DePIN nodes | Contributor-run nodes that collect and process data for the index, rewarded in $NTMPI tokens. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | No robots.txt statement for Timpibot on timpi.io (sitemap and bot paths checked 2026-11-05). | | Web Bot Auth | No statement | No key directory at timpi.io (HTTP 404, 2026-11-05). | | Published IP list | No statement | No IP list published (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://timpi.io/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Timpi does not do - Timpi publishes no user agent string, robots.txt statement, IP list or signature key directory for Timpibot (checked 2026-11-05). - Timpi gives site owners no stated opt-out route for its crawler on any page read. ## Reading notes The Timpibot token comes from robots.txt files in our own crawl of the web, not from Timpi; see the [Timpibot row](https://agentreadycrawlers.com/crawlers/timpibot/). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Timpibot](https://agentreadycrawlers.com/crawlers/timpibot/) token Timpibot ## Sources - [timpi.io](https://timpi.io) read 2026-11-05 - [About Timpi | Building the Infrastructure for the Next Web | Timpi](https://timpi.io/about) read 2026-11-05 - [Timpi | Building the Intelligent Web](https://timpi.io/) read 2026-11-05 - [DePIN | Timpi](https://timpi.io/depin) read 2026-11-05 - [Timpi vs Other Search Engines | Timpi Blog](https://timpi.io/timpi-vs-other-search-engines) read 2026-11-05 - [timpi.io sitemap](https://timpi.io/sitemap.xml) read 2026-11-05 - [Timpi key directory probe](https://timpi.io/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "TollBit: bot management profile | Agent Ready Crawlers" description: "TollBit: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15." canonical: https://agentreadycrawlers.com/vendors/tollbit site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # TollBit: bot management profile | Agent Ready Crawlers > TollBit: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15. Source: https://agentreadycrawlers.com/vendors/tollbit Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 6 (tied) of 15 # TollBit In short TollBit is a paywall for AI bots, charging per thousand pages: charging and reporting grade yes, while agent verification and per-purpose policy are partial because classifying visitors is left to CDN partners, for 6 of 10 points. Commerce agents are not published, and signing up costs a publisher nothing. Platform that lets publishers see AI bot traffic, set rules and per-access rates, and send bots to a paywall where AI companies with valid TollBit tokens get content; detection relies on CDN and bot-management partners. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to TollBit](https://agentreadycrawlers.com/vendors/tollbit/alternatives/) · [tollbit.com](https://tollbit.com) Type [Licensing intermediary](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) Homepage [tollbit.com](https://tollbit.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Partial | issues access tokens to AI companies and depends on CDN or bot-manager partners for detection; Web Bot Auth support not confirmed (checked 2026-11-04) | Fastly flow grants access to bots with valid TollBit tokens. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Partial | rules and rates per bot set by the publisher | Homepage: enforcing rules and rates for content access. Source, read 2026-11-04. | | Charging for access | Yes | bot paywall with publisher-set usage-based pricing | Homepage paywall section. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not on homepage. Source, read 2026-11-04. | | Cost | Not graded | free for publishers to start | Homepage: Get started free. Source, read 2026-11-04. | | Reporting | Yes | TollBit Analytics and portfolio analytics across sites | Homepage analytics section. Source, read 2026-11-04. | | Where it runs | Not graded | integrates at the CDN or bot-manager layer (Fastly, Akamai, DataDome, Arc XP) by redirect | Partner integrations. Source, read 2026-11-04. | ## Cost Free for publishers to start ('Get started free'); publishers set their own per-access rates; TollBit's fees to AI companies are not published (checked 2026-11-04). [Source](https://tollbit.com/), read 2026-11-04. ## What TollBit does not do - Does not detect bots on its own network edge; it relies on CDN and bot-management partners to classify and redirect (Fastly integration blog). - No published fee schedule for AI companies (homepage read 2026-11-04). - Usage statistics are TollBit's own counts, not independently audited (homepage). - TollBit does not verify a bot's identity cryptographically; routing to the paywall keys on the user agent string the CDN sees. - TollBit does not publish the transaction fee it adds for AI customers (docs read 2026-11-05). ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Vendor claim TollBit describes a free platform for publishers to see how agents use their sites, separate AI from human traffic, and enforce rules and rates for content access. [Source](https://tollbit.com/), read 2026-11-04. - Vendor claim TollBit's bot and agent paywall lets publishers set usage-based prices for AI access. [Source](https://tollbit.com/), read 2026-11-04. - Vendor claim TollBit reports for H1 2026 more than 987 billion website visits analyzed and over 22 billion AI bot scrapes detected. [Source](https://tollbit.com/), read 2026-11-04, dated 2026-06-30. - Vendor claim TollBit reports over 1.9 billion bot scrapes in H1 2026 bypassed or ignored publishers' robots.txt instructions. [Source](https://tollbit.com/), read 2026-11-04, dated 2026-06-30. - Vendor claim TollBit says more than 2.6 billion AI bots were directed to its Bot Paywall in H1 2026. [Source](https://tollbit.com/), read 2026-11-04, dated 2026-06-30. - Vendor claim TollBit says it has raised over USD 31 million. [Source](https://tollbit.com/), read 2026-11-04. - Vendor claim TollBit keeps a bot index, a directory of bots, agents and crawlers used across its analytics, routing and licensing. [Source](https://tollbit.com/bot-index/), read 2026-11-04. - Primary source In Fastly's integration, Fastly classifies the AI bot and redirects it to the publisher's TollBit Bot Paywall; AI companies with valid tokens get access. [Source](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit), read 2026-11-04. - Primary source Arc XP, the Washington Post's publishing platform, partnered with TollBit on 23 Mar 2026 so its publishers can block AI bots or redirect them to a TollBit paywall. [Source](https://arcxp.com/2026/03/23/arc-xp-partners-with-tollbit-to-help-publishers-monitor-control-and-monetize-ai-bot-traffic), read 2026-11-04, dated 2026-03-23. - Primary source DataDome and TollBit announced a partnership for real-time control and monetization of AI agents at Black Hat USA 2025. [Source](https://www.businesswire.com/news/home/20250728685477/en/DataDome-Brings-Real-Time-Control-Monetization-Of-AI-Agents-in-New-Partnership-with-TollBit-at-Black-Hat-USA-2025), read 2026-11-04, dated 2025-07-28. - Primary source Akamai partners with TollBit and Skyfire to turn AI bot traffic into revenue for publishers. [Source](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue), read 2026-11-04. ## Compared with - [TollBit vs Cloudflare](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/): TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. - [Skyfire vs TollBit](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit/): Skyfire is the payer side: agents carry kya-pay tokens with committed funds; TollBit is the publisher side: a paywall with publisher-set rates that AI companies access with TollBit tokens. - [TollBit vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control/): TollBit sells publisher analytics and a paywall reached by redirect; AWS WAF returns HTTP 402 itself and settles through Coinbase. ## What TollBit does for this layer ### From the vendor's own pages TollBit's Agent Site lives on a tollbit subdomain of the publisher's domain, for example tollbit.yourdomain.com. ([docs.tollbit.com](https://docs.tollbit.com/docs/step-3-configure-agent-site.md)) TollBit provisions that subdomain after the publisher proves ownership of the domain with a DNS TXT record and points NS records for the tollbit subdomain at four TollBit nameservers. ([docs.tollbit.com](https://docs.tollbit.com/docs/verifying-a-property.md), dated 2026-10-12) TollBit's redirect runs as edge logic at the publisher's CDN, such as Cloudflare or Akamai, which checks each request against a list of known bot user agents and sends matches to the tollbit subdomain. ([docs.tollbit.com](https://docs.tollbit.com/docs/platform-architecture.md)) TollBit says it concentrates on self-identified bots such as ChatGPT, Perplexity and Claude and leaves undeclared scrapers to partner tools from DataDome and HUMAN Security. ([docs.tollbit.com](https://docs.tollbit.com/docs/bot-management.md), dated 2026-10-23) TollBit updates its bot list about once a quarter and expects publishers to add new crawlers to their edge configuration or robots.txt by hand. ([docs.tollbit.com](https://docs.tollbit.com/docs/bot-management.md), dated 2026-10-23) TollBit tokens are JWTs of type content or crawl that expire 5 minutes after issuance and bind the requesting user agent in the claim x.ua. ([docs.tollbit.com](https://docs.tollbit.com/docs/tokens.md), dated 2026-03-20) A TollBit content token carries the maximum price in micros, where 1 USD equals 1,000,000 micros, plus the currency, the license type and the license ID. ([docs.tollbit.com](https://docs.tollbit.com/docs/tokens.md), dated 2026-03-20) TollBit offers publishers a Summarization License and a Full Display License, each priced per 1,000 pages accessed, and a Custom License attached to a named partner's user agents. ([docs.tollbit.com](https://docs.tollbit.com/docs/setting-rates.md), dated 2026-06-25) TollBit resolves a price by checking rates in the order bot, page, keyword, time and subdirectory; a page that matches no rate cannot be accessed by AI customers. ([docs.tollbit.com](https://docs.tollbit.com/docs/setting-rates.md), dated 2026-06-25) TollBit states it takes no percentage of publisher rates and instead charges AI customers a transaction fee on top of the rate the publisher sets. ([docs.tollbit.com](https://docs.tollbit.com/docs/setting-rates.md), dated 2026-06-25) TollBit documents integrations for Akamai, Arc XP, AWS CloudFront and ALB, Cloudflare, DataDome, Fastly, Google Cloud CDN, Azure Front Door, Vercel, WordPress VIP, Imperva and Netacea. ([docs.tollbit.com](https://docs.tollbit.com/llms.txt)) TollBit also exposes a publisher's content through NLWeb, with /ask and /mcp endpoints on the tollbit subdomain, and through Google's Agent2Agent protocol. ([docs.tollbit.com](https://docs.tollbit.com/docs/other-access-methods.md), dated 2026-06-14) After a bot presents a valid token, TollBit fetches the page through its reverse proxy at api.tollbit.com/GetContent and returns cleaned, structured content instead of the raw HTML. ([docs.tollbit.com](https://docs.tollbit.com/docs/bot-management.md), dated 2026-10-23) ### Claims the vendor makes about itself TollBit's homepage shows publisher sites including reuters.com, apnews.com, economist.com, forbes.com and pbs.org in its customer strip. ([tollbit.com](https://tollbit.com/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Agent Site | Subdomain gateway for AI agents with access controls, per-agent content and paywall enforcement. | free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee transaction fee amount not published | USD 0 | 2026-11-05 | | Bot Paywall | Paid access terms for AI bots, priced per 1,000 pages by license type. | No price recorded | | TollBit Analytics | Bot traffic analytics from CDN, CMS, security tool or cloud bucket log ingestion. | No price recorded | | Licensed RAG Access and Licensed Search | API for AI developers to search, check rates and retrieve licensed content with tokens. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | TollBit relies on edge redirects by user agent and asks publishers to keep robots.txt current with new crawlers; it does not manage robots.txt itself. (docs.tollbit.com) | | Web Bot Auth | No statement | TollBit's docs index and FAQ describe identification by user agent and partner tools, with no Web Bot Auth mention (checked 2026-11-05). | | Content Signals | No statement | No Content-Signals mention in the docs index (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All licensing intermediaries](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) 8 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Akamai (Bot & Agent Control)](https://agentreadycrawlers.com/vendors/akamai/) bot management - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management - [DataDome (Bot Protect, Agentic Trust)](https://agentreadycrawlers.com/vendors/datadome/) bot management - [Fastly (Bot Management, AI Bot Management)](https://agentreadycrawlers.com/vendors/fastly/) bot management - [HUMAN Security (Sightline, AgenticTrust)](https://agentreadycrawlers.com/vendors/human-security/) bot management - [Imperva, a Thales company (Advanced Bot Protection)](https://agentreadycrawlers.com/vendors/imperva/) bot management - [Vercel (BotID, Firewall bot management)](https://agentreadycrawlers.com/vendors/vercel-botid/) bot management - [Alternatives to TollBit](https://agentreadycrawlers.com/vendors/tollbit/alternatives/) graded on the rubric ## Sources - [tollbit.com: tollbit.com](https://tollbit.com) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [fastly.com: how to control and monetize ai bot traffic using fastly and tollbit](https://www.fastly.com/blog/how-to-control-and-monetize-ai-bot-traffic-using-fastly-and-tollbit) read 2026-11-04 - [arcxp.com: arc xp partners with tollbit to help publishers monitor control and monetize ai bot traffic](https://arcxp.com/2026/03/23/arc-xp-partners-with-tollbit-to-help-publishers-monitor-control-and-monetize-ai-bot-traffic) read 2026-11-04 - [tollbit.com: bot index](https://tollbit.com/bot-index/) read 2026-11-04 - [businesswire.com: DataDome Brings Real Time Control Monetization Of AI Agents in New Partnership with TollBit at Black Hat USA 2025](https://www.businesswire.com/news/home/20250728685477/en/DataDome-Brings-Real-Time-Control-Monetization-Of-AI-Agents-in-New-Partnership-with-TollBit-at-Black-Hat-USA-2025) read 2026-11-04 - [akamai.com: no free crawls akamai tollbit and skyfire turn traffic into revenue](https://www.akamai.com/newsroom/press-release/no-free-crawls-akamai-tollbit-and-skyfire-turn-traffic-into-revenue) read 2026-11-04 - [TollBit docs: Step 3, Configure Agent Site](https://docs.tollbit.com/docs/step-3-configure-agent-site.md) read 2026-11-05 - [Step 1: Verify Property (TollBit docs)](https://docs.tollbit.com/docs/verifying-a-property.md) read 2026-11-05 - [TollBit docs: Architecture FAQ](https://docs.tollbit.com/docs/platform-architecture.md) read 2026-11-05 - [TollBit docs: Agent Site FAQ (bot management, monetization)](https://docs.tollbit.com/docs/bot-management.md) read 2026-11-05 - [TollBit docs: Tokens](https://docs.tollbit.com/docs/tokens.md) read 2026-11-05 - [TollBit docs: Enabling Monetization](https://docs.tollbit.com/docs/setting-rates.md) read 2026-11-05 - [TollBit documentation index (llms.txt)](https://docs.tollbit.com/llms.txt) read 2026-11-05 - [TollBit docs: Access Methods (NLWeb, MCP, A2A)](https://docs.tollbit.com/docs/other-access-methods.md) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to TollBit | Agent Ready Crawlers" description: "Alternatives to TollBit: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/tollbit/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to TollBit | Agent Ready Crawlers > Alternatives to TollBit: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/tollbit/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to TollBit In short TollBit shares sixth place on 6 of 10 points with Skyfire and Vercel, and three head-to-head pages compare it with Cloudflare, Skyfire and AWS WAF Bot Control. It works as the paywall that CDN partners send AI bots to, and publishers can start free of charge. ## Head-to-head, with the sourced difference - [**TollBit vs Cloudflare**](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare/). TollBit is a paywall that bot-management partners redirect AI bots to, with publisher-set rates; Cloudflare pay per crawl answers the crawler with HTTP 402 and a crawler-price header from its own network. [Source](https://blog.cloudflare.com/introducing-pay-per-crawl/), read 2026-11-04. - [**TollBit vs Skyfire**](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit/). Skyfire is the payer side: agents carry kya-pay tokens with committed funds; TollBit is the publisher side: a paywall with publisher-set rates that AI companies access with TollBit tokens. [Source](https://docs.skyfire.xyz/docs/payments-settlement.md), read 2026-11-04. - [**TollBit vs AWS WAF Bot Control**](https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control/). TollBit sells publisher analytics and a paywall reached by redirect; AWS WAF returns HTTP 402 itself and settles through Coinbase. [Source](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with TollBit's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than TollBit on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | verifies agents, commerce agents | charging | No public price; sales-led | | AWS WAF Bot Control | 8 | verifies agents, per-purpose policy | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | verifies agents, per-purpose policy, commerce agents | charging | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | verifies agents, commerce agents | nothing | From USD 3,830/month | | HUMAN Security | 7 | verifies agents, per-purpose policy | charging | No public price; sales-led | | Skyfire | 6 | verifies agents, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | Vercel | 6 | verifies agents, per-purpose policy | charging | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | | Kasada | 5 | verifies agents | charging | No public price; sales-led | | Fastly | 4 | commerce agents | charging, reporting | No public price; sales-led | | Arcjet | 3 | nothing | charging, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | commerce agents | verifies agents, charging, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | per-purpose policy, charging | No public price; sales-led | | Stytch IsAgent | 2 | nothing | per-purpose policy, charging, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, charging, reporting | Platform plans from USD 0; extension price not stated | ## Where TollBit is not the fit - Does not detect bots on its own network edge; it relies on CDN and bot-management partners to classify and redirect (Fastly integration blog). - No published fee schedule for AI companies (homepage read 2026-11-04). - Usage statistics are TollBit's own counts, not independently audited (homepage). ## Sources - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 - [docs.skyfire.xyz: payments settlement.md](https://docs.skyfire.xyz/docs/payments-settlement.md) read 2026-11-04 - [aws.amazon.com: aws waf ai traffic monetization](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Trulioo: agent verification provider profile | Agent Ready Crawlers" description: "Trulioo: agent verification provider profile covering Know Your Agent (KYA). Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/trulioo site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Trulioo: agent verification provider profile | Agent Ready Crawlers > Trulioo: agent verification provider profile covering Know Your Agent (KYA). Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/trulioo Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Trulioo In short Trulioo proposed Know Your Agent in August 2025 with a five-step flow, from checking the developer to ongoing lookups, and Worldpay was named as a merchant-side partner. The product is now in alpha: businesses behind an agent get KYB and AML checks and a signed, revocable profile that counterparties confirm by API or MCP, though access is by request. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [trulioo.com](https://www.trulioo.com/products/kya-verification) In one line Identity verification company whose Know Your Agent programme, in alpha, runs business and AML checks on the company behind an AI agent, binds the agent's fingerprint to it in a signed Digital Agent Profile and lets counterparties check it by API or MCP. Last read 2026-11-05 ## What Trulioo does for this layer ### From the vendor's own pages Trulioo's product page says Know Your Agent "is in Alpha" and that the first release binds a verified developer organisation and an agent fingerprint in a verifiable Digital Agent Profile. ([trulioo.com](https://www.trulioo.com/products/kya-verification)) Trulioo's KYA checks the developer organisation behind an agent with Know Your Business and anti-money-laundering checks and confirms that the organisation controls its domain. ([trulioo.com](https://www.trulioo.com/products/kya-verification)) Trulioo issues a signed, time-bound credential recording an agent's permitted scope, which can be revoked, and packages the result as a continuously updated JWT called the Digital Agent Profile. ([trulioo.com](https://www.trulioo.com/products/kya-verification)) Trulioo gives counterparties API and Model Context Protocol access to confirm an agent's profile, scope and status before they apply their own access policies. ([trulioo.com](https://www.trulioo.com/products/kya-verification)) Trulioo introduced KYA in a report co-written with the payments platform PayOS, described on its blog on 25 Aug 2025. ([trulioo.com](https://www.trulioo.com/blog/trust-and-safety/future-agentic-commerce-know-your-agent-kya), dated 2025-08-25) Trulioo's August 2025 blog listed five KYA steps: verify the agent developer, lock the agent code, capture user permission, issue a Digital Agent Passport and run ongoing lookups. ([trulioo.com](https://www.trulioo.com/blog/trust-and-safety/future-agentic-commerce-know-your-agent-kya), dated 2025-08-25) Trulioo said Worldpay would let merchants use the KYA framework to validate consumer intent and the authority granted to shopping agents. ([trulioo.com](https://www.trulioo.com/blog/trust-and-safety/future-agentic-commerce-know-your-agent-kya), dated 2025-08-25) ### What our crawl saw Trulioo's August 2025 blog called the token a Digital Agent Passport, while the product page of 2026-11-05 calls it a Digital Agent Profile; both use the abbreviation DAP. ([trulioo.com](https://www.trulioo.com/products/kya-verification)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Know Your Agent (KYA) | Developer KYB and AML checks, agent fingerprinting and a signed Digital Agent Profile with scope, checkable by API or MCP. | alpha, access on request page offers "Request access" only | Not published | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The KYA page refers to an agent's "cryptographic identity" without naming Web Bot Auth or HTTP Message Signatures (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol mention on the KYA page or blog (checked 2026-11-05). | | Mastercard Agent Pay | No statement | No Agent Pay mention on the KYA page or blog (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Trulioo does not do - Trulioo's KYA does not sit in front of a website or block traffic; counterparties query the profile and enforce their own policy. - Trulioo's KYA is in alpha with access by request and no published price (product page read 2026-11-05). ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types ## Sources - [Know Your Agent (KYA): AI Agent Verification (Trulioo)](https://www.trulioo.com/products/kya-verification) read 2026-11-05 - [The Future of Agentic Commerce: Scaling Trust With Know Your Agent (KYA)](https://www.trulioo.com/blog/trust-and-safety/future-agentic-commerce-know-your-agent-kya) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Vercel: bot management profile | Agent Ready Crawlers" description: "Vercel: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15." canonical: https://agentreadycrawlers.com/vendors/vercel-botid site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Vercel: bot management profile | Agent Ready Crawlers > Vercel: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15. Source: https://agentreadycrawlers.com/vendors/vercel-botid Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendor, ranked 6 (tied) of 15 # Vercel In short Vercel's BotID and firewall verify agents and apply per-purpose policy and reporting, grading yes on all three, for 6 of 10 points. Basic BotID is free, commerce agents are not published, and the docs read describe no means of charging AI crawlers. Hosting platform with an invisible CAPTCHA (BotID, Deep Analysis by Kasada), a verified bot directory checked by IP, reverse DNS and Web Bot Auth, and a firewall ruleset that treats AI crawlers and AI assistants separately. Vendor description of itself, checked against its documents on 2026-11-04. [Alternatives to Vercel](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/) · [vercel.com](https://vercel.com) Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [vercel.com](https://vercel.com) Last read 2026-11-05 ## The rubric, with evidence | Question | Grade | What the vendor documents | Evidence | | --- | --- | --- | --- | | Tells a verified agent from a spoof | Yes | IP, reverse DNS and Web Bot Auth; directory at bots.fyi | Changelog 2025-08-12 on Web Bot Auth support. Source, read 2026-11-04. | | Per-purpose policy (train, search, answer, act) | Yes | AI crawlers and AI assistants handled separately since 2026-10-29 | AI Bots managed ruleset changelog. Source, read 2026-11-04. | | Charging for access | Not published | support (checked 2026-11-04) | No charging feature in BotID or firewall docs read. Source, read 2026-11-04. | | Lets signed commerce agents through | Not published | Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) | Not mentioned in pages read. Source, read 2026-11-04. | | Cost | Not graded | published: Basic free; Deep Analysis USD 1 per 1,000 checks on Pro | BotID pricing table. Source, read 2026-11-04. | | Reporting | Yes | Firewall tab BotID filter and Observability Plus metrics | BotID observability section. Source, read 2026-11-04. | | Where it runs | Not graded | both: managed rulesets at Vercel's edge firewall; BotID verdict read in server code via checkBotId() | BotID validation flow and AI Bots ruleset. Source, read 2026-11-04. | ## Cost BotID Basic free on all plans; Deep Analysis USD 1 per 1,000 checkBotId() calls on Pro, custom on Enterprise; AI Bots managed ruleset on all plans. [Source](https://vercel.com/docs/botid), read 2026-11-04. ## What Vercel does not do - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). - Vercel's Bot Protection is not meant to run behind another CDN or reverse proxy; Vercel says detection degrades in that setup. ## Documented facts Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim. - Primary source BotID is an invisible client-side challenge for high-value routes such as checkout, signup and APIs, available on all plans. [Source](https://vercel.com/docs/botid), read 2026-11-04, dated 2026-06-16. - Primary source BotID Basic is free on all plans; Deep Analysis costs USD 1 per 1,000 checkBotId() calls on Pro and is custom-priced on Enterprise. [Source](https://vercel.com/docs/botid), read 2026-11-04, dated 2026-06-16. - Primary source Only calls to checkBotId() are charged; passive page views are not. [Source](https://vercel.com/docs/botid), read 2026-11-04, dated 2026-06-16. - Primary source BotID Deep Analysis is powered by Kasada. [Source](https://vercel.com/docs/botid), read 2026-11-04, dated 2026-06-16. - Primary source Vercel verifies known bots by IP, reverse DNS and Web Bot Auth, so signed bots pass Bot Protection and Challenge Mode; ChatGPT's operator agent was the cited example. [Source](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth), read 2026-11-04, dated 2025-08-12. - Primary source checkBotId() returns isVerifiedBot, verifiedBotName and verifiedBotCategory, which requires botid 1.5.0 or later. [Source](https://vercel.com/docs/botid/verified-bots), read 2026-11-04, dated 2026-02-26. - Primary source Vercel publishes its directory of known and verified bots at bots.fyi. [Source](https://vercel.com/docs/botid/verified-bots), read 2026-11-04, dated 2026-02-26. - Primary source Since 29 Oct 2026 the AI Bots managed ruleset can log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) separately, on all plans. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04, dated 2026-10-29. - Primary source BotID checks show in the project Firewall tab traffic filter, with metrics in Observability Plus. [Source](https://vercel.com/docs/botid), read 2026-11-04, dated 2026-06-16. ## Dated news - 2026-10-29**Vercel splits AI crawlers and AI assistants in its AI Bots ruleset**. Vercel's AI Bots managed ruleset can now log or block AI crawlers, which collect content for indexing and training, separately from AI assistants, which fetch pages for a user or search query. Vercel keeps the lists current and the ruleset is on all plans. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules). ## Compared with - [Arcjet vs Vercel](https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid/): Arcjet verifies known bots by IP and reverse DNS inside application code through its SDK; Vercel verifies by IP, reverse DNS and Web Bot Auth at its edge and splits AI crawlers from AI assistants in a managed ruleset since 2026-10-29. - [Cloudflare vs Vercel](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid/): Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. - [Vercel vs Netlify](https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify/): Vercel verifies bots by IP, reverse DNS and Web Bot Auth; Netlify's AI bot control is a User Agent Blocker extension that matches strings only and can be active on one project at a time. - [Kasada vs Vercel](https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid/): Kasada powers Vercel BotID Deep Analysis, sold by Vercel at USD 1 per 1,000 checkBotId() calls on Pro; Kasada itself is sales-led and adds an AI agent directory and edge policies. - [Vercel vs AWS WAF Bot Control](https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control/): Vercel splits AI crawlers from AI assistants on all plans for free; AWS WAF labels bots by category and name and can charge them, at USD 10 a month per web ACL plus request fees. ## What Vercel does for this layer ### From the vendor's own pages Vercel's bot management docs say Bot Protection does not work when a reverse proxy such as Cloudflare or Azure sits in front of a Vercel deployment, because the proxy masks detection signals. ([vercel.com](https://vercel.com/docs/bot-management)) Vercel's bot protection managed ruleset, on all plans, serves a JavaScript challenge to clients that break browser-like behavior. ([vercel.com](https://vercel.com/docs/bot-management)) Vercel's docs give the reverse DNS example that an IP claiming to be Googlebot should resolve to \*.googlebot.com or \*.google.com. ([vercel.com](https://vercel.com/docs/bot-management)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | AI Bots managed ruleset | Log or block AI crawlers and AI assistants separately in Vercel Firewall. | included Available on all plans. | USD 0 | 2026-11-05 | | BotID | Invisible client-side challenge for routes such as checkout, signup and APIs. | per call Basic is free on all plans; Enterprise custom. | 1 per 1,000 checkBotId() calls (Deep Analysis, Pro) | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Vercel lists Web Bot Authentication with HTTP Message Signatures among its verification methods. (vercel.com) | | Published IP list | Supports | Vercel checks requests against known IP ranges of bot operators. (vercel.com) | | Reverse DNS check | Supports | Vercel performs reverse DNS lookups on claimed bots. (vercel.com) | | Visa Trusted Agent Protocol | No statement | No statement in the Vercel docs read 2026-11-05. | | Mastercard Agent Pay | No statement | No statement in the Vercel docs read 2026-11-05. | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Alternatives to Vercel (BotID, Firewall bot management)](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives/) graded on the rubric ## Sources - [vercel.com: vercel.com](https://vercel.com) read 2026-11-04 - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [vercel.com: vercels bot verification now supports web bot auth](https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [vercel.com: verified bots](https://vercel.com/docs/botid/verified-bots) read 2026-11-04 - [Bot Management | Vercel Docs](https://vercel.com/docs/bot-management) read 2026-11-05 - [bots.fyi (Vercel's known and verified bots directory)](https://bots.fyi/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Alternatives to Vercel | Agent Ready Crawlers" description: "Alternatives to Vercel: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04." canonical: https://agentreadycrawlers.com/vendors/vercel-botid/alternatives site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Vendors" --- # Alternatives to Vercel | Agent Ready Crawlers > Alternatives to Vercel: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. Source: https://agentreadycrawlers.com/vendors/vercel-botid/alternatives Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Alternatives to Vercel In short Vercel shares sixth place on 6 of 10 points with Skyfire and TollBit, and five head-to-head pages compare it with Arcjet, Cloudflare, Netlify, Kasada and AWS WAF Bot Control. The basic BotID tier is free, and Vercel's docs read offer no way to charge AI crawlers. ## Head-to-head, with the sourced difference - [**Vercel vs Arcjet**](https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid/). Arcjet verifies known bots by IP and reverse DNS inside application code through its SDK; Vercel verifies by IP, reverse DNS and Web Bot Auth at its edge and splits AI crawlers from AI assistants in a managed ruleset since 2026-10-29. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. - [**Vercel vs Cloudflare**](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid/). Cloudflare publishes a no-training preference into robots.txt with Disallow AI Training (2026-09-15); Vercel's AI Bots ruleset logs or blocks AI crawlers and AI assistants separately (2026-10-29) without writing robots.txt. [Source](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules), read 2026-11-04. - [**Vercel vs Netlify**](https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify/). Vercel verifies bots by IP, reverse DNS and Web Bot Auth; Netlify's AI bot control is a User Agent Blocker extension that matches strings only and can be active on one project at a time. [Source](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers), read 2026-11-04. - [**Vercel vs Kasada**](https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid/). Kasada powers Vercel BotID Deep Analysis, sold by Vercel at USD 1 per 1,000 checkBotId() calls on Pro; Kasada itself is sales-led and adds an AI agent directory and edge policies. [Source](https://vercel.com/docs/botid), read 2026-11-04. - [**Vercel vs AWS WAF Bot Control**](https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control/). Vercel splits AI crawlers from AI assistants on all plans for free; AWS WAF labels bots by category and name and can charge them, at USD 10 a month per web ACL plus request fees. [Source](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html), read 2026-11-04. ## All other vendors on the rubric "Stronger" and "weaker" compare each vendor's grade with Vercel's on the five graded questions. Cost is quoted as read on 2026-11-04. | Vendor | Points | Stronger than Vercel on | Weaker on | Cost | | --- | --- | --- | --- | --- | | Akamai | 8 | charging, commerce agents | per-purpose policy | No public price; sales-led | | AWS WAF Bot Control | 8 | charging | nothing | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance | | Cloudflare | 8 | charging, commerce agents | nothing | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led | | DataDome | 8 | charging, commerce agents | per-purpose policy | From USD 3,830/month | | HUMAN Security | 7 | charging | nothing | No public price; sales-led | | Skyfire | 6 | charging, commerce agents | per-purpose policy, reporting | Not published (checked 2026-11-04) | | TollBit | 6 | charging | verifies agents, per-purpose policy | Free for publishers to start | | Kasada | 5 | nothing | per-purpose policy | No public price; sales-led | | Fastly | 4 | charging, commerce agents | verifies agents, per-purpose policy, reporting | No public price; sales-led | | Arcjet | 3 | nothing | verifies agents, per-purpose policy, reporting | From USD 25/month plus USD 5 per 1M web requests | | Auth0 for AI Agents | 3 | commerce agents | verifies agents, per-purpose policy, reporting | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) | | Imperva | 3 | nothing | verifies agents, per-purpose policy | No public price; sales-led | | Stytch IsAgent | 2 | nothing | verifies agents, per-purpose policy, reporting | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 | | Netlify | 1 | nothing | verifies agents, per-purpose policy, reporting | Platform plans from USD 0; extension price not stated | ## Where Vercel is not the fit - No charging or pay-per-crawl for AI crawlers (BotID and firewall docs read 2026-11-04). - BotID Deep Analysis is not on the Hobby plan; it is listed for Pro and Enterprise only (BotID pricing table). - BotID does not inspect passive page views; it only runs where the app calls checkBotId() (BotID docs). ## Sources - [vercel.com: botid](https://vercel.com/docs/botid) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [docs.netlify.com: block ai crawlers](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers) read 2026-11-04 - [docs.aws.amazon.com: waf bot control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) read 2026-11-04 - [akamai.com: akamai unveils agentic security framework to power trusted ai driven interactions and commerce](https://www.akamai.com/newsroom/press-release/akamai-unveils-agentic-security-framework-to-power-trusted-ai-driven-interactions-and-commerce) read 2026-11-04 - [aws.amazon.com: pricing](https://aws.amazon.com/waf/pricing/) read 2026-11-04 - [cloudflare.com: network cdn.md](https://www.cloudflare.com/plans/network-cdn.md) read 2026-11-04 - [datadome.co: pricing](https://datadome.co/pricing/) read 2026-11-04 - [humansecurity.com: package plans](https://www.humansecurity.com/platform/package-plans) read 2026-11-04 - [skyfire.xyz: skyfire.xyz](https://skyfire.xyz/) read 2026-11-04 - [tollbit.com: tollbit.com](https://tollbit.com/) read 2026-11-04 - [kasada.io: kasada.io](https://www.kasada.io/) read 2026-11-04 Last reviewed 2026-11-04 --- --- title: "Verisoul: bot management vendor profile | Agent Ready Crawlers" description: "Verisoul: bot management vendor profile covering Agent & Bot Detection and Agent Atlas. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/verisoul site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Verisoul: bot management vendor profile | Agent Ready Crawlers > Verisoul: bot management vendor profile covering Agent & Bot Detection and Agent Atlas. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/verisoul Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Bot management vendor # Verisoul In short Verisoul classifies agents by how they announce themselves: of 490 in its public Agent Atlas, 430 sign their requests, 57 declare a user agent and 3 can only be inferred, with Grok Bot a noted case of an agent that stopped signing. Its detector reads clicks, mouse movement and device data without CAPTCHAs, and a 30-day trial is offered. Type [Bot management](https://agentreadycrawlers.com/vendors/by-type/bot-management/) Homepage [verisoul.ai](https://www.verisoul.ai) In one line Fake account and fraud detection platform whose Agent & Bot Detection names AI agents by signature, self-declared user agent or inference, and whose public Agent Atlas documents how each agent shows up on a site. Last read 2026-11-05 ## What Verisoul does for this layer ### From the vendor's own pages Verisoul's bot detection blocks bots without CAPTCHAs by analysing clicks, mouse moves, touches and keys together with device and browser data. ([verisoul.ai](https://www.verisoul.ai/products/bot-detection)) Verisoul offers a 30-day free trial of Agent & Bot Detection from its product page. ([verisoul.ai](https://www.verisoul.ai/products/bot-detection)) ### Claims the vendor makes about itself Verisoul's Agent Atlas, updated November 2026, lists 490 agents: 430 identified by signature, 57 self-declared and 3 inferred. ([verisoul.ai](https://www.verisoul.ai/agents)) Agent Atlas groups those agents into 102 AI agents, 13 AI assistants, 26 AI crawlers, 235 crawlers, 17 headless automation tools and 97 others such as link previews and uptime monitors. ([verisoul.ai](https://www.verisoul.ai/agents)) Verisoul's Agent Atlas says Grok Bot stopped signing requests in mid-October 2026 and uses the user's own IP, so only inference identifies it. ([verisoul.ai](https://www.verisoul.ai/agents)) Verisoul says OpenAI's always-on Dots browse from a cloud browser that OpenAI signs, while Claude Duet declares itself in its user agent. ([verisoul.ai](https://www.verisoul.ai/agents)) Verisoul announces a USD 9 million Series A on its homepage. ([verisoul.ai](https://www.verisoul.ai/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Agent & Bot Detection | CAPTCHA-free bot and agent detection inside sign-up and login flows. | sales-led, 30-day trial Pricing URL redirects to a demo page. | Not published | 2026-11-05 | | Agent Atlas | Public directory of 490 agents with detection guides for 32 of them. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | Agent Atlas counts 430 agents identified by signature. (verisoul.ai) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Verisoul does not do - Verisoul's product pages describe account-level fraud and bot decisions inside the application and name no CDN or edge integration (read 2026-11-05). - Verisoul publishes no price. ## Related pages on this site - [All bot management vendors](https://agentreadycrawlers.com/vendors/by-type/bot-management/) 23 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Grok Bot (agent)](https://agentreadycrawlers.com/crawlers/grok-bot/) no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) token Claude-Agent - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) no token - [Comet (browser assistant)](https://agentreadycrawlers.com/crawlers/perplexity-comet/) no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) no token - [Browser Use (library and cloud)](https://agentreadycrawlers.com/crawlers/browser-use/) no token ## Sources - [verisoul.ai](https://www.verisoul.ai) read 2026-11-05 - [Agent Atlas: every AI agent and bot Verisoul identifies](https://www.verisoul.ai/agents) read 2026-11-05 - [Verisoul | Agent & bot detection without CAPTCHAs](https://www.verisoul.ai/products/bot-detection) read 2026-11-05 - [Fake Account & Fraud Detection Platform | Verisoul](https://www.verisoul.ai/) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Visa: agent verification provider profile | Agent Ready Crawlers" description: "Visa: agent verification provider profile covering Trusted Agent Protocol, Visa Intelligent Commerce and Agentic Directory. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/visa site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Visa: agent verification provider profile | Agent Ready Crawlers > Visa: agent verification provider profile covering Trusted Agent Protocol, Visa Intelligent Commerce and Agentic Directory. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/visa Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Visa In short Visa's Trusted Agent Protocol has agents sign requests under the agent-browser-auth or agent-payer-auth tag, with Visa hosting the public keys, so a merchant or its site protection provider can verify them before letting an agent browse or pay. Akamai announced support on 18 December 2025, and no public list of registered agents is linked. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol) In one line Card network whose Trusted Agent Protocol lets a merchant or its CDN verify, from RFC 9421 signatures tagged agent-browser-auth or agent-payer-auth, that a visiting agent is approved under Visa Intelligent Commerce and acting for a known consumer. Last read 2026-11-05 ## What Visa does for this layer ### From the vendor's own pages Visa says it and more than 10 partners introduced Trusted Agent Protocol in October 2025 as an open framework on existing web infrastructure to help merchants tell legitimate AI agents from malicious bots. ([usa.visa.com](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21961.html), dated 2025-12-18) Visa's merchant specification builds the agent recognition signature on HTTP Message Signatures "defined by RCF 9421 \[and aligned with web-bot-auth\]", with the Signature-Input tag set to agent-browser-auth for browsing or agent-payer-auth for payment. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Visa's specification tells merchants to block a signed request when its created and expires timestamps are more than 8 minutes apart, and to block a nonce already seen in the last 8 minutes. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Trusted Agent Protocol carries three linked signatures: the agent recognition signature in the header, a signed consumer recognition object with a Visa ID Token in the body, and a signed payment container for checkout. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) For Visa's implementation, Visa hosts the public keys at a well-known URL with a GET /keys operation that returns a JSON Web Key for a given keyID, so merchants do not fetch keys from each agent's own domain. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Visa's specification says a signature that verifies against a Visa-associated key implies the agent's participation in and compliance with Visa Intelligent Commerce program requirements. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Visa's specification names Site Protection Providers, meaning CDNs, trust management systems and proxies, as an intended audience next to merchants, and lets merchants that have not adopted the protocol ignore the signatures. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Visa's specification foresees merchants using the protocol to request payment from an agent for access to certain information, giving customer product reviews as the example. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Visa's GitHub sample for Trusted Agent Protocol ships five components: an agent registry for public keys, a React merchant frontend, a FastAPI merchant backend, a Node.js CDN proxy that verifies RFC 9421 signatures, and a Streamlit TAP agent. ([github.com](https://github.com/visa/trusted-agent-protocol)) Reading Visa's Trusted Agent Protocol specification requires agreeing to the Visa Trusted Agent Protocol Product Terms, which the specification page states before its introduction. ([developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications)) Visa announced on 18 Dec 2025 that Akamai was the latest company to support Trusted Agent Protocol, integrating it with Akamai's edge behavioural intelligence and bot and abuse protection. ([usa.visa.com](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21961.html), dated 2025-12-18) Visa, BILL and Billtrust announced on 20 Oct 2026 a live pilot in which a BILL Payment Agent registered through Trusted Agent Protocol and Visa's Agentic Directory pays supplier invoices on Billtrust portals, with agent authentication standing in for CAPTCHA and bot blocking. ([usa.visa.com](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22856.html), dated 2026-10-20) ### Claims the vendor makes about itself Visa reported on 18 Dec 2025 more than 100 partners, over 30 building in the Visa Intelligent Commerce sandbox and over 20 agents and agent enablers integrating directly, with hundreds of agent-initiated transactions completed. ([usa.visa.com](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21961.html), dated 2025-12-18) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Trusted Agent Protocol | Merchant specification and sample code for verifying signed agent requests, consumer recognition and payment containers. | No price recorded | | Visa Intelligent Commerce | Visa's agentic commerce programme; agents onboard to it and their keys are hosted by Visa. | No price recorded | | Agentic Directory | Registration Visa names for trusted agents, used by the BILL Payment Agent in the October 2026 pilot. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | Supports | The agent recognition signature is RFC 9421 "aligned with web-bot-auth"; keys come from a Visa-hosted key store rather than each agent's directory. (developer.visa.com) | | Published IP list | No statement | The specification lists IP allowlisting as one possible way to identify agents and chooses signed credentials instead; Visa publishes no agent IP ranges (checked 2026-11-05). | | Visa Trusted Agent Protocol | Supports | Visa publishes and maintains the Trusted Agent Protocol specification and sample implementation. (developer.visa.com) | | Mastercard Agent Pay | No statement | The specification refers to a shared approach across Payment Schemes without naming Mastercard Agent Pay (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Visa does not do - Visa does not block or admit anything itself; the merchant or its site protection provider verifies the signature and decides. - Visa's Trusted Agent Protocol pages read on 2026-11-05 link no public list of the agents registered in the Agentic Directory. - Trusted Agent Protocol says nothing about crawlers that do not carry commerce intent; training and search bots stay with robots.txt and bot management. ## Reading notes Visa's investor-site release on the October 2025 launch is a JavaScript page that a plain fetch cannot read, so the launch is cited from Visa's release of 18 December 2025. The specification spells RFC 9421 as RCF 9421, and quotes here keep Visa's spelling. ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/) bot management - [Akamai (Bot & Agent Control)](https://agentreadycrawlers.com/vendors/akamai/) bot management - [Skyfire (KYA, KYAPay, Agentic Wallet)](https://agentreadycrawlers.com/vendors/skyfire/) licensing intermediary ## Sources - [Getting Started, Trusted Agent Protocol (Visa Developer)](https://developer.visa.com/docs/trusted-agent-protocol) read 2026-11-05 - [Visa and Partners Complete Secure AI Transactions, Setting the Stage for Mainstream Adoption in 2026](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21961.html) read 2026-11-05 - [Specifications, Trusted Agent Protocol (Visa Developer)](https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications) read 2026-11-05 - [visa/trusted-agent-protocol on GitHub](https://github.com/visa/trusted-agent-protocol) read 2026-11-05 - [Visa, BILL and Billtrust Launch First Live Pilot for Trusted AI Agent Payments](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22856.html) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Vouched: agent verification provider profile | Agent Ready Crawlers" description: "Vouched: agent verification provider profile covering Know Your Agent: Detect, Know Your Agent: Enforce and Govern, KnowThat.ai and 1 more. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/vouched site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Vouched: agent verification provider profile | Agent Ready Crawlers > Vouched: agent verification provider profile covering Know Your Agent: Detect, Know Your Agent: Enforce and Govern, KnowThat.ai and 1 more. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/vouched Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # Vouched In short Vouched's agent identity work comes as an open standard, KYA-OS, handed to the Decentralized Identity Foundation, and its Enforce product turns plain-language rules into Cedar policies backed by signed delegations and an audit trail. KnowThat.ai, its agent registry, held 8,400 behaviour reports by 5 November 2026, but no count of agents tracked. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [vouched.id](https://www.vouched.id/know-your-agent) In one line Identity verification company whose Know Your Agent suite detects AI agents on a site by behaviour, enforces allow, challenge, block or redirect rules, and ties each agent action to a verified person through signed delegations. Last read 2026-11-05 ## What Vouched does for this layer ### From the vendor's own pages Vouched launched its Know Your Agent product suite on 24 Feb 2026, according to the press list on its product page. ([vouched.id](https://www.vouched.id/know-your-agent), dated 2026-02-24) Vouched's Detect product names each AI agent as it arrives using behavioural detection rather than the identity the agent claims, and installs with one line on the site. ([vouched.id](https://www.vouched.id/know-your-agent)) Vouched's Enforce product applies allow, challenge, block or redirect decisions from plain-language rules compiled to Cedar, either at the edge or inside the application. ([vouched.id](https://www.vouched.id/know-your-agent)) Vouched issues agent delegations as Ed25519-signed W3C VC-JWTs under VC Data Model 1.1, with a cryptographic proof on every request and an audit trail. ([vouched.id](https://www.vouched.id/know-your-agent)) Vouched authored KYA-OS, an open standard for agent identity and authorization, and donated it to the Decentralized Identity Foundation, with a joint launch announced in July 2026. ([vouched.id](https://www.vouched.id/know-your-agent)) Vouched runs KnowThat.ai, a public agent registry where operators check an agent's trust score with one API call and report its behaviour, which the site calls "A credit bureau for AI agents". ([knowthat.ai](https://knowthat.ai/)) Vouched's pricing page states that Know Your Agent detection is free, while its human identity verification starts at USD 300 a month plus USD 0.75 per transaction. ([vouched.id](https://www.vouched.id/pricing)) Vouched's IdentiClaw gives an agent its own identity with scoped permissions and a kill switch, so the agent acts for a person rather than logging in as that person. ([vouched.id](https://www.vouched.id/know-your-agent)) ### Claims the vendor makes about itself Vouched lists Claude, ChatGPT, Gemini, Perplexity, Microsoft Copilot, Meta Muse, DeepSeek and Amazon Q among the agents it detects and claims more than 1,000 detection signals. ([vouched.id](https://www.vouched.id/know-your-agent)) Vouched's Know Your Agent page claims "70,000+" agents registered on KnowThat.ai. ([vouched.id](https://www.vouched.id/know-your-agent)) Vouched reports 2.4 times growth in AI agent traffic over 30 days across its Detect customers, citing its own June 2026 data. ([vouched.id](https://www.vouched.id/know-your-agent), dated 2026-06-30) ### What our crawl saw KnowThat.ai showed 8,400+ behaviour reports from 214 reporting operators when read on 2026-11-05, with its agents-tracked counter blank. ([knowthat.ai](https://knowthat.ai/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Know Your Agent: Detect | Behavioural detection and naming of AI agents on a site. | free "Detection is free"; enforce and govern tiers by quote | USD 0 | 2026-11-05 | | Know Your Agent: Enforce and Govern | Cedar policy enforcement at edge or app, signed delegations and audit trail. | No price recorded | | KnowThat.ai | Public reputation registry for AI agents with lookup and reporting APIs. | No price recorded | | Vouched Verify | Document and biometric identity verification for people. | monthly fee plus per transaction plus USD 0.75 per transaction, monthly billing | USD 300 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | The Know Your Agent page and KnowThat.ai describe behavioural detection and VC-JWT delegations; neither mentions Web Bot Auth (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol mention on the pages read (checked 2026-11-05). | | Mastercard Agent Pay | No statement | No Agent Pay mention on the pages read (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Vouched does not do - Vouched does not rely on an agent's own signature to name it; Detect classifies by behaviour, so an agent that signs with Web Bot Auth gets no documented fast path. - Vouched publishes no price for the Enforce and Govern tiers of Know Your Agent (pricing page read 2026-11-05). ## Reading notes The Ranked #1 and award claims on Vouched's product page are left out as marketing. ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Muse (personal agent)](https://agentreadycrawlers.com/crawlers/meta-muse/) no token - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) no token ## Sources - [Know Your Agent: AI Agent Identity and Verification (Vouched)](https://www.vouched.id/know-your-agent) read 2026-11-05 - [KnowThat.ai, The Trusted Agent Registry](https://knowthat.ai/) read 2026-11-05 - [Vouched pricing](https://www.vouched.id/pricing) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Webz.io: crawler operator profile | Agent Ready Crawlers" description: "Webz.io: crawler operator profile covering Open Web APIs (News, Blogs, Forums, Reviews, Search) and Dark Web API and Breach API. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/webz-io site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Webz.io: crawler operator profile | Agent Ready Crawlers > Webz.io: crawler operator profile covering Open Web APIs (News, Blogs, Forums, Reviews, Search) and Dark Web API and Breach API. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/webz-io Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Webz.io In short Webz.io, headquartered in Israel, resells crawled web data through APIs for news, blogs, forums and reviews, with separate dark web and breach feeds quoted on request. Webzio-extended exists only to label each collected item as usable or not for AI training, and the older omgili bot was succeeded in July 2024. Custom tiers reach back ten years of archive. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Based in Israel Homepage [webz.io](https://webz.io) In one line Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training. Last read 2026-11-05 ## What Webz.io does for this layer ### From the vendor's own pages Webz.io introduced two crawlers on 2024-07-17, Webzio and Webzio-extended, as the successors to its Omgilibot, and its webz.io/bot.html address now redirects to that announcement. ([webz.io](https://webz.io/blog/company/from-omgilibot-to-the-webzbot-duo-a-powerful-leap-for-ethical-and-comprehensive-data-collection/), dated 2024-07-17) Webz.io says the Webzio crawler collects and structures content, while Webzio-extended tags the collected data as usable or not usable for AI/ML training. ([webz.io](https://webz.io/blog/company/from-omgilibot-to-the-webzbot-duo-a-powerful-leap-for-ethical-and-comprehensive-data-collection/), dated 2024-07-17) Webz.io's 2017-12-28 post gives the Omgili bot's user agent as "omgili/0.5 +https://omgili.com" and says the bot was first built for the now discontinued Omgili search engine. ([webz.io](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/), dated 2017-12-28) The Omgili bot post lists news sites, blogs, reviews, ecommerce and message boards as the bot's focus and says site owners can refuse it through robots.txt or by telling Webz.io directly. ([webz.io](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/), dated 2017-12-28) Webz.io's pricing page lists a Free plan at USD 0 a month with USD 5 of usage credits every month, a pay-as-you-go Scale plan, and custom High Volume pricing with up to 10 years of archive data. ([webz.io](https://webz.io/pricing/)) Webz.io's About page states that the company is headquartered in Israel. ([webz.io](https://webz.io/about-us/)) ### Claims the vendor makes about itself Webz.io says the two newer crawlers adhere to robots.txt exclusions and that it only provides data tagged as permitted for AI purposes, a rule it says is reflected in its Terms of Service. ([webz.io](https://webz.io/blog/company/from-omgilibot-to-the-webzbot-duo-a-powerful-leap-for-ethical-and-comprehensive-data-collection/), dated 2024-07-17) Webz.io sells News, Blogs, Online Forums, Reviews, Dark Web and Breach APIs, and says its News API covers 350K+ sources in 170+ languages. ([webz.io](https://webz.io/)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Open Web APIs (News, Blogs, Forums, Reviews, Search) | Structured feeds built from Webz.io's crawl of the open web. | pay as you go Free plan with USD 5 monthly credits; High Volume is custom. | USD 0 | 2026-11-05 | | Dark Web API and Breach API | Data from deep and dark web sources; contact for pricing. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Webz.io says Webzio and Webzio-extended adhere to robots.txt exclusions. (webz.io) | | Web Bot Auth | No statement | No key directory at webz.io (HTTP 404, 2026-11-05). | | Published IP list | No statement | No IP list found on the pages read (2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://webz.io/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Webz.io does not do - Webz.io publishes no IP list, reverse DNS pattern or signature keys for Webzio, Webzio-extended or omgili (checked 2026-11-05). - Webz.io's 2024 announcement names the two new tokens but gives no full user agent strings for them. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [omgili (Omgilibot)](https://agentreadycrawlers.com/crawlers/omgili/) token omgili ## Sources - [webz.io](https://webz.io) read 2026-11-05 - [From Omgilibot to the Webzio Duo | Webz.io](https://webz.io/blog/company/from-omgilibot-to-the-webzbot-duo-a-powerful-leap-for-ethical-and-comprehensive-data-collection/) read 2026-11-05 - [What is the Omgili Bot, and why is it Crawling Your Website?](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/) read 2026-11-05 - [Webz.io | News, Open Web & Dark Web APIs for AI](https://webz.io/) read 2026-11-05 - [Plans & Pricing | Webz](https://webz.io/pricing/) read 2026-11-05 - [About Us | Webz](https://webz.io/about-us/) read 2026-11-05 - [Webz.io key directory probe](https://webz.io/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "World: agent verification provider profile | Agent Ready Crawlers" description: "World: agent verification provider profile covering AgentKit (beta) and World ID. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/world site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # World: agent verification provider profile | Agent Ready Crawlers > World: agent verification provider profile covering AgentKit (beta) and World ID. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/world Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Agent verification provider # World In short AgentKit is World's beta toolkit, built with Coinbase, for telling a human-backed agent from a script at an x402 endpoint. An agent's wallet address is recorded on World Chain through a command-line tool and a World App check, and the client tries that proof before payment. Integrations are listed for five platforms, Browserbase and Okta among them. Type [Verification provider](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) Homepage [world.org](https://world.org) In one line Proof-of-human network whose AgentKit lets a person with a verified World ID register an agent's wallet, so x402-protected sites can tell a human-backed agent from a script and limit access per unique person. Last read 2026-11-05 ## What World does for this layer ### From the vendor's own pages World announced AgentKit beta with Coinbase on 17 Mar 2026 as a toolkit that lets verified humans delegate their World ID to AI agents. ([world.org](https://world.org/blog/announcements/now-available-agentkit-proof-of-human-for-the-agentic-web), dated 2026-03-17) World's docs say AgentKit Beta extends x402 so that websites can tell human-backed agents from bots and scripts and block scalpers and spam. ([docs.world.org](https://docs.world.org/agents/agent-kit/integrate)) AgentKit registers an agent's wallet address in AgentBook on World Chain through the @worldcoin/agentkit-cli tool and a World App verification flow, then resolves the wallet to an anonymous human identifier at request time. ([docs.world.org](https://docs.world.org/agents/agent-kit/integrate)) World's AgentKit quickstart accepts payments on World Chain and Base, ships a free-trial mode of 3 uses, and uses Hono with @x402/hono as the reference server. ([docs.world.org](https://docs.world.org/agents/agent-kit/integrate)) AgentKit's agentkit.fetch client tries the AgentKit proof first and keeps the normal x402 payment as the fallback. ([docs.world.org](https://docs.world.org/agents/agent-kit/integrate)) World argues that micropayments alone cannot stop Sybil behaviour because one person can run thousands of paying agents, and offers proof of unique human as the missing signal. ([world.org](https://world.org/blog/announcements/now-available-agentkit-proof-of-human-for-the-agentic-web), dated 2026-03-17) An update on World's AgentKit announcement says AgentKit added agent delegation features and integrations with Browserbase, Exa, Okta, Shopify and Vercel. ([world.org](https://world.org/blog/announcements/now-available-agentkit-proof-of-human-for-the-agentic-web)) ### Claims the vendor makes about itself World claims nearly 18 million verified humans across more than 160 countries. ([world.org](https://world.org/blog/announcements/now-available-agentkit-proof-of-human-for-the-agentic-web), dated 2026-03-17) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | AgentKit (beta) | SDK and CLI that bind an agent wallet to an anonymous World ID holder and plug into x402 servers. | developer preview no price stated; x402 payments are set by each site | USD 0 | 2026-11-05 | | World ID | Anonymous proof of unique human; World's top verification level uses an Orb. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | Web Bot Auth | No statement | AgentKit proves the human behind a wallet through AgentBook and x402; the docs do not use Web Bot Auth signatures (checked 2026-11-05). | | Visa Trusted Agent Protocol | No statement | No Trusted Agent Protocol mention on the announcement or docs (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | ## What World does not do - World's AgentKit does not identify which company operates an agent; it proves only that a unique, anonymous human registered it. - AgentKit protects x402 endpoints; a site that does not run an x402 server gets no AgentKit check. ## Reading notes The company presents itself as World, formerly Worldcoin, built by Tools for Humanity; the pages read use the name World only. ## Related pages on this site - [All verification providers](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) 11 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Browserbase cloud browsers](https://agentreadycrawlers.com/crawlers/browserbase/) no token - [ExaSearchBot](https://agentreadycrawlers.com/crawlers/exasearchbot/) token ExaSearchBot ## Sources - [world.org](https://world.org) read 2026-11-05 - [Now available: AgentKit, Proof of Human for the agentic web](https://world.org/blog/announcements/now-available-agentkit-proof-of-human-for-the-agentic-web) read 2026-11-05 - [World Developer Docs: AgentKit integrate](https://docs.world.org/agents/agent-kit/integrate) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "xAI: crawler operator profile | Agent Ready Crawlers" description: "xAI: crawler operator profile covering Grok Bot and Web Search tool (xAI API). Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/xai site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # xAI: crawler operator profile | Agent Ready Crawlers > xAI: crawler operator profile covering Grok Bot and Web Search tool (xAI API). Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/xai Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # xAI In short xAI gives Grok Bot an always-on cloud computer of its own, shared by an account's other Bots, and lets it hand control back to the user at a CAPTCHA or a passkey. A site learns little about it: neither x.ai nor grok.com serves a key directory, and a third party saw signing end in mid-October. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [x.ai](https://x.ai) In one line xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic. Last read 2026-11-05 ## What xAI does for this layer ### From the vendor's own pages xAI's docs say each Grok Bot works on a persistent cloud computer with a browser, filesystem and terminal, and keeps working while the user's laptop is closed. ([docs.x.ai](https://docs.x.ai/grok-bot/overview)) xAI includes Grok Bot with the Premium, Super and Ultra tiers of The X Subscription and with paid individual plans on legacy Cursor plans. ([docs.x.ai](https://docs.x.ai/grok-bot/overview)) xAI says all Bots on one account share a single cloud computer, including browser cookies, signed-in sessions and files. ([docs.x.ai](https://docs.x.ai/grok-bot/computer-and-apps)) Grok Bot hands the computer back to the user for steps it cannot finish, which xAI lists as single sign-on, passkeys, approvals on another device and "A CAPTCHA it couldn't get past". ([docs.x.ai](https://docs.x.ai/grok-bot/computer-and-apps)) Grok Bot can pay at checkout with a card saved in its Vault, a card entered in a Secure Form, or a connected wallet such as Link or X Money. ([docs.x.ai](https://docs.x.ai/grok-bot/computer-and-apps)) xAI's API web\_search tool lets developers limit browsing with allowed\_domains or excluded\_domains, up to five domains each. ([docs.x.ai](https://docs.x.ai/developers/tools/web-search)) ### Reported by a third party Verisoul reported on 2026-11-03 that Grok Bot stopped signing requests with Web Bot Auth in mid-October 2026 and routes its cloud browser traffic through a reverse proxy on the user's own device, so sites see the user's IP. ([verisoul.ai](https://www.verisoul.ai/agents/grok-bot), dated 2026-11-03) ### What our crawl saw The Web Bot Auth key directory path returned HTTP 404 on both x.ai and grok.com on 2026-11-05. ([x.ai](https://x.ai/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Grok Bot | Always-on agent product; each Bot browses and acts from a persistent cloud computer. | No price recorded | | Web Search tool (xAI API) | Server-side search and browsing tool for API callers, with domain allow and exclude filters. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | No statement | xAI's docs index lists no crawler or robots.txt page (read 2026-11-05). | | Web Bot Auth | No statement | No key directory on x.ai or grok.com (observed 404, 2026-11-05). | | Published IP list | No statement | No IP list in xAI's docs (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 on x.ai and grok.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | ## What xAI does not do - xAI publishes no robots.txt token for any crawler or fetcher in its docs index. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Grok Bot (agent)](https://agentreadycrawlers.com/crawlers/grok-bot/) no token ## Sources - [x.ai](https://x.ai) read 2026-11-05 - [Grok Bot overview (xAI docs)](https://docs.x.ai/grok-bot/overview) read 2026-11-05 - [Grok Bot: Use the computer and apps (xAI docs)](https://docs.x.ai/grok-bot/computer-and-apps) read 2026-11-05 - [xAI API Tools: Web Search](https://docs.x.ai/developers/tools/web-search) read 2026-11-05 - [How to detect Grok Bot (Verisoul Agent Atlas)](https://www.verisoul.ai/agents/grok-bot) read 2026-11-05 - [x.ai Web Bot Auth key directory path (GET probe)](https://x.ai/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "You.com: crawler operator profile | Agent Ready Crawlers" description: "You.com: crawler operator profile covering YouBot. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/you-com site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # You.com: crawler operator profile | Agent Ready Crawlers > You.com: crawler operator profile covering YouBot. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/you-com Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # You.com In short You.com recommends checking YouBot by signature before reverse DNS or address range, and keeps its Ed25519 key at you.com rather than on a separate host. The bot slows on HTTP 429 and pauses on 503; the 68.67.112.0/24 range is given only in prose, so it cannot be loaded from a file. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [you.com](https://you.com) In one line You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24. Last read 2026-11-05 ## What You.com does for this layer ### From the vendor's own pages You.com says YouBot is the crawler that powers the You.com search engine by discovering and indexing web pages. ([you.com](https://you.com/docs/youbot.md)) YouBot's user agent carries YouBot/1.0, a link to docs.you.com/youbot and an env:prod environment marker inside a Chrome-style string. ([you.com](https://you.com/docs/youbot.md)) You.com says YouBot signs requests with Web Bot Auth and publishes Ed25519 public keys in JWK format at you.com/.well-known/http-message-signatures-directory. ([you.com](https://you.com/docs/youbot.md)) You.com names cryptographic signatures as the recommended check, ahead of reverse DNS and IP range validation. ([you.com](https://you.com/docs/youbot.md)) You.com says genuine YouBot requests originate from 68.67.112.0/24. ([you.com](https://you.com/docs/youbot.md)) You.com says YouBot honours robots.txt including crawl-delay and caches robots.txt for 30 minutes. ([you.com](https://you.com/docs/youbot.md)) You.com says YouBot reduces its crawl rate on HTTP 429 and pauses crawling on HTTP 503. ([you.com](https://you.com/docs/youbot.md)) ### What our crawl saw you.com served a key directory with one Ed25519 key and the content type application/http-message-signatures-directory+json on 2026-11-05. ([you.com](https://you.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | YouBot | Crawler for the You.com search index, signed with Web Bot Auth. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | YouBot fully respects robots.txt, including user-agent rules and crawl-delay. (you.com) | | Web Bot Auth | Supports | Requests signed; keys at you.com/.well-known/http-message-signatures-directory. (you.com) | | Published IP list | Partial | One /24 given in the docs, not a machine-readable range file. (you.com) | | Reverse DNS check | Supports | youbot-{octets-with-hyphens}.search.you.com. (you.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 200, application/http-message-signatures-directory+json, 1 key | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | ## What You.com does not do - You.com publishes its YouBot range in prose on the docs page, not as a machine-readable IP file. - You.com's crawler page covers YouBot only and documents no separate fetcher for user-triggered requests (read 2026-11-05). ## Reading notes The [YouBot row](https://agentreadycrawlers.com/crawlers/youbot/) in the Crawler Index, read on 4 November 2026, said You.com published no documentation. The page cited here, you.com/docs/youbot, exists and documents the bot. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [YouBot](https://agentreadycrawlers.com/crawlers/youbot/) token YouBot ## Sources - [you.com](https://you.com) read 2026-11-05 - [YouBot: You.com's Web Crawler](https://you.com/docs/youbot.md) read 2026-11-05 - [you.com Web Bot Auth key directory path (GET probe)](https://you.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05 --- --- title: "Verification Ledger: who publishes what | Agent Ready Crawlers" description: "The Verification Ledger: which of 31 operators publish IP lists, reverse DNS patterns or Web Bot Auth key directories, with URL and date read." canonical: https://agentreadycrawlers.com/verification site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Verification" --- # Verification Ledger: who publishes what | Agent Ready Crawlers > The Verification Ledger: which of 31 operators publish IP lists, reverse DNS patterns or Web Bot Auth key directories, with URL and date read. Source: https://agentreadycrawlers.com/verification Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The Verification Ledger In short The Verification Ledger lists what 31 operators publish so a site can confirm a request is theirs: 10 publish IP lists, 5 a reverse DNS pattern, 5 a Web Bot Auth key directory. 17 publish none of the three. Each row links the page read, dated 2026-11-04. A user-agent string proves nothing, because anyone can copy it. The operators below publish the material a site owner needs to check a request: a list of source addresses, a reverse DNS pattern, or the public keys behind a Web Bot Auth signature. "None" means we found nothing on the pages read, not that nothing exists. [Download the ledger](https://agentreadycrawlers.com/data/). | Operator | IP list | Reverse DNS | Key directory | Cloudflare | In plain words | Read | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | Yes | None | Yes | Listed | One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. | platform.openai.com 2026-11-04 | | Anthropic | Yes | None | None | None found | One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. | support.claude.com 2026-11-04 | | Perplexity | Yes | None | None | None found | Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. | docs.perplexity.ai 2026-11-04 | | Google | Yes | Yes | Yes | None found | Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. | developers.google.com 2026-11-04 | | Apple | Yes | Yes | None | None found | IP list and reverse DNS. | support.apple.com 2026-11-04 | | Microsoft | Yes | Yes | None | None found | bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. | bing.com 2026-11-04 | | Amazon | Yes | None | None | None found | Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. | developer.amazon.com 2026-11-04 | | Amazon Web Services | None | None | Yes | Listed | Signs on behalf of customers' agents when enabled. | docs.aws.amazon.com 2026-11-04 | | Meta | None | None | None | None found | User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. | developers.facebook.com 2026-11-04 | | ByteDance | None | None | None | None found | Publishes nothing. | centinelanalytica.com 2026-11-04 | | Common Crawl | Yes | Yes | None | None found | IP list with synctoken 20260811134000 and reverse DNS. | commoncrawl.org 2026-11-04 | | DuckDuckGo | Yes | None | None | None found | Two IP lists. | duckduckgo.com 2026-11-04 | | Mistral AI | Yes | None | None | None found | IP lists for User and Index; none linked for Training. | docs.mistral.ai 2026-11-04 | | Cohere | None | None | None | None found | Publishes nothing. | technologychecker.io 2026-11-04 | | You.com | None | None | None | None found | Publishes nothing we could read; a detection vendor reports signed requests. | verisoul.ai 2026-11-04 | | Diffbot | None | None | None | None found | Publishes nothing for verification; robots.txt can be ignored by customer setting. | docs.diffbot.com 2026-11-04 | | Timpi | None | None | None | None found | Publishes nothing. | timpi.io 2026-11-04 | | Webz.io | None | None | None | None found | Publishes nothing for verification. | webz.io 2026-11-04 | | Hive (ImageSift) | None | None | None | None found | User agent string and robots.txt behaviour only. | imagesift.com 2026-11-04 | | Huawei | None | Yes | None | None found | Reverse DNS only. | webmaster.petalsearch.com 2026-11-04 | | Browserbase | None | None | Yes | Listed | Key directory; signing is a customer feature. | docs.browserbase.com 2026-11-04 | | Apify | None | None | None | None found | Signing option documented; directory location not found. | blog.apify.com 2026-11-04 | | Browser Use | None | None | None | None found | Publishes nothing for site owners (no key directory at browser-use.com, 404, 2026-11-04). | verisoul.ai 2026-11-04 | | Firecrawl | None | None | None | None found | Robots.txt token only; Firecrawl advises against UA-based allowlisting. | docs.firecrawl.dev 2026-11-04 | | Cloudflare | None | None | None | Listed | Signs Browser Rendering traffic. | developers.cloudflare.com 2026-11-04 | | Anchor | None | None | None | Listed | Signing depends on customer configuration. | blog.cloudflare.com 2026-11-04 | | Block | None | None | None | Listed | Open-source agent. | blog.cloudflare.com 2026-11-04 | | xAI | None | None | None | None found | Publishes nothing; Verisoul reports Grok Bot stopped signing in mid-October 2026. | verisoul.ai 2026-11-04 | | Exa | None | None | None | None found | Publishes nothing we could read. | verisoul.ai 2026-11-04 | | Manus | None | None | None | None found | Publishes nothing we could read. | verisoul.ai 2026-11-04 | | Ora | None | None | Yes | None found | Signs every OraBot request with Web Bot Auth; the key directory answered on 2026-11-05. | ora.ai 2026-11-05 | ## The published material, row by row The addresses and patterns as the operators publish them. Rows with several lists are one list per crawler. OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot and ChatGPT agent) IP lists: [openai.com/gptbot.json](https://openai.com/gptbot.json), [openai.com/searchbot.json](https://openai.com/searchbot.json), [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json), [openai.com/adsbot.json](https://openai.com/adsbot.json). Key directory: [chatgpt.com/.well-known/http-message-signatures-directory](https://chatgpt.com/.well-known/http-message-signatures-directory). Cloudflare: GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581. Anthropic (ClaudeBot, Claude-User, Claude-SearchBot, Claude in Chrome and Claude Duet cloud browser) IP lists: [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json). Perplexity (PerplexityBot, Perplexity-User and Comet) IP lists: [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json), [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json). Google (Googlebot, Google-Extended, Google-CloudVertexBot, GoogleOther, Storebot-Google, Google-Agent, Google-GeminiNotebook and Gemini in Chrome) IP lists: [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json), [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json), [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json), [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json), [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json). Reverse DNS: crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com. Key directory: [agent.bot.goog/.well-known/http-message-signatures-directory](https://agent.bot.goog/.well-known/http-message-signatures-directory) (Google-Agent, subset of requests, experimental). Apple (Applebot and Applebot-Extended) IP lists: [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json). Reverse DNS: \*.applebot.apple.com. Microsoft (Bingbot, MicrosoftPreview and Browse with Copilot) IP lists: [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json). Reverse DNS: Verify Bingbot tool and documented methods. Amazon (Amazonbot, Amzn-SearchBot, Amzn-User, Buy for Me and Rufus) IP lists: [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/), [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/), [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/). Amazon Web Services (Amazon Bedrock AgentCore Browser) Key directory: Web Bot Auth signing in preview, opt-in (browserSigning). Cloudflare: listed by AWS as recognised by Cloudflare, Akamai and others. Common Crawl (CCBot) IP lists: [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json). Reverse DNS: \*.crawl.commoncrawl.org (IPv4 only). DuckDuckGo (DuckAssistBot and DuckDuckBot) IP lists: [duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json), [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json). Mistral AI (MistralAI-User, MistralAI-Index and MistralAI-Training) IP lists: [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json), [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json). Huawei (PetalBot) Reverse DNS: petalbot-\*.petalsearch.com. Browserbase (Browserbase cloud browsers and Stagehand) Key directory: [www.browserbase.com/.well-known/http-message-signatures-directory](https://www.browserbase.com/.well-known/http-message-signatures-directory). Cloudflare: Cloudflare signed agent (cohort of 2025-08-28). Cloudflare (Cloudflare Browser Rendering) Cloudflare: Cloudflare signed agent. Anchor (Anchor Browser) Cloudflare: Cloudflare signed agent (cohort of 2025-08-28). Block (Goose) Cloudflare: Cloudflare signed agent (cohort of 2025-08-28). Ora (OraBot) Key directory: [ora.ai/.well-known/http-message-signatures-directory](https://ora.ai/.well-known/http-message-signatures-directory?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=ledger). ## Card-network registries Visa's Trusted Agent Protocol and Mastercard Agent Pay register agents that shop on a merchant's behalf. For 31 operators the ledger found no public operator-level Visa registration (checked 2026-11-04); the two programmes are below. Visa Programme, not a crawler: the registry of TAP agents is not public. reference implementation at github.com/visa/trusted-agent-protocol (agent-registry folder); no public list of registered agents found. [Source](https://github.com/visa/trusted-agent-protocol), read 2026-11-04. Mastercard Programme, not a crawler. Mastercard: developer docs are a JavaScript shell to plain fetch; no public list of registered agents found (checked 2026-11-04). [Source](https://blog.cloudflare.com/secure-agentic-commerce/), read 2026-11-04. See [how to let TAP and Agent Pay agents through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/). ## Added on 2026-11-05 - [**Verified bot programmes compared**](https://agentreadycrawlers.com/verification/programmes-compared/) IP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay: what a site can check and who uses each. ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [docs.aws.amazon.com: browser web bot auth](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 - [centinelanalytica.com: bytespider](https://www.centinelanalytica.com/bots/bytespider) read 2026-11-04 - [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04 - [duckduckgo.com: duckassistbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) read 2026-11-04 - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-04 - [technologychecker.io: ai crawler list](https://technologychecker.io/blog/ai-crawler-list) read 2026-11-04 - [verisoul.ai: youbot](https://www.verisoul.ai/agents/youbot) read 2026-11-04 - [docs.diffbot.com: why is my crawl not crawling and other uncommon crawl problems](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems) read 2026-11-04 - [timpi.io: timpi.io](https://timpi.io/) read 2026-11-04 - [webz.io: bot](https://webz.io/bot.html) read 2026-11-04 - [imagesift.com: about](https://imagesift.com/about) read 2026-11-04 - [webmaster.petalsearch.com: petalbot](https://webmaster.petalsearch.com/site/petalbot) read 2026-11-04 - [docs.browserbase.com: overview](https://docs.browserbase.com/platform/identity/overview) read 2026-11-04 - [blog.apify.com: ai agent infrastructure](https://blog.apify.com/ai-agent-infrastructure/) read 2026-11-04 - [verisoul.ai: browser use](https://www.verisoul.ai/agents/browser-use) read 2026-11-04 - [docs.firecrawl.dev: advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide) read 2026-11-04 - [developers.cloudflare.com: automatic request headers](https://developers.cloudflare.com/browser-rendering/reference/automatic-request-headers/) read 2026-11-04 - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 - [verisoul.ai: xai bots](https://www.verisoul.ai/agents/xai-bots) read 2026-11-04 - [verisoul.ai: exasearchbot](https://www.verisoul.ai/agents/exasearchbot) read 2026-11-04 - [verisoul.ai: manus bot](https://www.verisoul.ai/agents/manus-bot) read 2026-11-04 - [ora.ai](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=ledger) read 2026-11-04 - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [Ledger downloads (this site)](https://agentreadycrawlers.com/data/) read 2026-11-04 Last reviewed 2026-11-05 --- --- title: "Verified bot programmes compared | Agent Ready Crawlers" description: "IP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay compared: what a site checks and which Index rows use each." canonical: https://agentreadycrawlers.com/verification/programmes-compared site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Verification" --- # Verified bot programmes compared | Agent Ready Crawlers > IP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay compared: what a site checks and which Index rows use each. Source: https://agentreadycrawlers.com/verification/programmes-compared Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Verified bot programmes compared In short Of 70 Index rows, 17 publish an IP list, 9 a reverse DNS pattern, 8 sign with Web Bot Auth, and Cloudflare's signed agent list covers 1; 35 publish nothing a site can check. Visa and Mastercard run programmes whose registries are not public. A user-agent string proves nothing, since anyone can send it. The ways to check who sent a request fall into four groups of proof, plus two card-network programmes for agents that buy. This page sets them side by side and counts which of the 70 Index rows use each. ## Index rows by method Rows in the Crawler Index by the verification method the operator publishes, read 2026-11-04; Ora's OraBot, read 2026-11-05, counts as Web Bot Auth. | Method | What it proves | Rows | Which rows | Note | | --- | --- | --- | --- | --- | | Published IP list | The request came from an address on the operator's list. | 17 | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot and Claude-User and 11 more | 10 ledger operators publish one. | | Reverse DNS | The address resolves to the operator's domain and back to the same address. | 9 | Googlebot, Google-CloudVertexBot, GoogleOther, Storebot-Google, Applebot and Bingbot and 3 more | No list to fetch, and it needs a lookup per address. | | Web Bot Auth | The request carries a signature that verifies against a public key the operator publishes. | 8 | ChatGPT agent, Google-Agent, Amazon Bedrock AgentCore Browser, Apify Website Content Crawler, Browserbase cloud browsers and Cloudflare Browser Rendering and 2 more | 4 ledger operators serve a key directory. | | Cloudflare signed agents and verified bots | Cloudflare has confirmed the bot belongs to its operator and lists it for rules. | 1 | Goose | 6 ledger operators show a Cloudflare listing. | | Nothing published | Nothing: a site has the user-agent string and the source address, and no operator material to compare them with. | 35 | Includes the agents that run in a visitor's own browser. | 50 percent of the Index. | ## Programmes for agents that buy Two card-network programmes build on Web Bot Auth for agents that pay. Neither is a crawler, so neither is a row in the Index, and the ledger found no public list of registered agents for either. [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/) Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. In the ledger: reference implementation at github.com/visa/trusted-agent-protocol (agent-registry folder); no public list of registered agents found. [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/) Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs can verify it. In the ledger: developer docs are a JavaScript shell to plain fetch; no public list of registered agents found (checked 2026-11-04). [How to let Visa TAP and Mastercard Agent Pay agents through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/) covers the rules. A site that wants to admit these agents checks the signature, since there is no registry to look up. ## Which proof to ask for Ask for the strongest proof the operator offers. A signature survives a change of address; an IP list and reverse DNS depend on the sender's network. Where an operator offers none, a site is left with the string, and a rule on the string alone admits any copy of it. [How to verify Web Bot Auth signatures](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) and [how to test whether agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) show the checks. ## Sources - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [openai.com: gptbot (JSON)](https://openai.com/gptbot.json) read 2026-11-04 - [Ora's scanner bot page (vendor)](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=comparison) read 2026-11-05 Last reviewed 2026-11-05 ---