--- title: "Verification Ledger: who publishes what | Agent Ready Crawlers" description: "The Verification Ledger: which of 31 operators publish IP lists, reverse DNS patterns or Web Bot Auth key directories, with URL and date read." canonical: https://agentreadycrawlers.com/verification site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Verification" --- # Verification Ledger: who publishes what | Agent Ready Crawlers > The Verification Ledger: which of 31 operators publish IP lists, reverse DNS patterns or Web Bot Auth key directories, with URL and date read. Source: https://agentreadycrawlers.com/verification Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The Verification Ledger In short The Verification Ledger lists what 31 operators publish so a site can confirm a request is theirs: 10 publish IP lists, 5 a reverse DNS pattern, 5 a Web Bot Auth key directory. 17 publish none of the three. Each row links the page read, dated 2026-11-04. A user-agent string proves nothing, because anyone can copy it. The operators below publish the material a site owner needs to check a request: a list of source addresses, a reverse DNS pattern, or the public keys behind a Web Bot Auth signature. "None" means we found nothing on the pages read, not that nothing exists. [Download the ledger](https://agentreadycrawlers.com/data/). | Operator | IP list | Reverse DNS | Key directory | Cloudflare | In plain words | Read | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | Yes | None | Yes | Listed | One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. | platform.openai.com 2026-11-04 | | Anthropic | Yes | None | None | None found | One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. | support.claude.com 2026-11-04 | | Perplexity | Yes | None | None | None found | Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. | docs.perplexity.ai 2026-11-04 | | Google | Yes | Yes | Yes | None found | Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. | developers.google.com 2026-11-04 | | Apple | Yes | Yes | None | None found | IP list and reverse DNS. | support.apple.com 2026-11-04 | | Microsoft | Yes | Yes | None | None found | bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. | bing.com 2026-11-04 | | Amazon | Yes | None | None | None found | Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. | developer.amazon.com 2026-11-04 | | Amazon Web Services | None | None | Yes | Listed | Signs on behalf of customers' agents when enabled. | docs.aws.amazon.com 2026-11-04 | | Meta | None | None | None | None found | User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. | developers.facebook.com 2026-11-04 | | ByteDance | None | None | None | None found | Publishes nothing. | centinelanalytica.com 2026-11-04 | | Common Crawl | Yes | Yes | None | None found | IP list with synctoken 20260811134000 and reverse DNS. | commoncrawl.org 2026-11-04 | | DuckDuckGo | Yes | None | None | None found | Two IP lists. | duckduckgo.com 2026-11-04 | | Mistral AI | Yes | None | None | None found | IP lists for User and Index; none linked for Training. | docs.mistral.ai 2026-11-04 | | Cohere | None | None | None | None found | Publishes nothing. | technologychecker.io 2026-11-04 | | You.com | None | None | None | None found | Publishes nothing we could read; a detection vendor reports signed requests. | verisoul.ai 2026-11-04 | | Diffbot | None | None | None | None found | Publishes nothing for verification; robots.txt can be ignored by customer setting. | docs.diffbot.com 2026-11-04 | | Timpi | None | None | None | None found | Publishes nothing. | timpi.io 2026-11-04 | | Webz.io | None | None | None | None found | Publishes nothing for verification. | webz.io 2026-11-04 | | Hive (ImageSift) | None | None | None | None found | User agent string and robots.txt behaviour only. | imagesift.com 2026-11-04 | | Huawei | None | Yes | None | None found | Reverse DNS only. | webmaster.petalsearch.com 2026-11-04 | | Browserbase | None | None | Yes | Listed | Key directory; signing is a customer feature. | docs.browserbase.com 2026-11-04 | | Apify | None | None | None | None found | Signing option documented; directory location not found. | blog.apify.com 2026-11-04 | | Browser Use | None | None | None | None found | Publishes nothing for site owners (no key directory at browser-use.com, 404, 2026-11-04). | verisoul.ai 2026-11-04 | | Firecrawl | None | None | None | None found | Robots.txt token only; Firecrawl advises against UA-based allowlisting. | docs.firecrawl.dev 2026-11-04 | | Cloudflare | None | None | None | Listed | Signs Browser Rendering traffic. | developers.cloudflare.com 2026-11-04 | | Anchor | None | None | None | Listed | Signing depends on customer configuration. | blog.cloudflare.com 2026-11-04 | | Block | None | None | None | Listed | Open-source agent. | blog.cloudflare.com 2026-11-04 | | xAI | None | None | None | None found | Publishes nothing; Verisoul reports Grok Bot stopped signing in mid-October 2026. | verisoul.ai 2026-11-04 | | Exa | None | None | None | None found | Publishes nothing we could read. | verisoul.ai 2026-11-04 | | Manus | None | None | None | None found | Publishes nothing we could read. | verisoul.ai 2026-11-04 | | Ora | None | None | Yes | None found | Signs every OraBot request with Web Bot Auth; the key directory answered on 2026-11-05. | ora.ai 2026-11-05 | ## The published material, row by row The addresses and patterns as the operators publish them. Rows with several lists are one list per crawler. OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot and ChatGPT agent) IP lists: [openai.com/gptbot.json](https://openai.com/gptbot.json), [openai.com/searchbot.json](https://openai.com/searchbot.json), [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json), [openai.com/adsbot.json](https://openai.com/adsbot.json). Key directory: [chatgpt.com/.well-known/http-message-signatures-directory](https://chatgpt.com/.well-known/http-message-signatures-directory). Cloudflare: GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581. Anthropic (ClaudeBot, Claude-User, Claude-SearchBot, Claude in Chrome and Claude Duet cloud browser) IP lists: [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json). Perplexity (PerplexityBot, Perplexity-User and Comet) IP lists: [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json), [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json). Google (Googlebot, Google-Extended, Google-CloudVertexBot, GoogleOther, Storebot-Google, Google-Agent, Google-GeminiNotebook and Gemini in Chrome) IP lists: [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json), [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json), [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json), [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json), [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json). Reverse DNS: crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com. Key directory: [agent.bot.goog/.well-known/http-message-signatures-directory](https://agent.bot.goog/.well-known/http-message-signatures-directory) (Google-Agent, subset of requests, experimental). Apple (Applebot and Applebot-Extended) IP lists: [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json). Reverse DNS: \*.applebot.apple.com. Microsoft (Bingbot, MicrosoftPreview and Browse with Copilot) IP lists: [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json). Reverse DNS: Verify Bingbot tool and documented methods. Amazon (Amazonbot, Amzn-SearchBot, Amzn-User, Buy for Me and Rufus) IP lists: [developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/), [developer.amazon.com/amazonbot/searchbot-ip-addresses/](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/), [developer.amazon.com/amazonbot/live-ip-addresses/](https://developer.amazon.com/amazonbot/live-ip-addresses/). Amazon Web Services (Amazon Bedrock AgentCore Browser) Key directory: Web Bot Auth signing in preview, opt-in (browserSigning). Cloudflare: listed by AWS as recognised by Cloudflare, Akamai and others. Common Crawl (CCBot) IP lists: [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json). Reverse DNS: \*.crawl.commoncrawl.org (IPv4 only). DuckDuckGo (DuckAssistBot and DuckDuckBot) IP lists: [duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json), [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json). Mistral AI (MistralAI-User, MistralAI-Index and MistralAI-Training) IP lists: [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json), [mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json). Huawei (PetalBot) Reverse DNS: petalbot-\*.petalsearch.com. Browserbase (Browserbase cloud browsers and Stagehand) Key directory: [www.browserbase.com/.well-known/http-message-signatures-directory](https://www.browserbase.com/.well-known/http-message-signatures-directory). Cloudflare: Cloudflare signed agent (cohort of 2025-08-28). Cloudflare (Cloudflare Browser Rendering) Cloudflare: Cloudflare signed agent. Anchor (Anchor Browser) Cloudflare: Cloudflare signed agent (cohort of 2025-08-28). Block (Goose) Cloudflare: Cloudflare signed agent (cohort of 2025-08-28). Ora (OraBot) Key directory: [ora.ai/.well-known/http-message-signatures-directory](https://ora.ai/.well-known/http-message-signatures-directory?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=ledger). ## Card-network registries Visa's Trusted Agent Protocol and Mastercard Agent Pay register agents that shop on a merchant's behalf. For 31 operators the ledger found no public operator-level Visa registration (checked 2026-11-04); the two programmes are below. Visa Programme, not a crawler: the registry of TAP agents is not public. reference implementation at github.com/visa/trusted-agent-protocol (agent-registry folder); no public list of registered agents found. [Source](https://github.com/visa/trusted-agent-protocol), read 2026-11-04. Mastercard Programme, not a crawler. Mastercard: developer docs are a JavaScript shell to plain fetch; no public list of registered agents found (checked 2026-11-04). [Source](https://blog.cloudflare.com/secure-agentic-commerce/), read 2026-11-04. See [how to let TAP and Agent Pay agents through](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/). ## Added on 2026-11-05 - [**Verified bot programmes compared**](https://agentreadycrawlers.com/verification/programmes-compared/) IP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay: what a site can check and who uses each. ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [docs.aws.amazon.com: browser web bot auth](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 - [centinelanalytica.com: bytespider](https://www.centinelanalytica.com/bots/bytespider) read 2026-11-04 - [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04 - [duckduckgo.com: duckassistbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) read 2026-11-04 - [Mistral crawlers](https://docs.mistral.ai/robots) read 2026-11-04 - [technologychecker.io: ai crawler list](https://technologychecker.io/blog/ai-crawler-list) read 2026-11-04 - [verisoul.ai: youbot](https://www.verisoul.ai/agents/youbot) read 2026-11-04 - [docs.diffbot.com: why is my crawl not crawling and other uncommon crawl problems](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems) read 2026-11-04 - [timpi.io: timpi.io](https://timpi.io/) read 2026-11-04 - [webz.io: bot](https://webz.io/bot.html) read 2026-11-04 - [imagesift.com: about](https://imagesift.com/about) read 2026-11-04 - [webmaster.petalsearch.com: petalbot](https://webmaster.petalsearch.com/site/petalbot) read 2026-11-04 - [docs.browserbase.com: overview](https://docs.browserbase.com/platform/identity/overview) read 2026-11-04 - [blog.apify.com: ai agent infrastructure](https://blog.apify.com/ai-agent-infrastructure/) read 2026-11-04 - [verisoul.ai: browser use](https://www.verisoul.ai/agents/browser-use) read 2026-11-04 - [docs.firecrawl.dev: advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide) read 2026-11-04 - [developers.cloudflare.com: automatic request headers](https://developers.cloudflare.com/browser-rendering/reference/automatic-request-headers/) read 2026-11-04 - [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04 - [verisoul.ai: xai bots](https://www.verisoul.ai/agents/xai-bots) read 2026-11-04 - [verisoul.ai: exasearchbot](https://www.verisoul.ai/agents/exasearchbot) read 2026-11-04 - [verisoul.ai: manus bot](https://www.verisoul.ai/agents/manus-bot) read 2026-11-04 - [ora.ai](https://ora.ai/bot?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=ledger) read 2026-11-04 - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Securing agentic commerce: helping AI Agents transact with Visa and Mastercard](https://blog.cloudflare.com/secure-agentic-commerce/) read 2026-11-04 - [Ledger downloads (this site)](https://agentreadycrawlers.com/data/) read 2026-11-04 Last reviewed 2026-11-05