The Verification Ledger
In short
The Verification Ledger lists what 31 operators publish so a site can confirm a request is theirs: 10 publish IP lists, 5 a reverse DNS pattern, 5 a Web Bot Auth key directory. 17 publish none of the three. Each row links the page read, dated 2026-11-04.
A user-agent string proves nothing, because anyone can copy it. The operators below publish the material a site owner needs to check a request: a list of source addresses, a reverse DNS pattern, or the public keys behind a Web Bot Auth signature. "None" means we found nothing on the pages read, not that nothing exists. Download the ledger.
| Operator | In plain words |
|---|---|
| OpenAI | One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. |
| Anthropic | One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. |
| Perplexity | Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. |
| Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. | |
| Apple | IP list and reverse DNS. |
| Microsoft | bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. |
| Amazon | Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. |
| Amazon Web Services | Signs on behalf of customers' agents when enabled. |
| Meta | User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. |
| ByteDance | Publishes nothing. |
| Common Crawl | IP list with synctoken 20260811134000 and reverse DNS. |
| DuckDuckGo | Two IP lists. |
| Mistral AI | IP lists for User and Index; none linked for Training. |
| Cohere | Publishes nothing. |
| You.com | Publishes nothing we could read; a detection vendor reports signed requests. |
| Diffbot | Publishes nothing for verification; robots.txt can be ignored by customer setting. |
| Timpi | Publishes nothing. |
| Webz.io | Publishes nothing for verification. |
| Hive (ImageSift) | User agent string and robots.txt behaviour only. |
| Huawei | Reverse DNS only. |
| Browserbase | Key directory; signing is a customer feature. |
| Apify | Signing option documented; directory location not found. |
| Browser Use | Publishes nothing for site owners (no key directory at browser-use.com, 404, 2026-11-04). |
| Firecrawl | Robots.txt token only; Firecrawl advises against UA-based allowlisting. |
| Cloudflare | Signs Browser Rendering traffic. |
| Anchor | Signing depends on customer configuration. |
| Block | Open-source agent. |
| xAI | Publishes nothing; Verisoul reports Grok Bot stopped signing in mid-October 2026. |
| Exa | Publishes nothing we could read. |
| Manus | Publishes nothing we could read. |
| Ora | Signs every OraBot request with Web Bot Auth; the key directory answered on 2026-11-05. |
The published material, row by row
The addresses and patterns as the operators publish them. Rows with several lists are one list per crawler.
- OpenAI
- IP lists: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json. Key directory: chatgpt.com/.well-known/http-message-signatures-directory. Cloudflare: GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581.
- Anthropic
- IP lists: claude.com/crawling/bots.json.
- Perplexity
- IP lists: www.perplexity.ai/perplexitybot.json, www.perplexity.ai/perplexity-user.json.
- IP lists: developers.google.com/static/search/apis/ipranges/googlebot.json, developers.google.com/static/search/apis/ipranges/special-crawlers.json, developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json, developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json, developers.google.com/static/search/apis/ipranges/user-triggered-agents.json. Reverse DNS: crawl-*.googlebot.com, geo-crawl-*.geo.googlebot.com, rate-limited-proxy-*.google.com, *.gae.googleusercontent.com, google-proxy-*.google.com. Key directory: agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental).
- Apple
- IP lists: search.developer.apple.com/applebot.json. Reverse DNS: *.applebot.apple.com.
- Microsoft
- IP lists: www.bing.com/toolbox/bingbot.json. Reverse DNS: Verify Bingbot tool and documented methods.
- Amazon
- IP lists: developer.amazon.com/amazonbot/ip-addresses/, developer.amazon.com/amazonbot/searchbot-ip-addresses/, developer.amazon.com/amazonbot/live-ip-addresses/.
- Amazon Web Services
- Key directory: Web Bot Auth signing in preview, opt-in (browserSigning). Cloudflare: listed by AWS as recognised by Cloudflare, Akamai and others.
- Common Crawl
- IP lists: index.commoncrawl.org/ccbot.json. Reverse DNS: *.crawl.commoncrawl.org (IPv4 only).
- DuckDuckGo
- IP lists: duckduckgo.com/duckassistbot.json, duckduckgo.com/duckduckbot.json.
- Mistral AI
- IP lists: mistral.ai/mistralai-user-ips.json, mistral.ai/mistralai-index-ips.json.
- Huawei
- Reverse DNS: petalbot-*.petalsearch.com.
- Browserbase
- Key directory: www.browserbase.com/.well-known/http-message-signatures-directory. Cloudflare: Cloudflare signed agent (cohort of 2025-08-28).
- Cloudflare
- Cloudflare: Cloudflare signed agent.
- Anchor
- Cloudflare: Cloudflare signed agent (cohort of 2025-08-28).
- Block
- Cloudflare: Cloudflare signed agent (cohort of 2025-08-28).
- Ora
- Key directory: ora.ai/.well-known/http-message-signatures-directory.
Card-network registries
Visa's Trusted Agent Protocol and Mastercard Agent Pay register agents that shop on a merchant's behalf. For 31 operators the ledger found no public operator-level Visa registration (checked 2026-11-04); the two programmes are below.
- Visa
- Programme, not a crawler: the registry of TAP agents is not public. reference implementation at github.com/visa/trusted-agent-protocol (agent-registry folder); no public list of registered agents found. Source, read 2026-11-04.
- Mastercard
- Programme, not a crawler. Mastercard: developer docs are a JavaScript shell to plain fetch; no public list of registered agents found (checked 2026-11-04). Source, read 2026-11-04.
See how to let TAP and Agent Pay agents through.
Added on 2026-11-05
- Verified bot programmes compared
IP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay: what a site can check and who uses each.
Sources
- Overview of OpenAI Crawlers
- Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity Crawlers
- Verify requests from Google
- About Applebot
- Which Crawlers Does Bing Use?
- Amazon Web Search crawlers
- docs.aws.amazon.com: browser web bot auth
- Meta Web Crawlers
- centinelanalytica.com: bytespider
- CCBot
- duckduckgo.com: duckassistbot
- Mistral crawlers
- technologychecker.io: ai crawler list
- verisoul.ai: youbot
- docs.diffbot.com: why is my crawl not crawling and other uncommon crawl problems
- timpi.io: timpi.io
- webz.io: bot
- imagesift.com: about
- webmaster.petalsearch.com: petalbot
- docs.browserbase.com: overview
- blog.apify.com: ai agent infrastructure
- verisoul.ai: browser use
- docs.firecrawl.dev: advanced scraping guide
- developers.cloudflare.com: automatic request headers
- The age of agents: cryptographically recognizing agent traffic
- verisoul.ai: xai bots
- verisoul.ai: exasearchbot
- verisoul.ai: manus bot
- ora.ai
- Visa Trusted Agent Protocol (GitHub)
- Securing agentic commerce: helping AI Agents transact with Visa and Mastercard
- Ledger downloads (this site)
Last reviewed 2026-11-05