Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Verified bot programmes compared

In short

Of 70 Index rows, 17 publish an IP list, 9 a reverse DNS pattern, 8 sign with Web Bot Auth, and Cloudflare's signed agent list covers 1; 35 publish nothing a site can check. Visa and Mastercard run programmes whose registries are not public.

A user-agent string proves nothing, since anyone can send it. The ways to check who sent a request fall into four groups of proof, plus two card-network programmes for agents that buy. This page sets them side by side and counts which of the 70 Index rows use each.

Index rows by method

Crawler Index rows by the verification method the operator publishesPublished IP list (JSON) 17, Reverse DNS 9, Web Bot Auth signature 8, Cloudflare signed agent 1, Nothing published 35, out of 70 rows.010203040Published IP list (JSON)17Reverse DNS9Web Bot Auth signature8Cloudflare signed agent1Nothing published35Crawler Index rows by the verification method the operator publishesPublished IP list (JSON) 17, Reverse DNS 9, Web Bot Auth signature 8, Cloudflare signed agent 1, Nothing published 35, out of 70 rows.02040Published IP list (JSON)17Reverse DNS9Web Bot Auth signature8Cloudflare signed agent1Nothing published35
Rows in the Crawler Index by the verification method the operator publishes, read 2026-11-04; Ora's OraBot, read 2026-11-05, counts as Web Bot Auth.
What each method lets a site prove, and which rows use it.
MethodWhat it provesRowsWhich rowsNote
Published IP listThe request came from an address on the operator's list.17GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot and Claude-User and 11 more10 ledger operators publish one.
Reverse DNSThe address resolves to the operator's domain and back to the same address.9Googlebot, Google-CloudVertexBot, GoogleOther, Storebot-Google, Applebot and Bingbot and 3 moreNo list to fetch, and it needs a lookup per address.
Web Bot AuthThe request carries a signature that verifies against a public key the operator publishes.8ChatGPT agent, Google-Agent, Amazon Bedrock AgentCore Browser, Apify Website Content Crawler, Browserbase cloud browsers and Cloudflare Browser Rendering and 2 more4 ledger operators serve a key directory.
Cloudflare signed agents and verified botsCloudflare has confirmed the bot belongs to its operator and lists it for rules.1Goose6 ledger operators show a Cloudflare listing.
Nothing publishedNothing: a site has the user-agent string and the source address, and no operator material to compare them with.35Includes the agents that run in a visitor's own browser.50 percent of the Index.

Programmes for agents that buy

Two card-network programmes build on Web Bot Auth for agents that pay. Neither is a crawler, so neither is a row in the Index, and the ledger found no public list of registered agents for either.

Visa Trusted Agent Protocol
Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. In the ledger: reference implementation at github.com/visa/trusted-agent-protocol (agent-registry folder); no public list of registered agents found.
Mastercard Agent Pay
Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs can verify it. In the ledger: developer docs are a JavaScript shell to plain fetch; no public list of registered agents found (checked 2026-11-04).

How to let Visa TAP and Mastercard Agent Pay agents through covers the rules. A site that wants to admit these agents checks the signature, since there is no registry to look up.

Which proof to ask for

Ask for the strongest proof the operator offers. A signature survives a change of address; an IP list and reverse DNS depend on the sender's network. Where an operator offers none, a site is left with the string, and a rule on the string alone admits any copy of it. How to verify Web Bot Auth signatures and how to test whether agents can reach your site show the checks.

Sources

Last reviewed 2026-11-05