Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Corrections and known gaps

Corrections log

No corrections yet. The first edition was published on 2026-11-04. Each correction will appear here with its date, the page and what changed.

To report an error, write to editor@agentreadycrawlers.com with the page and the source that shows the correct fact.

Known gaps in this edition

These are the things we looked for on 2026-11-04 and could not confirm. They are listed so that an absence in the Index is not read as a fact.

  • Cloudflare Radar verified status was confirmed only for GPTBot and ChatGPT agent; other slugs (claudebot, bytespider) returned 404, so the full cross-check needs the Radar API with a token or a browser session.
  • Cloudflare's managed ruleset names for Visa TAP and Mastercard Agent Pay agents are not in its docs (checked 2026-11-04).
  • No operator publishes a public list of Visa TAP or Mastercard Agent Pay registered agents; the ledger says so for every row.
  • Claude Duet's Claude-Agent token, Muse, Comet, Copilot and Gemini auto-browse facts come from detection vendors (Verisoul, Centinel Analytica); Anthropic, Meta, Perplexity, Microsoft and Google publish nothing for site owners on these.
  • vendor_rule_names are filled only where an operator or vendor published them (OpenAI for ChatGPT agent, Perplexity for its bots); Akamai, Netlify and DataDome rule names for other crawlers were not found.

Sources

Last reviewed 2026-11-04