About Agent Ready Crawlers | Agent Ready Crawlers

Agent Ready Crawlers is a reference publication on the AI crawlers and agents that visit websites, and the products that decide whether they get in. It publishes 70 crawler and agent profiles, 15 vendor profiles, a crawl of 13,071 sites and a ledger of verification methods. Every fact has a source and a date.

Agent Ready Crawlers is written for the people who own robots.txt and the WAF: web platform engineers, security engineers, technical SEOs, publishers' operations leads and the ecommerce engineer who has to let a card network's agents through while keeping scrapers out. It uses exact tokens, exact IP ranges and exact dates, and it names no alarm.

What it publishes

The Crawler Index: 70 rows, one profile page each, with the operator's own statement on robots.txt and the way to verify the agent.
The Block Rate Report: the share of sites that disallow each token and the share of home pages that answer 401 or 403, from a crawl.
The Verification Ledger: which operators publish IP lists, reverse DNS patterns and key directories, and which publish nothing.
15 vendors on a published rubric, with head-to-heads and alternatives.
A robots.txt builder, 14 guides, a glossary, dated news and a FAQ.

How it is made

Rows are read from operators' own pages and dated. The block rates come from one crawl of 13,071 domains on 2026-11-04, parsed the way RFC 9309 describes. Vendors are graded on what they document, with the evidence beside each grade. The method page has the detail, including what our own crawl does and where Ora's scan fits on this layer.

The site is static HTML. Every page has a markdown twin, llms.txt and llms-full.txt list the whole site, and its own robots.txt allows the AI crawlers it writes about.