Data downloads

In short

Five datasets, each as CSV and some as JSON: the Crawler Index with 70 rows, the Block Rate Report from a crawl of 13,071 domains, the Verification Ledger with 33 operators, and two slices of the same crawl. Every row carries a source URL and the date read. Edition 2026-11-04.

Crawler Index

70 AI crawlers and agents: operator, user-agent string, robots.txt token, purpose, operator statement, verification, allow and refuse lines, block rate.

• crawler-index-2026-11-04.csv text/csv, about 47 KB
• crawler-index-2026-11-04.json application/json, about 91 KB

Cite this. Agent Ready Crawlers, "Crawler Index", edition 2026-11-04, https://agentreadycrawlers.com/data/.

Block Rate Report

Per-token robots.txt block rates by site list, rank band and Ora category, the 401 and 403 share of home pages and bot-management header prevalence, from 13,071 crawled domains.

• block-rates-2026-11-04.csv text/csv, about 62 KB
• block-rates-2026-11-04.json application/json, about 203 KB
• home-status-2026-11-04.csv text/csv, about 1 KB
• bot-signals-2026-11-04.csv text/csv, about 1 KB

Cite this. Agent Ready Crawlers, "Block Rate Report", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/. Crawl of 13,071 rows, read 2026-11-04.

Verification Ledger

33 operators and programmes: IP lists, reverse DNS, Web Bot Auth key directories, Cloudflare status and card-network registries, each with the URL and the date read.

• verification-ledger-2026-11-04.csv text/csv, about 14 KB
• verification-ledger-2026-11-04.json application/json, about 22 KB

Cite this. Agent Ready Crawlers, "Verification Ledger", edition 2026-11-04, https://agentreadycrawlers.com/verification/.

Names written in robots.txt

The most written user-agent names in the parsed robots.txt files of the crawl, top 10,000 and all crawled, with the share of sites that write each.

• named-agents-2026-11-04.csv text/csv, about 6 KB

Cite this. Agent Ready Crawlers, "Names written in robots.txt", edition 2026-11-04, https://agentreadycrawlers.com/crawlers/named-in-robots-txt/.

Training and search policy mix

How parsed robots.txt files combine refusals of the six training tokens and the three AI search tokens, top 10,000 and all crawled.

• policy-mix-2026-11-04.csv text/csv, about 2 KB

Cite this. Agent Ready Crawlers, "Training and search policy mix", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/training-and-search/.

Terms of use

The data is free to use with the cite-this line. Operator statements are quoted from the operators' own pages and keep their source URL. See the terms and the corrections log.

Last reviewed 2026-11-05