--- title: "Data downloads: CSV and JSON | Agent Ready Crawlers" description: "Download the Crawler Index, the Block Rate Report, the Verification Ledger and two crawl slices as CSV and JSON. 13,071 crawl rows." canonical: https://agentreadycrawlers.com/data site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Data" --- # Data downloads: CSV and JSON | Agent Ready Crawlers > Download the Crawler Index, the Block Rate Report, the Verification Ledger and two crawl slices as CSV and JSON. 13,071 crawl rows. Source: https://agentreadycrawlers.com/data Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Data downloads In short Five datasets, each as CSV and some as JSON: the Crawler Index with 70 rows, the Block Rate Report from a crawl of 13,071 domains, the Verification Ledger with 33 operators, and two slices of the same crawl. Every row carries a source URL and the date read. Edition 2026-11-04. ## Crawler Index 70 AI crawlers and agents: operator, user-agent string, robots.txt token, purpose, operator statement, verification, allow and refuse lines, block rate. - [crawler-index-2026-11-04.csv](https://agentreadycrawlers.com/data/crawler-index-2026-11-04.csv) text/csv, about 47 KB - [crawler-index-2026-11-04.json](https://agentreadycrawlers.com/data/crawler-index-2026-11-04.json) application/json, about 91 KB **Cite this.** Agent Ready Crawlers, "Crawler Index", edition 2026-11-04, https://agentreadycrawlers.com/data/. ## Block Rate Report Per-token robots.txt block rates by site list, rank band and Ora category, the 401 and 403 share of home pages and bot-management header prevalence, from 13,071 crawled domains. - [block-rates-2026-11-04.csv](https://agentreadycrawlers.com/data/block-rates-2026-11-04.csv) text/csv, about 62 KB - [block-rates-2026-11-04.json](https://agentreadycrawlers.com/data/block-rates-2026-11-04.json) application/json, about 203 KB - [home-status-2026-11-04.csv](https://agentreadycrawlers.com/data/home-status-2026-11-04.csv) text/csv, about 1 KB - [bot-signals-2026-11-04.csv](https://agentreadycrawlers.com/data/bot-signals-2026-11-04.csv) text/csv, about 1 KB **Cite this.** Agent Ready Crawlers, "Block Rate Report", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/. Crawl of 13,071 rows, read 2026-11-04. ## Verification Ledger 33 operators and programmes: IP lists, reverse DNS, Web Bot Auth key directories, Cloudflare status and card-network registries, each with the URL and the date read. - [verification-ledger-2026-11-04.csv](https://agentreadycrawlers.com/data/verification-ledger-2026-11-04.csv) text/csv, about 14 KB - [verification-ledger-2026-11-04.json](https://agentreadycrawlers.com/data/verification-ledger-2026-11-04.json) application/json, about 22 KB **Cite this.** Agent Ready Crawlers, "Verification Ledger", edition 2026-11-04, https://agentreadycrawlers.com/verification/. ## Names written in robots.txt The most written user-agent names in the parsed robots.txt files of the crawl, top 10,000 and all crawled, with the share of sites that write each. - [named-agents-2026-11-04.csv](https://agentreadycrawlers.com/data/named-agents-2026-11-04.csv) text/csv, about 6 KB **Cite this.** Agent Ready Crawlers, "Names written in robots.txt", edition 2026-11-04, https://agentreadycrawlers.com/crawlers/named-in-robots-txt/. ## Training and search policy mix How parsed robots.txt files combine refusals of the six training tokens and the three AI search tokens, top 10,000 and all crawled. - [policy-mix-2026-11-04.csv](https://agentreadycrawlers.com/data/policy-mix-2026-11-04.csv) text/csv, about 2 KB **Cite this.** Agent Ready Crawlers, "Training and search policy mix", edition 2026-11-04, https://agentreadycrawlers.com/block-rates/training-and-search/. ## Terms of use The data is free to use with the cite-this line. Operator statements are quoted from the operators' own pages and keep their source URL. See [the terms](https://agentreadycrawlers.com/terms/) and [the corrections log](https://agentreadycrawlers.com/corrections/). ## Sources - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 Last reviewed 2026-11-05