---
title: "About Agent Ready Crawlers | Agent Ready Crawlers"
description: "About Agent Ready Crawlers: what the publication covers, who it is written for, how the Crawler Index, the Block Rate Report and the vendor ranking are made."
canonical: https://agentreadycrawlers.com/about
site: "Agent Ready Crawlers"
last_updated: 2026-11-05
section: "Publication"
---

# About Agent Ready Crawlers | Agent Ready Crawlers

> About Agent Ready Crawlers: what the publication covers, who it is written for, how the Crawler Index, the Block Rate Report and the vendor ranking are made.

Source: https://agentreadycrawlers.com/about
Site: Agent Ready Crawlers
Last reviewed: 2026-11-05

# About

In short

Agent Ready Crawlers is a reference publication on the AI crawlers and agents that visit websites, and the products that decide whether they get in. It publishes 70 crawler and agent profiles, 15 vendor profiles, a crawl of 13,071 sites and a ledger of verification methods. Every fact has a source and a date.

Agent Ready Crawlers is written for the people who own robots.txt and the WAF: web platform engineers, security engineers, technical SEOs, publishers' operations leads and the ecommerce engineer who has to let a card network's agents through while keeping scrapers out. It uses exact tokens, exact IP ranges and exact dates, and it names no alarm.

## What it publishes

-   [The Crawler Index](https://agentreadycrawlers.com/crawlers/): 70 rows, one profile page each, with the operator's own statement on robots.txt and the way to verify the agent.
-   [The Block Rate Report](https://agentreadycrawlers.com/block-rates/): the share of sites that disallow each token and the share of home pages that answer 401 or 403, from a crawl.
-   [The Verification Ledger](https://agentreadycrawlers.com/verification/): which operators publish IP lists, reverse DNS patterns and key directories, and which publish nothing.
-   [15 vendors](https://agentreadycrawlers.com/vendors/ranking/) on a published rubric, with head-to-heads and alternatives.
-   [A robots.txt builder](https://agentreadycrawlers.com/robots-builder/), [14 guides](https://agentreadycrawlers.com/guides/), [a glossary](https://agentreadycrawlers.com/glossary/), [dated news](https://agentreadycrawlers.com/news/) and [a FAQ](https://agentreadycrawlers.com/faq/).

## How it is made

Rows are read from operators' own pages and dated. The block rates come from one crawl of 13,071 domains on 2026-11-04, parsed the way RFC 9309 describes. Vendors are graded on what they document, with the evidence beside each grade. [The method page](https://agentreadycrawlers.com/method/) has the detail, including what our own crawl does and where Ora's scan fits on this layer.

The site is static HTML. Every page has a markdown twin, [llms.txt](https://agentreadycrawlers.com/llms.txt) and [llms-full.txt](https://agentreadycrawlers.com/llms-full.txt) list the whole site, and its own robots.txt allows the AI crawlers it writes about.

## Sources

-   [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04
-   [Editorial policy (this site)](https://agentreadycrawlers.com/editorial-policy/) read 2026-11-04
-   [Corrections (this site)](https://agentreadycrawlers.com/corrections/) read 2026-11-04

Last reviewed 2026-11-05