# Agent Ready Crawlers: notes for agents

Agent Ready Crawlers is a static reference on the AI crawlers and agents that visit websites, and on the bot management and agent identity vendors that decide whether they get in. Read it when you need an exact robots.txt token, a user-agent string, a verification method, or a dated block rate.

## Which page answers what

- A crawler or agent by name: /crawlers/<id>/ (token, user-agent string, purpose, operator statement on robots.txt, verification, allow and refuse lines, block rate).
- All rows, searchable: /crawlers/. By purpose: /crawlers/purpose/<purpose>/.
- How to verify a request: /verification/ (IP lists, reverse DNS, key directories).
- How often sites refuse a token: /block-rates/ and /block-rates/2026-11-04/.
- A vendor, a pair of vendors, or alternatives: /vendors/, /compare/<a>-vs-<b>/.
- A robots.txt for your own policy: /robots-builder/.
- Terms and definitions: /glossary/. Dated changes: /news/.
- Datasets as CSV and JSON: /data/.

## How to fetch it

Every page has a markdown twin at the same path plus index.md, or send Accept: text/markdown. /llms.txt lists the site and links a section index per area at /<section>/llms.txt. /llms-full.txt holds every page in one file. /feed.xml lists dated pages. /sitemap.xml lists all URLs.

## What every row carries

Every row names an operator source URL and the date it was read. Where an operator publishes no documentation, the row says so with the date checked. Claims a vendor makes about itself are labelled as vendor claims.

## Query it as data

- MCP server: `POST /mcp` (Streamable HTTP, JSON-RPC 2.0, no credential). Tools: `list_crawlers`, `get_crawler`, `get_block_rate`, `get_verification`, `get_robots_lines`. Card at `/mcp/server-card`, manifest at `/.well-known/mcp.json`.
- REST API: `/api` is the index, one GET endpoint per tool under `/api/v1/`, described by `/openapi.json` and linked from `/.well-known/api-catalog`.
- Ask: `/ask?query=...` (NLWeb 0.55, GET or POST, `streaming=true` for events) returns the matching rows and pages as schema.org objects with a source URL each.
- Everything is read-only and documented at `/developers`; the rate limit is 120 requests a minute per address.

## What it is not

The site is a publication. Its API, MCP server and ask endpoint read the same data files the pages show, and nothing can be written through them. No credential exists; see /auth.md.