Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

API and MCP server

In short

The Crawler Index, the Block Rate Report and the Verification Ledger are also data an agent can query: an MCP server at /mcp with 5 read-only tools, a REST API under /api/v1 described by openapi.json, and an NLWeb /ask endpoint. No credential, 120 requests a minute, edition 2026-11-04.

Connect the MCP server

The server speaks the Streamable HTTP transport at https://agentreadycrawlers.com/mcp. Send JSON-RPC 2.0 messages with POST and read the JSON answer. It keeps no session and has no stream the server starts, so one request gets one answer. A client that accepts a URL connects with that address alone; the card at /mcp/server-card and the manifest at /.well-known/mcp.json carry the same facts in the forms clients read.

curl -s https://agentreadycrawlers.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"my-agent","version":"1.0"}}}'
curl -s https://agentreadycrawlers.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_robots_lines","arguments":{"tokens":["GPTBot","ClaudeBot"],"action":"refuse"}}}'

A tool answers with its result as structuredContent and the same JSON as text. A wrong argument answers isError: true with the parameter list, and an unknown tool answers JSON-RPC error -32602. resources/list names the page index, the methodology, this page and the dataset files, which resources/read returns.

The tools

Every tool is read-only (readOnlyHint true, destructiveHint false) and answers from the data files of edition 2026-11-04.
ToolWhat it returnsParameters
list_crawlers Lists rows of the Crawler Index (id, name, operator, robots.txt token, purpose, how the operator says it treats robots.txt, verification method, block rates, profile URL). Filter by operator, purpose, verification method, robots.txt reading or words. Call it to find a crawler id before get_crawler, or to answer which crawlers do X. operator?: Operator name as in the index, case-insensitive, for example OpenAI, Anthropic, Google or Perplexity. purpose?: training, search-index, live-answer-fetch, task-agent, shopping-agent or other. verification_method?: How the operator lets a site verify a request: ip-json, reverse-dns, web-bot-auth, cloudflare-verified or none-published. respects_robots?: The reading of the operator statement on robots.txt, for example "Yes, per operator", "May not apply" or "No documentation". q?: Words to look for in the id, name, operator, token or user-agent string, for example "agent" or "Mozilla". sort?: block_rate_top10000 or block_rate_all (highest first, rows without a rate last) or name. Default is the order of the Crawler Index. limit?: Page size, 1 to 100 (default 20). cursor?: Opaque cursor from next_cursor of the previous page.
get_crawler Returns one Crawler Index row by id, robots.txt token or name (for example gptbot, GPTBot or ClaudeBot): operator, exact user-agent string, purpose, the operator statement on robots.txt with its source URL, verification method and IP list URL, rate behaviour, the allow and refuse lines, the vendor rule names, both block rates and the profile page URL to cite. id: Crawler id (gptbot), robots.txt token (GPTBot) or name, case-insensitive.
get_block_rate Returns how many sites in the crawl refuse one robots.txt token (for example GPTBot or ClaudeBot) at the path /: the parsed-site count, the refused count, the part where a robots.txt group names the token, the shares, and the 401 and 403 share of home pages in the same cut. Without stratum it returns all four site lists; with category it returns that category. token: A robots.txt token such as GPTBot, or a crawler id such as gptbot, case-insensitive. stratum?: all (every crawled domain with a parsed robots.txt), tranco (top 10,000), tranco_1_1000 or tranco_1001_10000. Omit for all four. category?: One of the report's categories, for example "Developer Tools" or "Healthcare"; only categories with at least 30 parsed sites are published. Cannot be combined with stratum.
get_verification Returns one Verification Ledger row: the IP list URLs, the reverse DNS pattern, the Web Bot Auth key directory, the Cloudflare verified status, the card-network registry lines, a summary, the source URL and the date read, plus the operator's crawlers with their verification method. Call it to check a request is really from an operator. operator: Operator name as in the ledger (OpenAI, Anthropic, Google, Perplexity), or the id or robots.txt token of one of its crawlers, case-insensitive.
get_robots_lines Returns the allow lines or the refuse lines the Crawler Index holds for up to 20 crawlers (robots.txt tokens or crawler ids), per crawler and joined into one robots.txt body in the order asked. It only joins the rows' own lines; crawlers without a robots.txt token are listed as unmatched. Call it when writing a robots.txt policy for named crawlers. tokens: robots.txt tokens or crawler ids, for example ["GPTBot", "ClaudeBot"]; in a query string, a comma-separated list. 1 to 20 entries. action: allow returns the Allow lines, refuse returns the Disallow lines.

A question mark after a parameter name marks it optional.

The REST API

Each tool is also one GET endpoint under /api/v1/, with the parameters in the query string and the same JSON in the answer. /api is the index. /openapi.json describes every operation with its parameters, response schema and the Error schema, and /.well-known/api-catalog links that description the way RFC 9727 says. List endpoints take limit (1 to 100) and the next_cursor of the previous page as cursor. A list parameter is a comma-separated value.

curl -s 'https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot&stratum=tranco' | jq '.rates[0].pct_effective'

Ask in plain words

The /ask endpoint follows NLWeb 0.55. It searches the crawler rows, the block-rate tokens, the ledger rows and every page, and returns schema.org objects that each carry a grounding.source URL to cite. GET takes query; POST takes the NLWeb request object. Add streaming=true, or prefer.streaming, for server-sent events named start, result and complete. mode=summarize adds one SearchSummary item assembled from the results by template, with no model involved.

curl -s 'https://agentreadycrawlers.com/ask?query=ClaudeBot+block+rate'
curl -s https://agentreadycrawlers.com/ask -H 'Content-Type: application/json' -d '{"query":{"text":"which operators publish a Web Bot Auth key directory"},"prefer":{"mode":"list"},"meta":{"version":"0.55"}}'

A query with no match answers _meta.response_type: failure with code NO_RESULTS, and a missing query answers 400 with INVALID_QUERY. Context objects are accepted and ignored, because every answer comes from the data and not from a conversation.

Discovery files

Access, limits and errors

Credential
None. The OpenAPI document declares security: [] and no security schemes, and auth.md says the same. The site publishes no OAuth metadata, because no endpoint takes a token and metadata for an authorization server that does not exist would point agents at nothing.
Rate limit
120 requests a minute per client address, counted per running instance and announced in RateLimit and RateLimit-Policy headers. A 429 carries Retry-After, and the host may add limits of its own above that. Each dataset is one download from /data/, so a bulk reader never needs the API.
Errors
JSON and never HTML: {"error":{"code","message","hint","status","docs"}} with the codes invalid_params, not_found, method_not_allowed, unsupported_media_type, rate_limited and internal. A missing endpoint under /api answers 404 and lists the endpoints in the hint.
Writes
There are none, so every call is safe to retry and no idempotency key exists. Every call finishes in one response; nothing runs long enough to need a job pattern.
CORS
Every answer carries Access-Control-Allow-Origin: *, so an agent that runs in a browser can call the API and the MCP server.

Versioning

The path prefix /api/v1 is the contract. A breaking change, such as a field removed or renamed or a tool removed, ships under /api/v2 while /api/v1 keeps answering for at least 180 days, announced on this page and in the info.version of openapi.json. Additive changes, such as a new field or tool, keep the prefix. The MCP server follows the same rule, and its card and serverInfo.version carry the version number of the OpenAPI document. The data behind every answer is edition 2026-11-04, named in the X-Dataset-Edition header and in each tool result.

Cite the data

Every result names the page it comes from; cite that URL. The data files are published under CC BY 4.0 and the cite-this line for each dataset is on the data page.

What the surface does not do

It does not write, hold accounts, generate text with a model or search beyond this site's own rows and pages. A block rate is a count from one crawl and a robots.txt line is a statement of preference, so the API reports both as the pages do. The crawler pages stay the record; the method page explains how the rows are made.

Sources

Last reviewed 2026-11-04