--- title: "API and MCP server | Agent Ready Crawlers" description: "How to query the crawler data: the MCP server at /mcp, the REST API described by openapi.json and the NLWeb ask endpoint, all read-only and open." canonical: https://agentreadycrawlers.com/developers site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Developers" --- # API and MCP server | Agent Ready Crawlers > How to query the crawler data: the MCP server at /mcp, the REST API described by openapi.json and the NLWeb ask endpoint, all read-only and open. Source: https://agentreadycrawlers.com/developers Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # API and MCP server In short The Crawler Index, the Block Rate Report and the Verification Ledger are also data an agent can query: an MCP server at /mcp with 5 read-only tools, a REST API under /api/v1 described by openapi.json, and an NLWeb /ask endpoint. No credential, 120 requests a minute, edition 2026-11-04. ## Connect the MCP server The server speaks the Streamable HTTP transport at `https://agentreadycrawlers.com/mcp`. Send JSON-RPC 2.0 messages with POST and read the JSON answer. It keeps no session and has no stream the server starts, so one request gets one answer. A client that accepts a URL connects with that address alone; the card at [/mcp/server-card](https://agentreadycrawlers.com/mcp/server-card) and the manifest at [/.well-known/mcp.json](https://agentreadycrawlers.com/.well-known/mcp.json) carry the same facts in the forms clients read. ```bash curl -s https://agentreadycrawlers.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"my-agent","version":"1.0"}}}' ``` ```bash curl -s https://agentreadycrawlers.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_robots_lines","arguments":{"tokens":["GPTBot","ClaudeBot"],"action":"refuse"}}}' ``` A tool answers with its result as `structuredContent` and the same JSON as text. A wrong argument answers `isError: true` with the parameter list, and an unknown tool answers JSON-RPC error -32602. `resources/list` names the page index, the methodology, this page and the dataset files, which `resources/read` returns. ## The tools | Tool | What it returns | Parameters | | --- | --- | --- | | list_crawlers | Lists rows of the Crawler Index (id, name, operator, robots.txt token, purpose, how the operator says it treats robots.txt, verification method, block rates, profile URL). Filter by operator, purpose, verification method, robots.txt reading or words. Call it to find a crawler id before get_crawler, or to answer which crawlers do X. | operator?: Operator name as in the index, case-insensitive, for example OpenAI, Anthropic, Google or Perplexity. purpose?: training, search-index, live-answer-fetch, task-agent, shopping-agent or other. verification_method?: How the operator lets a site verify a request: ip-json, reverse-dns, web-bot-auth, cloudflare-verified or none-published. respects_robots?: The reading of the operator statement on robots.txt, for example "Yes, per operator", "May not apply" or "No documentation". q?: Words to look for in the id, name, operator, token or user-agent string, for example "agent" or "Mozilla". sort?: block_rate_top10000 or block_rate_all (highest first, rows without a rate last) or name. Default is the order of the Crawler Index. limit?: Page size, 1 to 100 (default 20). cursor?: Opaque cursor from next_cursor of the previous page. | | get_crawler | Returns one Crawler Index row by id, robots.txt token or name (for example gptbot, GPTBot or ClaudeBot): operator, exact user-agent string, purpose, the operator statement on robots.txt with its source URL, verification method and IP list URL, rate behaviour, the allow and refuse lines, the vendor rule names, both block rates and the profile page URL to cite. | id: Crawler id (gptbot), robots.txt token (GPTBot) or name, case-insensitive. | | get_block_rate | Returns how many sites in the crawl refuse one robots.txt token (for example GPTBot or ClaudeBot) at the path /: the parsed-site count, the refused count, the part where a robots.txt group names the token, the shares, and the 401 and 403 share of home pages in the same cut. Without stratum it returns all four site lists; with category it returns that category. | token: A robots.txt token such as GPTBot, or a crawler id such as gptbot, case-insensitive. stratum?: all (every crawled domain with a parsed robots.txt), tranco (top 10,000), tranco_1_1000 or tranco_1001_10000. Omit for all four. category?: One of the report's categories, for example "Developer Tools" or "Healthcare"; only categories with at least 30 parsed sites are published. Cannot be combined with stratum. | | get_verification | Returns one Verification Ledger row: the IP list URLs, the reverse DNS pattern, the Web Bot Auth key directory, the Cloudflare verified status, the card-network registry lines, a summary, the source URL and the date read, plus the operator's crawlers with their verification method. Call it to check a request is really from an operator. | operator: Operator name as in the ledger (OpenAI, Anthropic, Google, Perplexity), or the id or robots.txt token of one of its crawlers, case-insensitive. | | get_robots_lines | Returns the allow lines or the refuse lines the Crawler Index holds for up to 20 crawlers (robots.txt tokens or crawler ids), per crawler and joined into one robots.txt body in the order asked. It only joins the rows' own lines; crawlers without a robots.txt token are listed as unmatched. Call it when writing a robots.txt policy for named crawlers. | tokens: robots.txt tokens or crawler ids, for example ["GPTBot", "ClaudeBot"]; in a query string, a comma-separated list. 1 to 20 entries. action: allow returns the Allow lines, refuse returns the Disallow lines. | A question mark after a parameter name marks it optional. ## The REST API Each tool is also one GET endpoint under `/api/v1/`, with the parameters in the query string and the same JSON in the answer. [/api](https://agentreadycrawlers.com/api) is the index. [/openapi.json](https://agentreadycrawlers.com/openapi.json) describes every operation with its parameters, response schema and the Error schema, and [/.well-known/api-catalog](https://agentreadycrawlers.com/.well-known/api-catalog) links that description the way RFC 9727 says. List endpoints take `limit` (1 to 100) and the `next_cursor` of the previous page as `cursor`. A list parameter is a comma-separated value. - [/api/v1/list\_crawlers?purpose=training](https://agentreadycrawlers.com/api/v1/list_crawlers?purpose=training) List crawlers and agents. Paginated. - [/api/v1/get\_crawler?id=gptbot](https://agentreadycrawlers.com/api/v1/get_crawler?id=gptbot) One crawler in full. - [/api/v1/get\_block\_rate?token=GPTBot](https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot) Block rate of one robots.txt token. - [/api/v1/get\_verification?operator=OpenAI](https://agentreadycrawlers.com/api/v1/get_verification?operator=OpenAI) How an operator lets you verify its requests. - [/api/v1/get\_robots\_lines?tokens=GPTBot,ClaudeBot&action=refuse](https://agentreadycrawlers.com/api/v1/get_robots_lines?tokens=GPTBot,ClaudeBot&action=refuse) The robots.txt lines for a set of crawlers. - [/api](https://agentreadycrawlers.com/api) The index: endpoints, datasets, limits and the versioning policy. ```bash curl -s 'https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot&stratum=tranco' | jq '.rates[0].pct_effective' ``` ## Ask in plain words The `/ask` endpoint follows NLWeb 0.55. It searches the crawler rows, the block-rate tokens, the ledger rows and every page, and returns schema.org objects that each carry a `grounding.source` URL to cite. GET takes `query`; POST takes the NLWeb request object. Add `streaming=true`, or `prefer.streaming`, for server-sent events named start, result and complete. `mode=summarize` adds one SearchSummary item assembled from the results by template, with no model involved. ```bash curl -s 'https://agentreadycrawlers.com/ask?query=ClaudeBot+block+rate' ``` ```bash curl -s https://agentreadycrawlers.com/ask -H 'Content-Type: application/json' -d '{"query":{"text":"which operators publish a Web Bot Auth key directory"},"prefer":{"mode":"list"},"meta":{"version":"0.55"}}' ``` A query with no match answers `_meta.response_type: failure` with code NO\_RESULTS, and a missing query answers 400 with INVALID\_QUERY. Context objects are accepted and ignored, because every answer comes from the data and not from a conversation. ## Discovery files - [/openapi.json](https://agentreadycrawlers.com/openapi.json) OpenAPI 3.1 for the REST API and /ask, linked from every page as rel=service-desc and from the Link header of the home page. - [/.well-known/api-catalog](https://agentreadycrawlers.com/.well-known/api-catalog) RFC 9727 linkset: the API, the MCP server and the ask endpoint with their descriptions. - [/mcp/server-card](https://agentreadycrawlers.com/mcp/server-card) The MCP server card (name, version, remote URL, protocol versions), also at /.well-known/mcp/server-card.json. - [/.well-known/mcp.json](https://agentreadycrawlers.com/.well-known/mcp.json) The manifest: endpoint, transport and the tools with their input schemas. - [/.well-known/ai-catalog.json](https://agentreadycrawlers.com/.well-known/ai-catalog.json) AI Catalog 1.0 with entries for the MCP card, the OpenAPI document, the ask endpoint and every document; the same file is at /.well-known/ard.json. - [/schemamap.xml](https://agentreadycrawlers.com/schemamap.xml) Schema feeds: the rows as JSON-LD lines under /feeds/, named in robots.txt. ## Access, limits and errors Credential None. The OpenAPI document declares `security: []` and no security schemes, and [auth.md](https://agentreadycrawlers.com/auth.md) says the same. The site publishes no OAuth metadata, because no endpoint takes a token and metadata for an authorization server that does not exist would point agents at nothing. Rate limit 120 requests a minute per client address, counted per running instance and announced in `RateLimit` and `RateLimit-Policy` headers. A 429 carries `Retry-After`, and the host may add limits of its own above that. Each dataset is one download from [/data/](https://agentreadycrawlers.com/data/), so a bulk reader never needs the API. Errors JSON and never HTML: `{"error":{"code","message","hint","status","docs"}}` with the codes invalid\_params, not\_found, method\_not\_allowed, unsupported\_media\_type, rate\_limited and internal. A missing endpoint under /api answers 404 and lists the endpoints in the hint. Writes There are none, so every call is safe to retry and no idempotency key exists. Every call finishes in one response; nothing runs long enough to need a job pattern. CORS Every answer carries `Access-Control-Allow-Origin: *`, so an agent that runs in a browser can call the API and the MCP server. ## Versioning The path prefix `/api/v1` is the contract. A breaking change, such as a field removed or renamed or a tool removed, ships under `/api/v2` while `/api/v1` keeps answering for at least 180 days, announced on this page and in the `info.version` of openapi.json. Additive changes, such as a new field or tool, keep the prefix. The MCP server follows the same rule, and its card and `serverInfo.version` carry the version number of the OpenAPI document. The data behind every answer is edition 2026-11-04, named in the `X-Dataset-Edition` header and in each tool result. ## Cite the data Every result names the page it comes from; cite that URL. The data files are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) and the cite-this line for each dataset is on the [data page](https://agentreadycrawlers.com/data/). ## What the surface does not do It does not write, hold accounts, generate text with a model or search beyond this site's own rows and pages. A block rate is a count from one crawl and a robots.txt line is a statement of preference, so the API reports both as the pages do. The crawler pages stay the record; the [method page](https://agentreadycrawlers.com/method/) explains how the rows are made. ## Sources - [MCP specification, Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) read 2026-11-04 - [NLWeb specification v0.55](https://github.com/nlweb-ai/website/blob/main/NLWEBSPEC.md) read 2026-11-04 - [RFC 9727, api-catalog](https://www.rfc-editor.org/rfc/rfc9727) read 2026-11-04 - [OpenAPI 3.1.0](https://spec.openapis.org/oas/v3.1.0) read 2026-11-04 - [Schema Feeds specification 0.1](https://github.com/nlweb-ai/website/blob/main/SCHEMA_SPEC.md) read 2026-11-04 - [Data downloads (this site)](https://agentreadycrawlers.com/data/) read 2026-11-04 - [Method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 Last reviewed 2026-11-04