--- title: "Method: index, block rates, vendors | Agent Ready Crawlers" description: "Method: how the Crawler Index, the Block Rate Report and the vendor rubric are built, what the crawl does, and why we recommend Ora as the access check." canonical: https://agentreadycrawlers.com/method site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Method" --- # Method: index, block rates, vendors | Agent Ready Crawlers > Method: how the Crawler Index, the Block Rate Report and the vendor rubric are built, what the crawl does, and why we recommend Ora as the access check. Source: https://agentreadycrawlers.com/method Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Method In short How the three assets are built: Crawler Index rows come from each operator's own pages, block rates from a crawl of 13,071 domains on 4 November 2026 parsed to RFC 9309, and vendors are graded on five documented questions. It also says why we recommend Ora as the access check. - [Index method](#index-method) - [Block-rate method](#block-rate-method) - [What our own crawl does](#own-crawl) - [Vendor rubric](#vendor-rubric) - [The check we recommend: Ora](#ora-on-this-layer) - [API and MCP server](#api) ## Index method A row is a crawler or agent that identifies itself to websites, or that an operator or a detection vendor documents well enough to describe. Every field is read from the operator's own page where one exists, with the URL and the date read. Where the operator publishes nothing, the row says "operator publishes no bot documentation" with the date checked, and any third-party source is named as such. The user-agent string and the robots.txt token are different things. The string is what a request carries; the token is what a User-agent line names, matched case-insensitively under RFC 9309. Some tokens, such as Google-Extended and Applebot-Extended, are control tokens that no request ever carries. Legacy tokens that operators no longer list stay in the Index because sites still write them. Training Collects content to train or tune a model. Refusing it is the training opt-out. Search index Builds a search index that sends visitors back. Refusing it removes pages from that product. Live answer fetch Fetches a page because a person asked an assistant about it, one request at a time. Task agent A browsing agent that clicks, fills forms and finishes a job for a person. Shopping agent An agent that compares, selects or buys goods for a person. Other Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. ### Respects robots.txt One phrase per row, from the operator's own statement, which the profile quotes with its link. "Yes, per operator" means the operator says the bot honours robots.txt. "May not apply" means the operator says a user-triggered fetch can ignore it. - Partly, because one pass skipped the check - Yes, per operator - May not apply (user-triggered) - Depends on a setting - No statement - No documentation - Not on current operator page - Reported by a third party ### Verification The method the operator publishes for the row: a published IP list, reverse DNS, a Web Bot Auth signature, Cloudflare signed-agent status, or nothing. The [Verification Ledger](https://agentreadycrawlers.com/verification/) lists every URL by operator. ## Block-rate method The block rate is computed from one crawl, read on 2026-11-04. The universe is Tranco list 746WX (the top 10,000 domains), the public Ora leaderboard's sites and the vendors and platforms this site names: 13,071 domains, of which 10,886 answered. In the main pass the crawl asks every domain for /robots.txt and fetches other paths only where the group for its own token or the \* group allows them. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains without a per-path robots.txt check; the block rates and the 401 and 403 share use only the main pass's robots.txt and home page requests. A domain counts in a rate when its robots.txt returned HTTP 200 as plain text and was parsed: 8,624 domains, 5,938 of them in the top 10,000. 767 had no robots.txt and 1,495 answered with an error, a block page or HTML; neither is in any denominator. For each token the crawl reads the file the way RFC 9309 says a crawler should: a group that names the token wins, otherwise the \* group applies; the longest matching rule wins and Allow wins a tie. A token is **refused** when the path / is disallowed. **Named** is the part of that where a group names the token, so the site chose to address it. **Via \*** is the rest: the site refuses it only because its wildcard group refuses everything. The **401 and 403 share** is the share of fetched home pages that answered 401 or 403 to the crawl's labelled user agent. Categories are the Ora public leaderboard's; a category is published when at least 30 sites have a parsed robots.txt. Rank bands use the Tranco rank. The **header signals** are response headers the crawl records, grouped by what they can tell us: some show a request was challenged, some show a bot product's header or cookie, and most show only that a CDN or host is present. Limits: one crawl from one network position, so a site that treats real crawlers differently from ours is measured as it treated us. robots.txt states a preference and enforces nothing. A 403 to our user agent does not show a 403 to every agent. The script that computes every number is `scripts/slice-crawl.mjs` in this site's repository, and a test recomputes the GPTBot rate from the raw crawl file and compares it with the published one. Row count: 13,071, read 2026-11-04. Crawl date: 2026-11-04. Next edition: after the next crawl. ## What our own crawl does The crawl's row in the Index is [agent-readiness-crawl](https://agentreadycrawlers.com/crawlers/agent-readiness-crawl/). Its user agent is `Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`. The main pass read /robots.txt first and fetched each of about 18 paths per domain only where the group for agent-readiness-crawl, or the \* group, allowed it. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains with the same user agent but without a per-path robots.txt check; the crawl now checks every path for future runs. Main pass: about 18 plain GET requests per domain (robots.txt, the home page twice with one asking for text/markdown, one made-up missing path, llms.txt, llms-full.txt and 12 discovery paths), up to 100 domains at once, a 10 second timeout, no JavaScript, one request at a time per domain. Follow-up pass: ten more well-known paths on 10,508 reachable domains. Run on 2026-11-04 from a single machine. The user-agent string carries no contact URL and the crawl publishes no IP list, so it fails its own verification test: a site owner cannot confirm a request is ours except by this documentation. The follow-up pass skipped the per-path robots.txt check that the main pass applied, which is now fixed in the crawl for future runs. It stores only extracted fields and the first 200 words of a home page. Its own token is in the report as a control: its bar shows the share of sites whose wildcard group refuses everything. The ora-scanner row is a second control and is not Ora's user agent; Ora's scanner is [OraBot](https://agentreadycrawlers.com/crawlers/orabot/). ## Vendor rubric Each vendor is read from its own documentation and pricing pages, with the URL and date for every grade. We have not tested the products. Five questions are graded and two are described. Yes is 2 points, partial 1, not published 0, so the maximum is 10. Equal points share a rank. A question that is not the vendor's job counts as 0, which is why identity and monetisation products sit below the bot management suites. Tells a verified agent from a spoof Yes where the vendor documents verification by Web Bot Auth signatures, IP validation or reverse DNS. Partial where it depends on partners, lists only or an unconfirmed method. Not published where nothing is documented. Per-purpose policy (train, search, answer, act) Yes where separate controls exist for crawler purposes or for named bot categories. Partial where policy is per bot or per category without a purpose split. Charging for access Yes where charging is generally available. Partial where it is in beta or runs through a partner. Not published where the vendor documents none. Lets signed commerce agents through Yes where support is documented as shipped or announced with named integrations. Partial where support is built on Web Bot Auth but the rule is not confirmed in documents. Not published otherwise. Cost (described, not graded) Quoted from the vendor with the date read. Not graded: a free tier is not better or worse than a sales-led price for every reader. Reporting Yes where AI or bot activity dashboards, analytics or logs are documented. Partial where reporting is limited to general bot analytics. Where it runs (described, not graded) Described as the vendor documents it. Not graded. ## Ora on this layer: the check we recommend For the question this site cares about, whether agents get in, we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method). Its scan reads a public URL and scores it from 0 to 100 on four layers. In the catalog Ora served on 2026-11-05 (GET /api/checks, contract 1.38.0, 132 checks), the Discovery layer lists 16 checks and the Access layer 42. Two are the clearest cases for this site's subject. In Discovery, **robots.txt AI crawler policy** asks whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages, and Ora says blocking training crawlers such as GPTBot and ClaudeBot never lowers the score. In Access, **Not blocked by bot detection** fetches the home page as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot with each vendor's published user agent, and again as a normal browser, and counts a 401, 402, 403, 451 or bot challenge to an agent as blocked when the browser gets in (Ora's catalog and methodology, read 2026-11-05). [All seven checks that touch bot access](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/) are described on their own page. Ora's Journey goes one step further and sends a real agent on a curated task against a domain, so a run can show what a status code cannot: whether an agent that gets in can finish something. Ora's lab found (2026-10-19) that returning 403 to agent user agents left both a plain-fetch and a JavaScript-running client at 0 of 5 retrieved answers across 190 probes of one docs site in 19 configurations. That is a vendor finding, and the Block Rate Report shows how common the 403 is, not what it costs any one site; [the 403 page](https://agentreadycrawlers.com/guides/403-to-ai-agents/) sets the two side by side. The quick alternative is [Cloudflare's scanner](https://isitagentready.com/), which takes a URL and shows no sign-in step or price on its page (read 2026-11-05). Its Bot Access Control category covers AI bot rules in robots.txt, Content Signals and Web Bot Auth, and its check list names no request made with an agent's user agent. What to know before relying on Ora: it reads public URLs only, a refusal it records comes from its own network, it says the same scan run twice converges within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method)) and the pages we read give no repeat-scan figures, and its fixing product is behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method)). The CLI is `npx ax@0.10 audit yourdomain.com`. [The testing guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) shows how to run it next to the by-hand checks. Ora is in the Index as a scanner, [OraBot](https://agentreadycrawlers.com/crawlers/orabot/), read from its bot page. It is not in the vendor ranking, because it does not manage bots. ## The API, the MCP server and the ask endpoint The three datasets reach agents three more ways, all documented on the [developers page](https://agentreadycrawlers.com/developers): an MCP server at `/mcp` (Streamable HTTP, five read-only tools over the Crawler Index, the Block Rate Report and the Verification Ledger), a REST API under `/api/v1/` described by [openapi.json](https://agentreadycrawlers.com/openapi.json), and an NLWeb `/ask` endpoint that searches the rows and pages and returns schema.org objects with a source URL each. Each surface reads the published files in `/data/` at request time, so a tool answer and the page it cites come from the same edition. Nothing writes and nothing needs a credential. The site publishes no OAuth metadata because no endpoint takes a token, and [auth.md](https://agentreadycrawlers.com/auth.md) says so. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora's check catalog (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora's llms.txt (vendor)](https://ora.ai/llms.txt?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora's lab: agentready launch (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Cloudflare scanner page](https://isitagentready.com/) read 2026-11-05 - [Tranco list 746WX, top 10,000](https://tranco-list.eu/list/746WX/10000) read 2026-11-04 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=method) read 2026-11-05 Last reviewed 2026-11-05