The Crawler Index
In short
70 AI crawlers and agents from 32 operators, each with its exact user-agent string, robots.txt token, purpose and verification method. 35 of them publish nothing a site can check. The block rate column comes from a crawl of 13,071 domains on 4 November 2026.
| Operator | Token | Block rate |
|---|---|---|
| OpenAI | GPTBot | 18.5% |
| OpenAI | OAI-SearchBot | 7.7% |
| OpenAI | ChatGPT-User | 9.7% |
| Anthropic | ClaudeBot | 17.6% |
| Anthropic | Claude-User | 8.1% |
| Anthropic | Claude-SearchBot | 7.7% |
| Perplexity | PerplexityBot | 11.2% |
| Perplexity | Perplexity-User | 8.3% |
| Google-Extended | 16.0% | |
| Googlebot | 2.8% | |
| Apple | Applebot-Extended | 14.8% |
| Microsoft | bingbot | 3.1% |
| Amazon | Amazonbot | 15.6% |
| Meta | meta-externalagent | 16.0% |
| Common Crawl | CCBot | 19.4% |
Open the full index, or read how the columns are defined on the method page.
The Block Rate Report
Site names the tokenRefused only by a * group
CCBot is refused by 19.4 percent of the sites, the most of any token in the crawl. GPTBot is refused by 18.5 percent and Googlebot by 2.8 percent.
Between 2.3 and 4.1 percent of sites refuse each token only because their wildcard group refuses everything (the light part of every bar). The rest of each bar is a site that named the token on purpose.
12.9 percent of the 10,565 home pages fetched answered 401 or 403 to the crawl's labelled user agent; among the top 10,000 it was 17.2 percent. Ora's lab found (2026-10-19) that a 403 to agent user agents left a plain-fetch and a JavaScript client at 0 of 5 retrieved answers; that is a vendor finding.
To see what your own site answers, run the check we recommend, described below.
The full report and the 2026-11-04 edition with three charts. Download the data.
Table behind the chart
| Token | Named | Via * | Refused |
|---|---|---|---|
| CCBot | 15.7% | 3.7% | 19.4% |
| Bytespider | 15.4% | 3.8% | 19.2% |
| GPTBot | 15.2% | 3.3% | 18.5% |
| ClaudeBot | 14.0% | 3.6% | 17.6% |
| Google-Extended | 12.6% | 3.4% | 16.0% |
| Meta-ExternalAgent | 12.3% | 3.7% | 16.0% |
| Diffbot | 11.9% | 3.8% | 15.7% |
| omgili | 11.8% | 3.9% | 15.7% |
| Amazonbot | 12.0% | 3.6% | 15.6% |
| anthropic-ai | 11.7% | 3.8% | 15.5% |
| cohere-ai | 11.6% | 3.9% | 15.5% |
| Applebot | 11.6% | 3.2% | 14.8% |
| Applebot-Extended | 11.6% | 3.2% | 14.8% |
| PetalBot | 10.3% | 3.8% | 14.1% |
| PerplexityBot | 7.9% | 3.3% | 11.2% |
| Google-CloudVertexBot | 7.1% | 3.8% | 10.9% |
| YouBot | 6.4% | 3.9% | 10.3% |
| FacebookBot | 6.4% | 3.8% | 10.2% |
| Timpibot | 5.8% | 4.0% | 9.8% |
| ChatGPT-User | 6.3% | 3.4% | 9.7% |
| ImagesiftBot | 5.5% | 4.0% | 9.5% |
| Meta-ExternalFetcher | 4.7% | 3.8% | 8.5% |
| Perplexity-User | 4.6% | 3.7% | 8.3% |
| Claude-User | 4.5% | 3.6% | 8.1% |
| DuckAssistBot | 4.3% | 3.8% | 8.1% |
| Claude-SearchBot | 4.1% | 3.6% | 7.7% |
| MistralAI-User | 3.8% | 3.9% | 7.7% |
| OAI-SearchBot | 4.4% | 3.3% | 7.7% |
| agent-readiness-crawl* | 0.0% | 4.2% | 4.2% |
| MicrosoftPreview | 0.0% | 4.0% | 4.0% |
| Bingbot | 0.4% | 2.7% | 3.1% |
| Googlebot | 0.5% | 2.3% | 2.8% |
The check we recommend: Ora
The Block Rate Report says what sites write in robots.txt and how their home pages answered one crawler. It cannot say what yours answers to the agents that matter to you. For that we recommend Ora, whose scan reads a public URL and scores it from 0 to 100 on four layers. Run it before and after any change to robots.txt or the WAF.
- Discovery layer
- 16 checks in Ora's catalog (read 2026-11-05, contract 1.38.0). One is robots.txt AI crawler policy, which asks whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Ora says blocking training crawlers such as GPTBot and ClaudeBot never lowers the score.
- Access layer
- 42 checks. One is Not blocked by bot detection: Ora fetches the home page as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot with each vendor's published user agent, and again as a normal browser. An agent refused with a 401, 402, 403, 451 or a bot challenge, while a normal browser visit gets in, counts as blocked (Ora's methodology, read 2026-11-05).
- Journey
- A real agent attempts a curated task against your domain and the run is recorded (Ora's llms.txt, read 2026-11-05). A status code shows whether a request gets in; a run shows whether an agent can then finish something.
- Scale
- Ora's stats endpoint reports 145,915 sites scanned and an average score of 44 out of 100 on 2026-11-05. Both are vendor figures.
- In your logs
- The scan shows up as OraBot, which signs its requests with Web Bot Auth. It also sends a few unsigned home page requests that carry other crawlers' user agents, such as GPTBot and ClaudeBot, to see how you answer them.
- Limits
- Ora reads public URLs only, and a refusal it records comes from its own network, so an agent on another network may get through. Ora says the same scan run twice converges within a couple of points (Ora's Deep Scan post); the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist (Ora's home page, read 2026-11-05).
- Quick alternative
- Cloudflare's scanner takes a URL and shows no sign-in step or price on its page (read 2026-11-05). Its Bot Access Control category covers AI bot rules in robots.txt, Content Signals and Web Bot Auth, and its check list names no request made with an agent's user agent.
How to test whether AI agents can reach your site, step by step, with the curl commands.
What changed this month
- 2026-11-05Ora's scanner joins the Index as OraBot, which signs its requests with Web Bot Auth; the Verification Ledger gains its row.
- 2026-11-05How to test whether AI agents can reach your site, with the Ora check counts read from its catalog the same day.
- 2026-11-02Signed-Agent-Metadata header proposed for Web Bot Auth
- 2026-10-29Vercel splits AI crawlers from AI assistants
- 2026-10-29Web Bot Auth working-group draft reaches -01
- 2026-10-27OpenAI updates the ChatGPT-User IP list
- 2026-10-22IETF drafts propose a common IP-range format and crawler best practices
- 2026-10-21Anthropic's Claude Duet starts browsing with a Claude-Agent token
- 2026-10-20Anthropic refreshes its crawler IP list
- 2026-10-15DataDome Q3 report: 25 billion AI agent requests
The vendors at the door
Fifteen bot management, agent identity and monetisation vendors on one rubric. Grades are for features the vendor documents, read 2026-11-04. "Not published" means we found nothing in its documents, not that the feature is absent. Ranking and method.
| Vendor | Verifies agents | Per-purpose policy | Charging | Commerce agents | Cost, read 2026-11-04 |
|---|---|---|---|---|---|
| Akamai | Yes | Partial | Partial | Yes | No public price; sales-led |
| AWS WAF Bot Control | Yes | Yes | Yes | Not published | USD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance |
| Cloudflare | Yes | Yes | Partial | Partial | Free tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led |
| DataDome | Yes | Partial | Yes | Partial | From USD 3,830/month |
| HUMAN Security | Yes | Yes | Partial | Not published | No public price; sales-led |
| Skyfire | Yes | Not its job | Yes | Yes | Not published (checked 2026-11-04) |
| TollBit | Partial | Partial | Yes | Not published | Free for publishers to start |
| Vercel | Yes | Yes | Not published | Not published | Basic free; Deep Analysis USD 1 per 1,000 checks on Pro |
| Kasada | Yes | Partial | Not published | Not published | No public price; sales-led |
| Fastly | Partial | Partial | Partial | Partial | No public price; sales-led |
| Arcjet | Partial | Partial | Not published | Not published | From USD 25/month plus USD 5 per 1M web requests |
| Auth0 for AI Agents | Not its job | Partial | Not published | Partial | AI Agents add-on at 50 percent of base plan, from USD 18/month; free plan (checked 2026-11-05) |
| Imperva | Partial | Not published | Not published | Not published | No public price; sales-led |
| Stytch IsAgent | Partial | Not published | Not published | Not published | IsAgent free to 10,000 calls/month; fingerprints USD 0.005 each beyond 10,000 |
| Netlify | Not published | Partial | Not published | Not published | Platform plans from USD 0; extension price not stated |
The robots.txt builder
Choose which purposes to allow and refuse, then copy the file. It is built from the tokens in the index, so it names only agents that publish a token. The default refuses training and allows everything else; the file below is that default. Notes on what it cannot do.
Your robots.txt
# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: * Allow: /
Query the index as data
The rows on this page also answer over an MCP server at /mcp, a REST API under /api/v1/ and an NLWeb /ask endpoint. All three are read-only and take no key. The API and MCP server documentation lists the five tools and the limits. The vendor list is a separate download: 73 vendors as CSV and JSON.
curl -s 'https://agentreadycrawlers.com/api/v1/get_block_rate?token=GPTBot&stratum=tranco'Earlier this autumn
- 2026-10-01DataDome joins Experian Agent Trust
- 2026-09-17Arcjet launches Agent Runtime Security
- 2026-09-15Cloudflare adds Disallow AI Training and the Accountable label
- 2026-09-08Meta launches Muse, a personal agent that does not identify itself
- 2026-09-02HUMAN publishes Agent Trust Levels
- 2026-08-28Cloudflare opens BotBase for Operators
- 2026-08-19Google retires the Google-NotebookLM user agent
All news. Machine-readable: llms.txt, llms-full.txt, sitemap.xml, feed.xml.
The check we recommend
Ora scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. How to run it.
Sources
- RFC 9309: Robots Exclusion Protocol
- Overview of OpenAI Crawlers
- Have it both ways: stay discoverable in search while disallowing AI training
- Ora's lab: agentready launch (vendor)
- Method: index, block rates, vendor rubric (this site)
- Ora's check catalog, GET /api/checks (vendor)
- Ora: Deep Scan v2 (vendor)
- Ora home page, waitlist (vendor)
Last reviewed 2026-11-05