Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

The full Crawler Index

In short

The full Crawler Index: 70 rows from 32 operators, one profile page each. By purpose: 9 training, 15 search index, 8 live answer fetch, 20 task agent, 3 shopping agent, 15 other. Search filters the table as you type; the purpose pages are static lists. Block rates are from the 13,071-domain crawl of 2026-11-04.

Operators with two or more rows have their own page: Google, Anthropic, Meta, OpenAI, Amazon, Perplexity, Microsoft, Mistral AI, Apple, Browserbase, DuckDuckGo.

All 70 rows, grouped by operator. Block rate: share of top-10,000 sites with a parsed robots.txt that disallow the token at the root.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
Agent Ready Crawlers agent-readiness-crawl Other Partly None published 4.2% Profile
Amazon Amazonbot Training Yes IP list 15.6% Profile
Amazon Amzn-SearchBot Search index Yes IP list not measured Profile
Amazon Amzn-User Live answer fetch May not apply IP list not measured Profile
Amazon Buy for Me (shopping agent) no token Shopping agent No docs None published no token Profile
Amazon Rufus (shopping assistant) no token Shopping agent No docs None published no token Profile
Amazon Web Services Amazon Bedrock AgentCore Browser no token Task agent No statement Web Bot Auth no token Profile
Anchor Anchor Browser no token Task agent No docs Web Bot Auth no token Profile
Anthropic anthropic-ai Training Legacy None published 15.5% Profile
Anthropic Claude computer use (API tool) no token Task agent No docs None published no token Profile
Anthropic Claude-Agent Task agent Reported None published not measured Profile
Anthropic Claude in Chrome no token Task agent No statement None published no token Profile
Anthropic Claude-SearchBot Search index Yes IP list 7.7% Profile
Anthropic Claude-User Live answer fetch Yes IP list 8.1% Profile
Anthropic ClaudeBot Training Yes IP list 17.6% Profile
Apify Apify Website Content Crawler no token Task agent Setting Web Bot Auth no token Profile
Apple Applebot Search index Yes Reverse DNS 14.8% Profile
Apple Applebot-Extended Training Yes None published 14.8% Profile
Block Goose (open-source agent) no token Task agent No docs Cloudflare signed no token Profile
Browser Use Browser Use (library and cloud) no token Task agent No docs None published no token Profile
Browserbase Browserbase cloud browsers no token Task agent No docs Web Bot Auth no token Profile
Browserbase Stagehand (open-source framework) no token Task agent No docs None published no token Profile
ByteDance Bytespider Other No docs None published 19.2% Profile
Cloudflare Cloudflare Browser Rendering no token Task agent No statement Web Bot Auth no token Profile
Cohere cohere-ai Other No docs None published 15.5% Profile
Common Crawl CCBot Training Yes Reverse DNS 19.4% Profile
Diffbot Diffbot Other Setting None published 15.7% Profile
DuckDuckGo DuckAssistBot Live answer fetch Yes IP list 8.1% Profile
DuckDuckGo DuckDuckBot Search index Yes IP list not measured Profile
Exa ExaSearchBot Search index No docs None published not measured Profile
Firecrawl FirecrawlAgent Other Yes None published not measured Profile
Google Gemini in Chrome (auto browse) no token Task agent No docs None published no token Profile
Google Google-Agent no token Task agent May not apply Web Bot Auth no token Profile
Google Google-CloudVertexBot Other Yes Reverse DNS 10.9% Profile
Google Google-Extended Training Yes None published 16.0% Profile
Google Google-GeminiNotebook (was Google-NotebookLM) no token Live answer fetch May not apply IP list no token Profile
Google Googlebot Search index Yes Reverse DNS 2.8% Profile
Google GoogleOther Other Yes Reverse DNS not measured Profile
Google Storebot-Google Search index Yes Reverse DNS not measured Profile
Hive (ImageSift) ImagesiftBot Other Yes None published 9.5% Profile
Huawei PetalBot Search index Yes Reverse DNS 14.1% Profile
Manus Manus cloud browser no token Task agent No docs None published no token Profile
Meta FacebookBot Other Legacy None published 10.2% Profile
Meta facebookexternalhit Other May not apply None published not measured Profile
Meta meta-externalads Other Yes None published not measured Profile
Meta meta-externalagent Training Yes None published 16.0% Profile
Meta meta-externalfetcher Live answer fetch May not apply None published 8.5% Profile
Meta meta-webindexer Search index Yes None published not measured Profile
Meta Muse (personal agent) no token Shopping agent No docs None published no token Profile
Microsoft bingbot Search index Yes Reverse DNS 3.1% Profile
Microsoft Browse with Copilot (formerly Copilot Actions) no token Task agent No docs None published no token Profile
Microsoft MicrosoftPreview Other Yes Reverse DNS 4.0% Profile
Mistral AI MistralAI-Index Search index Yes IP list not measured Profile
Mistral AI MistralAI-Training Training Yes None published not measured Profile
Mistral AI MistralAI-User Live answer fetch Yes IP list 7.7% Profile
OpenAI ChatGPT agent (Cloud browser, formerly Operator) no token Task agent No statement Web Bot Auth no token Profile
OpenAI ChatGPT Atlas (browser agent mode) no token Task agent No docs None published no token Profile
OpenAI ChatGPT-User Live answer fetch May not apply IP list 9.7% Profile
OpenAI GPTBot Training Yes IP list 18.5% Profile
OpenAI OAI-AdsBot Other No statement IP list not measured Profile
OpenAI OAI-SearchBot Search index Yes IP list 7.7% Profile
Ora OraBot Other No statement Web Bot Auth not measured Profile
Perplexity Comet (browser assistant) no token Task agent No docs None published no token Profile
Perplexity Perplexity Computer no token Task agent No docs None published no token Profile
Perplexity Perplexity-User Live answer fetch May not apply IP list 8.3% Profile
Perplexity PerplexityBot Search index Yes IP list 11.2% Profile
Timpi Timpibot Search index No docs None published 9.8% Profile
Webz.io omgili Other Yes None published 15.7% Profile
xAI Grok Bot (agent) no token Task agent No docs None published no token Profile
You.com YouBot Search index No docs None published 10.3% Profile

Added on 2026-11-05

Sources

Last reviewed 2026-11-05