---
title: "Crawler Index: every row | Agent Ready Crawlers"
description: "All 70 AI crawlers and agents in the Crawler Index with operator, token, purpose, verification and block rate, and a profile page per row. Read 2026-11-05."
canonical: https://agentreadycrawlers.com/crawlers
site: "Agent Ready Crawlers"
last_updated: 2026-11-05
section: "Crawlers"
---

# Crawler Index: every row | Agent Ready Crawlers

> All 70 AI crawlers and agents in the Crawler Index with operator, token, purpose, verification and block rate, and a profile page per row. Read 2026-11-05.

Source: https://agentreadycrawlers.com/crawlers
Site: Agent Ready Crawlers
Last reviewed: 2026-11-05

# The full Crawler Index

In short

The full Crawler Index: 70 rows from 32 operators, one profile page each. By purpose: 9 training, 15 search index, 8 live answer fetch, 20 task agent, 3 shopping agent, 15 other. Search filters the table as you type; the purpose pages are static lists. Block rates are from the 13,071-domain crawl of 2026-11-04.

-   [Training](https://agentreadycrawlers.com/crawlers/purpose/training/)
-   [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/)
-   [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/)
-   [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/)
-   [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/)
-   [Other](https://agentreadycrawlers.com/crawlers/purpose/other/)

Operators with two or more rows have their own page: [Google](https://agentreadycrawlers.com/crawlers/operator/google/), [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/), [Meta](https://agentreadycrawlers.com/crawlers/operator/meta/), [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/), [Amazon](https://agentreadycrawlers.com/crawlers/operator/amazon/), [Perplexity](https://agentreadycrawlers.com/crawlers/operator/perplexity/), [Microsoft](https://agentreadycrawlers.com/crawlers/operator/microsoft/), [Mistral AI](https://agentreadycrawlers.com/crawlers/operator/mistral-ai/), [Apple](https://agentreadycrawlers.com/crawlers/operator/apple/), [Browserbase](https://agentreadycrawlers.com/crawlers/operator/browserbase/), [DuckDuckGo](https://agentreadycrawlers.com/crawlers/operator/duckduckgo/).

All 70 rows, grouped by operator. Block rate: share of top-10,000 sites with a parsed robots.txt that disallow the token at the root.

| Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile |
| --- | --- | --- | --- | --- | --- | --- |
| Agent Ready Crawlers | agent-readiness-crawl | Other | Partly | None published | 4.2% | Profile |
| Amazon | Amazonbot | Training | Yes | IP list | 15.6% | Profile |
| Amazon | Amzn-SearchBot | Search index | Yes | IP list | not measured | Profile |
| Amazon | Amzn-User | Live answer fetch | May not apply | IP list | not measured | Profile |
| Amazon | Buy for Me (shopping agent) no token | Shopping agent | No docs | None published | no token | Profile |
| Amazon | Rufus (shopping assistant) no token | Shopping agent | No docs | None published | no token | Profile |
| Amazon Web Services | Amazon Bedrock AgentCore Browser no token | Task agent | No statement | Web Bot Auth | no token | Profile |
| Anchor | Anchor Browser no token | Task agent | No docs | Web Bot Auth | no token | Profile |
| Anthropic | anthropic-ai | Training | Legacy | None published | 15.5% | Profile |
| Anthropic | Claude computer use (API tool) no token | Task agent | No docs | None published | no token | Profile |
| Anthropic | Claude-Agent | Task agent | Reported | None published | not measured | Profile |
| Anthropic | Claude in Chrome no token | Task agent | No statement | None published | no token | Profile |
| Anthropic | Claude-SearchBot | Search index | Yes | IP list | 7.7% | Profile |
| Anthropic | Claude-User | Live answer fetch | Yes | IP list | 8.1% | Profile |
| Anthropic | ClaudeBot | Training | Yes | IP list | 17.6% | Profile |
| Apify | Apify Website Content Crawler no token | Task agent | Setting | Web Bot Auth | no token | Profile |
| Apple | Applebot | Search index | Yes | Reverse DNS | 14.8% | Profile |
| Apple | Applebot-Extended | Training | Yes | None published | 14.8% | Profile |
| Block | Goose (open-source agent) no token | Task agent | No docs | Cloudflare signed | no token | Profile |
| Browser Use | Browser Use (library and cloud) no token | Task agent | No docs | None published | no token | Profile |
| Browserbase | Browserbase cloud browsers no token | Task agent | No docs | Web Bot Auth | no token | Profile |
| Browserbase | Stagehand (open-source framework) no token | Task agent | No docs | None published | no token | Profile |
| ByteDance | Bytespider | Other | No docs | None published | 19.2% | Profile |
| Cloudflare | Cloudflare Browser Rendering no token | Task agent | No statement | Web Bot Auth | no token | Profile |
| Cohere | cohere-ai | Other | No docs | None published | 15.5% | Profile |
| Common Crawl | CCBot | Training | Yes | Reverse DNS | 19.4% | Profile |
| Diffbot | Diffbot | Other | Setting | None published | 15.7% | Profile |
| DuckDuckGo | DuckAssistBot | Live answer fetch | Yes | IP list | 8.1% | Profile |
| DuckDuckGo | DuckDuckBot | Search index | Yes | IP list | not measured | Profile |
| Exa | ExaSearchBot | Search index | No docs | None published | not measured | Profile |
| Firecrawl | FirecrawlAgent | Other | Yes | None published | not measured | Profile |
| Google | Gemini in Chrome (auto browse) no token | Task agent | No docs | None published | no token | Profile |
| Google | Google-Agent no token | Task agent | May not apply | Web Bot Auth | no token | Profile |
| Google | Google-CloudVertexBot | Other | Yes | Reverse DNS | 10.9% | Profile |
| Google | Google-Extended | Training | Yes | None published | 16.0% | Profile |
| Google | Google-GeminiNotebook (was Google-NotebookLM) no token | Live answer fetch | May not apply | IP list | no token | Profile |
| Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile |
| Google | GoogleOther | Other | Yes | Reverse DNS | not measured | Profile |
| Google | Storebot-Google | Search index | Yes | Reverse DNS | not measured | Profile |
| Hive (ImageSift) | ImagesiftBot | Other | Yes | None published | 9.5% | Profile |
| Huawei | PetalBot | Search index | Yes | Reverse DNS | 14.1% | Profile |
| Manus | Manus cloud browser no token | Task agent | No docs | None published | no token | Profile |
| Meta | FacebookBot | Other | Legacy | None published | 10.2% | Profile |
| Meta | facebookexternalhit | Other | May not apply | None published | not measured | Profile |
| Meta | meta-externalads | Other | Yes | None published | not measured | Profile |
| Meta | meta-externalagent | Training | Yes | None published | 16.0% | Profile |
| Meta | meta-externalfetcher | Live answer fetch | May not apply | None published | 8.5% | Profile |
| Meta | meta-webindexer | Search index | Yes | None published | not measured | Profile |
| Meta | Muse (personal agent) no token | Shopping agent | No docs | None published | no token | Profile |
| Microsoft | bingbot | Search index | Yes | Reverse DNS | 3.1% | Profile |
| Microsoft | Browse with Copilot (formerly Copilot Actions) no token | Task agent | No docs | None published | no token | Profile |
| Microsoft | MicrosoftPreview | Other | Yes | Reverse DNS | 4.0% | Profile |
| Mistral AI | MistralAI-Index | Search index | Yes | IP list | not measured | Profile |
| Mistral AI | MistralAI-Training | Training | Yes | None published | not measured | Profile |
| Mistral AI | MistralAI-User | Live answer fetch | Yes | IP list | 7.7% | Profile |
| OpenAI | ChatGPT agent (Cloud browser, formerly Operator) no token | Task agent | No statement | Web Bot Auth | no token | Profile |
| OpenAI | ChatGPT Atlas (browser agent mode) no token | Task agent | No docs | None published | no token | Profile |
| OpenAI | ChatGPT-User | Live answer fetch | May not apply | IP list | 9.7% | Profile |
| OpenAI | GPTBot | Training | Yes | IP list | 18.5% | Profile |
| OpenAI | OAI-AdsBot | Other | No statement | IP list | not measured | Profile |
| OpenAI | OAI-SearchBot | Search index | Yes | IP list | 7.7% | Profile |
| Ora | OraBot | Other | No statement | Web Bot Auth | not measured | Profile |
| Perplexity | Comet (browser assistant) no token | Task agent | No docs | None published | no token | Profile |
| Perplexity | Perplexity Computer no token | Task agent | No docs | None published | no token | Profile |
| Perplexity | Perplexity-User | Live answer fetch | May not apply | IP list | 8.3% | Profile |
| Perplexity | PerplexityBot | Search index | Yes | IP list | 11.2% | Profile |
| Timpi | Timpibot | Search index | No docs | None published | 9.8% | Profile |
| Webz.io | omgili | Other | Yes | None published | 15.7% | Profile |
| xAI | Grok Bot (agent) no token | Task agent | No docs | None published | no token | Profile |
| You.com | YouBot | Search index | No docs | None published | 10.3% | Profile |

## Added on 2026-11-05

-   [**The names sites write in robots.txt**](https://agentreadycrawlers.com/crawlers/named-in-robots-txt/)  
    The user-agent names most often written in the parsed robots.txt files of the crawl, and which have no row in the Index.
-   [**Agentic browsers and how they identify**](https://agentreadycrawlers.com/crawlers/agentic-browsers/)  
    Every task-agent row in the Index: token, user-agent string, verification and what the operator says about robots.txt.

## Sources

-   [Index method (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04
-   [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04
-   [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04

Last reviewed 2026-11-05