--- title: "Search index crawlers and agents | Agent Ready Crawlers" description: "Search index in the Crawler Index: 15 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/search-index site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Search index crawlers and agents | Agent Ready Crawlers > Search index in the Crawler Index: 15 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/search-index Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Search index: 15 rows In short 15 rows in the Index are filed under search index, from 14 operators. 4 publish nothing a site can verify and 0 have no robots.txt token. Of the 9 tokens the crawl measured, Applebot is refused by 14.8 percent of top-10,000 sites and Googlebot by 2.8 percent. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Builds a search index that sends visitors back. Refusing it removes pages from that product. Search index: 15 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | OAI-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Anthropic | Claude-SearchBot | Search index | Yes | IP list | 7.7% | Profile | | Perplexity | PerplexityBot | Search index | Yes | IP list | 11.2% | Profile | | Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile | | Google | Storebot-Google | Search index | Yes | Reverse DNS | not measured | Profile | | Apple | Applebot | Search index | Yes | Reverse DNS | 14.8% | Profile | | Microsoft | bingbot | Search index | Yes | Reverse DNS | 3.1% | Profile | | Amazon | Amzn-SearchBot | Search index | Yes | IP list | not measured | Profile | | Meta | meta-webindexer | Search index | Yes | None published | not measured | Profile | | DuckDuckGo | DuckDuckBot | Search index | Yes | IP list | not measured | Profile | | Mistral AI | MistralAI-Index | Search index | Yes | IP list | not measured | Profile | | You.com | YouBot | Search index | No docs | None published | 10.3% | Profile | | Timpi | Timpibot | Search index | No docs | None published | 9.8% | Profile | | Huawei | PetalBot | Search index | Yes | Reverse DNS | 14.1% | Profile | | Exa | ExaSearchBot | Search index | No docs | None published | not measured | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Applebot | 14.8% | 11.6% | 3.1% | 11.1% | | PetalBot | 14.1% | 10.3% | 3.8% | 10.9% | | PerplexityBot | 11.2% | 7.9% | 3.4% | 7.9% | | YouBot | 10.3% | 6.4% | 3.8% | 7.3% | | Timpibot | 9.8% | 5.8% | 4.0% | 7.0% | | OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5.4% | | Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5.5% | | Bingbot | 3.1% | 0.4% | 2.7% | 2.2% | | Googlebot | 2.8% | 0.5% | 2.3% | 2.1% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 Last reviewed 2026-11-04