Search index: 15 rows
In short
15 rows in the Index are filed under search index, from 14 operators. 4 publish nothing a site can verify and 0 have no robots.txt token. Of the 9 tokens the crawl measured, Applebot is refused by 14.8 percent of top-10,000 sites and Googlebot by 2.8 percent.
Builds a search index that sends visitors back. Refusing it removes pages from that product.
| Operator | Token | Block rate |
|---|---|---|
| OpenAI | OAI-SearchBot | 7.7% |
| Anthropic | Claude-SearchBot | 7.7% |
| Perplexity | PerplexityBot | 11.2% |
| Googlebot | 2.8% | |
| Storebot-Google | ||
| Apple | Applebot | 14.8% |
| Microsoft | bingbot | 3.1% |
| Amazon | Amzn-SearchBot | |
| Meta | meta-webindexer | |
| DuckDuckGo | DuckDuckBot | |
| Mistral AI | MistralAI-Index | |
| You.com | YouBot | 10.3% |
| Timpi | Timpibot | 9.8% |
| Huawei | PetalBot | 14.1% |
| Exa | ExaSearchBot |
Block rates in this purpose
Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via *" counts sites that refuse it only because their wildcard group refuses everything.
| Token | Refused | Named | Via * | All crawled |
|---|---|---|---|---|
| Applebot | 14.8% | 11.6% | 3.1% | 11.1% |
| PetalBot | 14.1% | 10.3% | 3.8% | 10.9% |
| PerplexityBot | 11.2% | 7.9% | 3.4% | 7.9% |
| YouBot | 10.3% | 6.4% | 3.8% | 7.3% |
| Timpibot | 9.8% | 5.8% | 4.0% | 7.0% |
| OAI-SearchBot | 7.7% | 4.4% | 3.3% | 5.4% |
| Claude-SearchBot | 7.7% | 4.1% | 3.6% | 5.5% |
| Bingbot | 3.1% | 0.4% | 2.7% | 2.2% |
| Googlebot | 2.8% | 0.5% | 2.3% | 2.1% |
Sources
Last reviewed 2026-11-04