Other: 15 rows
In short
15 rows in the Index are filed under other, from 12 operators. 10 publish nothing a site can verify and 0 have no robots.txt token. Of the 9 tokens the crawl measured, Bytespider is refused by 19.2 percent of top-10,000 sites and MicrosoftPreview by 4.0 percent.
Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.
| Operator | Token | Block rate |
|---|---|---|
| OpenAI | OAI-AdsBot | |
| Google-CloudVertexBot | 10.9% | |
| GoogleOther | ||
| Microsoft | MicrosoftPreview | 4.0% |
| Meta | meta-externalads | |
| Meta | facebookexternalhit | |
| Meta | FacebookBot | 10.2% |
| ByteDance | Bytespider | 19.2% |
| Cohere | cohere-ai | 15.5% |
| Diffbot | Diffbot | 15.7% |
| Webz.io | omgili | 15.7% |
| Hive (ImageSift) | ImagesiftBot | 9.5% |
| Firecrawl | FirecrawlAgent | |
| Ora | OraBot | |
| Agent Ready Crawlers | agent-readiness-crawl | 4.2% |
Block rates in this purpose
Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via *" counts sites that refuse it only because their wildcard group refuses everything.
| Token | Refused | Named | Via * | All crawled |
|---|---|---|---|---|
| Bytespider | 19.2% | 15.4% | 3.8% | 15.0% |
| Diffbot | 15.7% | 11.9% | 3.8% | 11.8% |
| omgili | 15.7% | 11.8% | 3.9% | 11.8% |
| cohere-ai | 15.5% | 11.6% | 3.8% | 11.7% |
| Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 8.3% |
| FacebookBot | 10.2% | 6.4% | 3.7% | 7.3% |
| ImagesiftBot | 9.5% | 5.5% | 3.9% | 6.9% |
| agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 3.0% |
| MicrosoftPreview | 4.0% | 0.0% | 3.9% | 2.9% |
Sources
Last reviewed 2026-11-04