--- title: "Other crawlers and agents | Agent Ready Crawlers" description: "Other in the Crawler Index: 15 AI crawlers and agents from 12 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/other site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Other crawlers and agents | Agent Ready Crawlers > Other in the Crawler Index: 15 AI crawlers and agents from 12 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/other Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Other: 15 rows In short 15 rows in the Index are filed under other, from 12 operators. 10 publish nothing a site can verify and 0 have no robots.txt token. Of the 9 tokens the crawl measured, Bytespider is refused by 19.2 percent of top-10,000 sites and MicrosoftPreview by 4.0 percent. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. Other: 15 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | OAI-AdsBot | Other | No statement | IP list | not measured | Profile | | Google | Google-CloudVertexBot | Other | Yes | Reverse DNS | 10.9% | Profile | | Google | GoogleOther | Other | Yes | Reverse DNS | not measured | Profile | | Microsoft | MicrosoftPreview | Other | Yes | Reverse DNS | 4.0% | Profile | | Meta | meta-externalads | Other | Yes | None published | not measured | Profile | | Meta | facebookexternalhit | Other | May not apply | None published | not measured | Profile | | Meta | FacebookBot | Other | Legacy | None published | 10.2% | Profile | | ByteDance | Bytespider | Other | No docs | None published | 19.2% | Profile | | Cohere | cohere-ai | Other | No docs | None published | 15.5% | Profile | | Diffbot | Diffbot | Other | Setting | None published | 15.7% | Profile | | Webz.io | omgili | Other | Yes | None published | 15.7% | Profile | | Hive (ImageSift) | ImagesiftBot | Other | Yes | None published | 9.5% | Profile | | Firecrawl | FirecrawlAgent | Other | Yes | None published | not measured | Profile | | Ora | OraBot | Other | No statement | Web Bot Auth | not measured | Profile | | Agent Ready Crawlers | agent-readiness-crawl | Other | Partly | None published | 4.2% | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | Bytespider | 19.2% | 15.4% | 3.8% | 15.0% | | Diffbot | 15.7% | 11.9% | 3.8% | 11.8% | | omgili | 15.7% | 11.8% | 3.9% | 11.8% | | cohere-ai | 15.5% | 11.6% | 3.8% | 11.7% | | Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 8.3% | | FacebookBot | 10.2% | 6.4% | 3.7% | 7.3% | | ImagesiftBot | 9.5% | 5.5% | 3.9% | 6.9% | | agent-readiness-crawl | 4.2% | 0.0% | 4.1% | 3.0% | | MicrosoftPreview | 4.0% | 0.0% | 3.9% | 2.9% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Which Crawlers Does Bing Use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04