--- title: "Crawler operators: 31 vendor profiles | Agent Ready Crawlers" description: "Crawler operators: 31 profiles with standards, dated prices, sources and what each does not do. 16 of 31 document support for RFC 9309. Read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/by-type/crawler-operator site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Crawler operators: 31 vendor profiles | Agent Ready Crawlers > Crawler operators: 31 profiles with standards, dated prices, sources and what each does not do. 16 of 31 document support for RFC 9309. Read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/by-type/crawler-operator Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Vendors by type # Crawler operators In short 31 crawler operators are profiled, from OpenAI and Google to browser-agent companies such as Kernel and Notte. 16 of 31 document support for RFC 9309, 9 for Web Bot Auth and 9 for published IP lists. A site owner can read down the standards column to see which of them publish a signature, an IP file or neither. The 31 crawler operators run alphabetically from Amazon to You.com, and carry 298 sourced facts between them. The price models on record include subscription plus credits; platform usage; subscription plus usage; usage-based; subscription with credits; and 4 more. 30 profiles carry our own probes of key directories or IP files. Each profile lists what the vendor does not do next to what it does. - [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) - [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/) - [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/) - [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/) - [All 73](https://agentreadycrawlers.com/vendors/) | Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read | | --- | --- | --- | --- | --- | --- | --- | | Amazon | Crawler operator | Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 202, 4 probes 2026-11-05 | 2026-11-05 | | Anchor Browser | Crawler operator | Cloud browser infrastructure for computer-use agents that can sign sessions with Cloudflare Web Bot Auth on its upper plans, and also sells stealth fingerprinting, proxies, CAPTCHA bypass and a pay-on-success web unlocker. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Anthropic | Crawler operator | Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Apify | Crawler operator | Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. | 0; partial: rfc9309, web-bot-auth, cloudflare-verified-bots | platform usage; subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Apple | Crawler operator | Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Block | Crawler operator | Company that built goose, an open-source local AI agent Cloudflare classed as a signed agent in 2025; Block has since donated goose to the Linux Foundation's Agentic AI Foundation and publishes no key directory for it. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Browser Use | Crawler operator | Maker of the open-source browser-use agent library and a hosted cloud browser for agents that ships with stealth fingerprinting, residential proxies and CAPTCHA solving on by default, and no published identifier for site owners. | 0 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Browserbase | Crawler operator | Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned. | 2 web-bot-auth, cloudflare-verified-bots; partial: rfc9309 | subscription plus usage | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Cohere | Crawler operator | Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page. | 1 rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Common Crawl | Crawler operator | Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Diffbot | Crawler operator | Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. | 1 rfc9309 | subscription with credits; credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | DuckDuckGo | Crawler operator | DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Exa | Crawler operator | AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities. | 2 rfc9309, web-bot-auth | pay as you go | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Firecrawl | Crawler operator | Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends. | 1 rfc9309 | credit subscription | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Google | Crawler operator | Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic. | 2 ip-list, reverse-dns; partial: rfc9309, web-bot-auth | No price recorded | Homepage HTTP 200, 5 probes 2026-11-05 | 2026-11-05 | | Hive | Crawler operator | AI model company, renamed from Hive to Deploy in October 2026, whose ImageSift unit runs ImagesiftBot to collect public images and page text for a reverse image search index. | 1 rfc9309 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Huawei | Crawler operator | Huawei runs PetalBot for the Petal search engine, whose index also feeds Huawei Assistant and AI Search; PetalBot follows robots.txt and is verified by reverse DNS under aspiegel.com or an IP lookup tool. | 2 rfc9309, reverse-dns | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 | | Hyperbrowser | Crawler operator | Cloud browser platform for AI agents whose sessions can sign requests with Web Bot Auth under the fixed Signature-Agent www.hyperbrowser.ai, on request per team, alongside stealth, proxies and CAPTCHA solving. | 1 web-bot-auth | credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Kernel | Crawler operator | Cloud browser company for AI agents that runs two signed identities, Kernel Agent for user-directed browsing and KernelSearchBot for search crawling, each with its own Web Bot Auth key directory so sites can treat them separately. | 3 rfc9309, web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Manus | Crawler operator | General-purpose AI agent, owned by Meta since December 2025, that browses either in an isolated cloud browser or through an extension inside the user's own browser, and publishes no identifier, robots.txt policy or signing keys for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Meta | Crawler operator | Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt. | 0; partial: rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Microsoft | Crawler operator | Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | Mistral AI | Crawler operator | Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two. | 1 rfc9309; partial: ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Notte | Crawler operator | Browser platform for AI agents (sessions, agents and serverless browser functions) that says it is registered as a verified bot with Cloudflare and signs session requests with Web Bot Auth when a customer turns it on. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 | | OpenAI | Crawler operator | OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. | 2 web-bot-auth, ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 403, 5 probes 2026-11-05 | 2026-11-05 | | Parallel Web Systems | Crawler operator | Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only. | 2 rfc9309, web-bot-auth; partial: ip-list | pay per request | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Perplexity | Crawler operator | Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 | | Timpi | Crawler operator | Operator of an independent search index built by a decentralised network of contributor nodes; its crawler appears in robots.txt files as Timpibot, but Timpi publishes no crawler documentation for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | Webz.io | Crawler operator | Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training. | 1 rfc9309 | pay as you go | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | | xAI | Crawler operator | xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic. | 0 | No price recorded | Homepage HTTP 403, 1 probe 2026-11-05 | 2026-11-05 | | You.com | Crawler operator | You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 | ## Sources - [The full vendor list (this site)](https://agentreadycrawlers.com/vendors/) read 2026-11-04 - [Amazon: amazon.com](https://www.amazon.com) read 2026-11-05 - [Anchor Browser: anchorbrowser.io](https://anchorbrowser.io) read 2026-11-05 - [Anthropic: anthropic.com](https://www.anthropic.com) read 2026-11-05 - [Apify: apify.com](https://apify.com) read 2026-11-05 - [Apple: apple.com](https://www.apple.com) read 2026-11-05 - [Block: block.xyz](https://block.xyz) read 2026-11-05 - [Browser Use: browser-use.com](https://browser-use.com) read 2026-11-05 - [Browserbase: browserbase.com](https://www.browserbase.com) read 2026-11-05 - [Cohere: cohere.com](https://cohere.com) read 2026-11-05 - [Common Crawl: commoncrawl.org](https://commoncrawl.org) read 2026-11-05 - [Diffbot: diffbot.com](https://www.diffbot.com) read 2026-11-05 - [DuckDuckGo: duckduckgo.com](https://duckduckgo.com) read 2026-11-05 - [Exa: exa.ai](https://exa.ai) read 2026-11-05 - [Firecrawl: firecrawl.dev](https://www.firecrawl.dev) read 2026-11-05 - [Google: google.com](https://www.google.com) read 2026-11-05 - [Hive: deploy.com](https://deploy.com) read 2026-11-05 - [Huawei: huawei.com](https://www.huawei.com) read 2026-11-05 - [Hyperbrowser: hyperbrowser.ai](https://www.hyperbrowser.ai) read 2026-11-05 - [Kernel: kernel.sh](https://www.kernel.sh) read 2026-11-05 - [Manus: manus.im](https://manus.im) read 2026-11-05 - [Meta: meta.com](https://www.meta.com) read 2026-11-05 - [Microsoft: microsoft.com](https://www.microsoft.com) read 2026-11-05 - [Mistral AI: mistral.ai](https://mistral.ai) read 2026-11-05 - [Notte: notte.cc](https://www.notte.cc) read 2026-11-05 - [OpenAI: openai.com](https://openai.com) read 2026-11-05 - [Parallel Web Systems: parallel.ai](https://parallel.ai) read 2026-11-05 - [Perplexity: perplexity.ai](https://www.perplexity.ai) read 2026-11-05 - [Timpi: timpi.io](https://timpi.io) read 2026-11-05 - [Webz.io: webz.io](https://webz.io) read 2026-11-05 - [xAI: x.ai](https://x.ai) read 2026-11-05 - [You.com: you.com](https://you.com) read 2026-11-05 Last reviewed 2026-11-05