Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

73 companies in four types

Vendors

In short

73 companies are profiled in four types: 31 crawler operators, 23 bot management vendors, 11 agent verification providers and 8 licensing intermediaries. Each profile holds sourced facts, a standards table, dated prices where a vendor publishes them and what the vendor does not do. Read 2026-11-05: 671 facts from 331 sources.

This list profiles 73 companies that sit between a crawler and a page: the operators that run crawlers and agents, the vendors that decide at the door whether they get in, the providers that let a site check who an agent is, and the intermediaries that turn access into a licence or a price.

Crawler operators are every company behind a row of the Crawler Index (27 of them), plus 4 browser-agent and search-API companies whose own pages say how their traffic identifies itself. 15 of the vendors keep the ids and rubric pages they had before; their profiles now carry the same sections as the rest. A company is listed only when its own pages state what it does for this layer. Where a vendor's site refused the crawl's user agent, the profile cites a wire reprint, trade press or the vendor's developer documentation and says so; 12 profiles carry at least one third-party source.

671 facts come from 331 pages read on 2026-11-05, each with its source in the download and on its profile. A sentence that repeats one printed on another profile, or in a graded vendor's rubric text, appears once; 68 such sentences are kept in the download only. 50 vendors carry a dated pricing entry, and 34 of them state an amount. 32 profiles carry our own probes of key directories and IP files. Every homepage was fetched once on 2026-11-05 with its status recorded: 68 answered 200; Amazon answered 202, OpenAI answered 403, xAI answered 403, Akamai answered 403 and DataDome answered 403.

15 of these vendors are also graded on a five-question rubric. The grades and the rubric are on their own page.

Standards are counted when the vendor's own pages state support, and listed by id: rfc9309 is RFC 9309 (robots.txt); web-bot-auth is Web Bot Auth; content-signals is Content Signals; ip-list is Published IP list; reverse-dns is Reverse DNS check; cloudflare-verified-bots is Cloudflare verified bots; visa-tap is Visa Trusted Agent Protocol; mastercard-agent-pay is Mastercard Agent Pay. Rows are alphabetical inside each type.

Crawler operators

The companies behind the crawlers and agents in the Crawler Index, plus browser-agent and search-API companies whose own pages say how their traffic identifies itself. The 31 crawler operators on a page of their own.

Vendor Type What it does for this layer Standards supported Pricing model Our check Last read
Amazon Crawler operator Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent. 1 ip-list
partial: rfc9309		 No price recorded Homepage HTTP 202, 4 probes 2026-11-05 2026-11-05
Anchor Browser Crawler operator Cloud browser infrastructure for computer-use agents that can sign sessions with Cloudflare Web Bot Auth on its upper plans, and also sells stealth fingerprinting, proxies, CAPTCHA bypass and a pay-on-success web unlocker. 2 web-bot-auth, cloudflare-verified-bots subscription plus credits Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Anthropic Crawler operator Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. 2 rfc9309, ip-list No price recorded Homepage HTTP 200, 3 probes 2026-11-05 2026-11-05
Apify Crawler operator Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. 0
partial: rfc9309, web-bot-auth, cloudflare-verified-bots		 platform usage; subscription plus usage Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Apple Crawler operator Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file. 3 rfc9309, ip-list, reverse-dns No price recorded Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Block Crawler operator Company that built goose, an open-source local AI agent Cloudflare classed as a signed agent in 2025; Block has since donated goose to the Linux Foundation's Agentic AI Foundation and publishes no key directory for it. 0 No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Browser Use Crawler operator Maker of the open-source browser-use agent library and a hosted cloud browser for agents that ships with stealth fingerprinting, residential proxies and CAPTCHA solving on by default, and no published identifier for site owners. 0 usage-based Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Browserbase Crawler operator Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned. 2 web-bot-auth, cloudflare-verified-bots
partial: rfc9309		 subscription plus usage Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Cohere Crawler operator Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page. 1 rfc9309 No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Common Crawl Crawler operator Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests. 3 rfc9309, ip-list, reverse-dns No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Diffbot Crawler operator Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. 1 rfc9309 subscription with credits; credits Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
DuckDuckGo Crawler operator DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models. 2 rfc9309, ip-list No price recorded Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Exa Crawler operator AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities. 2 rfc9309, web-bot-auth pay as you go Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Firecrawl Crawler operator Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends. 1 rfc9309 credit subscription Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Google Crawler operator Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic. 2 ip-list, reverse-dns
partial: rfc9309, web-bot-auth		 No price recorded Homepage HTTP 200, 5 probes 2026-11-05 2026-11-05
Hive Crawler operator AI model company, renamed from Hive to Deploy in October 2026, whose ImageSift unit runs ImagesiftBot to collect public images and page text for a reverse image search index. 1 rfc9309 usage-based Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Huawei Crawler operator Huawei runs PetalBot for the Petal search engine, whose index also feeds Huawei Assistant and AI Search; PetalBot follows robots.txt and is verified by reverse DNS under aspiegel.com or an IP lookup tool. 2 rfc9309, reverse-dns No price recorded Homepage HTTP 200 2026-11-05 2026-11-05
Hyperbrowser Crawler operator Cloud browser platform for AI agents whose sessions can sign requests with Web Bot Auth under the fixed Signature-Agent www.hyperbrowser.ai, on request per team, alongside stealth, proxies and CAPTCHA solving. 1 web-bot-auth credits Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Kernel Crawler operator Cloud browser company for AI agents that runs two signed identities, Kernel Agent for user-directed browsing and KernelSearchBot for search crawling, each with its own Web Bot Auth key directory so sites can treat them separately. 3 rfc9309, web-bot-auth, cloudflare-verified-bots subscription plus usage Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Manus Crawler operator General-purpose AI agent, owned by Meta since December 2025, that browses either in an isolated cloud browser or through an extension inside the user's own browser, and publishes no identifier, robots.txt policy or signing keys for site owners. 0 No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Meta Crawler operator Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt. 0
partial: rfc9309		 No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Microsoft Crawler operator Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding. 2 rfc9309, ip-list No price recorded Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Mistral AI Crawler operator Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two. 1 rfc9309
partial: ip-list		 No price recorded Homepage HTTP 200, 3 probes 2026-11-05 2026-11-05
Notte Crawler operator Browser platform for AI agents (sessions, agents and serverless browser functions) that says it is registered as a verified bot with Cloudflare and signs session requests with Web Bot Auth when a customer turns it on. 2 web-bot-auth, cloudflare-verified-bots subscription plus usage Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
OpenAI Crawler operator OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. 2 web-bot-auth, ip-list
partial: rfc9309		 No price recorded Homepage HTTP 403, 5 probes 2026-11-05 2026-11-05
Parallel Web Systems Crawler operator Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only. 2 rfc9309, web-bot-auth
partial: ip-list		 pay per request Homepage HTTP 200, 3 probes 2026-11-05 2026-11-05
Perplexity Crawler operator Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF. 1 ip-list
partial: rfc9309		 No price recorded Homepage HTTP 200, 3 probes 2026-11-05 2026-11-05
Timpi Crawler operator Operator of an independent search index built by a decentralised network of contributor nodes; its crawler appears in robots.txt files as Timpibot, but Timpi publishes no crawler documentation for site owners. 0 No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
Webz.io Crawler operator Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training. 1 rfc9309 pay as you go Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
xAI Crawler operator xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic. 0 No price recorded Homepage HTTP 403, 1 probe 2026-11-05 2026-11-05
You.com Crawler operator You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24. 3 rfc9309, web-bot-auth, reverse-dns
partial: ip-list		 No price recorded Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05

Bot management vendors

CDN, WAF, CAPTCHA and bot-detection vendors that decide at the door whether a crawler or agent gets in, and whether it pays. The 23 bot management vendors on a page of their own.

Vendor Type What it does for this layer Standards supported Pricing model Our check Last read
Akamai Bot management CDN and security company whose Bot & Agent Control solutions judge AI agents at the edge, take identity from Visa's Trusted Agent Protocol and Know Your Agent tokens, and send AI traffic to TollBit or Skyfire for paid access. 2 web-bot-auth, visa-tap sales-led Homepage HTTP 403 2026-11-05 2026-11-05
AWS WAF Bot Control Bot management Cloud provider whose WAF Bot Control labels and verifies bots, checks Web Bot Auth signatures and can charge AI bots with HTTP 402, and whose AgentCore Browser signs its own agent traffic with Web Bot Auth. 2 web-bot-auth, ip-list per web ACL plus per request Homepage HTTP 200 2026-11-05 2026-11-05
Arcjet Bot management Developer security SDK for JavaScript, Python and Go that detects bots inside application code, lets developers allow or deny named bots or categories such as CATEGORY:AI, and adds runtime guards for agent frameworks. 2 ip-list, reverse-dns monthly plan plus usage Homepage HTTP 200 2026-11-05 2026-11-05
Castle Bot management Bot and account abuse API whose risk responses include an agent object that names AI agents and browser automation, identified by a verified Web Bot Auth signature, a browser fingerprint or a user-agent string. 1 web-bot-auth usage-based plans Homepage HTTP 200 2026-11-05 2026-11-05
Centinel Analytica Bot management Bot and AI crawler protection vendor whose edge integration verifies declared search and AI crawlers against operators' published IP ranges and detects scrapers that run real browsers behind residential proxies. 1 ip-list quoted per deployment Homepage HTTP 200 2026-11-05 2026-11-05
Cequence Security Bot management Application and API protection vendor whose Bot Management includes Agent Trust for personal AI shopping agents, and whose AI Gateway governs enterprise agents' tool calls behind OAuth 2.1 authentication. 0 sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Cloudflare Bot management Connectivity cloud whose AI Crawl Control, Bot Management and verified bot program sort crawlers by purpose, verify them by IP list or Web Bot Auth signature, and let site owners charge crawlers per request in a private beta. 5 rfc9309, web-bot-auth, content-signals, ip-list, cloudflare-verified-bots included in every plan; site owner sets one per-request price; 1 more Homepage HTTP 200, 1 probe 2026-11-05 2026-11-05
DataDome Bot management Bot and fraud protection vendor whose Agentic Trust layer names each AI agent, rates how strong that identification is, applies per-agent and per-endpoint policies, and hands chosen agents to TollBit for paid access. 3 web-bot-auth, ip-list, reverse-dns monthly, by request volume Homepage HTTP 403 2026-11-05 2026-11-05
F5 Bot management Application delivery and security company whose Distributed Cloud Bot Defense puts humans, trusted AI agents and malicious bots under one policy, accepts Web Bot Auth and Skyfire's Know Your Agent tokens, and runs on F5's platform, BIG-IP or public cloud. 1 web-bot-auth sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Fastly Bot management Edge cloud vendor whose Bot Management adds edge detection (ContentGuard), TLS fingerprinting, client challenges and a verified bots signal, with separate AI crawler and AI fetcher signals for training bots and user-triggered fetchers. 0 contact sales Homepage HTTP 200 2026-11-05 2026-11-05
Fingerprint Bot management Device intelligence vendor whose Bot Detection and Authorized AI Agent Detection signals separate good bots, bad bots and AI agents, and verify signed agents from providers such as OpenAI, AWS AgentCore and Browserbase. 1 web-bot-auth monthly plan plus usage Homepage HTTP 200 2026-11-05 2026-11-05
Google Cloud Fraud Defense Bot management Google Cloud's bot and fraud service, the evolution of reCAPTCHA, which verifies AI agents by Web Bot Auth signature or by user agent matched to published IP ranges or reverse DNS, and reports verified and suspected agent sessions. 3 web-bot-auth, ip-list, reverse-dns per assessment Homepage HTTP 200 2026-11-05 2026-11-05
hCaptcha Bot management Bot detection and CAPTCHA vendor whose Enterprise Agent Controls identify AI agents, verify partner platforms that sign with Web Bot Auth, and let sites allow browsing while requiring verification before an agent reserves inventory. 1 web-bot-auth contact sales; monthly plus usage Homepage HTTP 200 2026-11-05 2026-11-05
HUMAN Security Bot management Bot and fraud defense vendor whose AgenticTrust module in Sightline names AI agents, grades them High, Medium or Low trust, verifies signed agents, and lets sites set what each agent may do from browsing to checkout. 1 web-bot-auth sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Imperva Bot management Application security vendor owned by Thales whose Advanced Bot Protection sorts human, good bot and bad bot traffic across web, mobile and API; verified-agent controls so far ship in a controlled-availability Google Cloud offering. 0 sales-led, free trial Homepage HTTP 200 2026-11-05 2026-11-05
Kasada Bot management Bot defense vendor founded in Sydney whose AI Agent Trust adds a verified agent directory with Web Bot Auth support and endpoint-level access rules, enforced at the edge before automated traffic reaches the application. 1 web-bot-auth sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Known Agents Bot management Agent analytics and access-control service, formerly Dark Visitors, that identifies AI agents and crawlers from server-side data, keeps an automatic robots.txt current, and blocks bots that ignore it or spoof trusted identities. 1 rfc9309 monthly by events Homepage HTTP 200 2026-11-05 2026-11-05
Link11 Bot management German security provider whose WAAP platform joins WAF, API security, Bot Management and Web DDoS protection, with a setting to monitor or block AI crawlers; the reblaze.com domain now redirects to Link11. 0 sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Netacea Bot management Server-side bot protection vendor that integrates at the CDN or load balancer without client code, classifies humans, good bots, bad bots and AI agents, and enforces access by intent rather than declared identity. 0 sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Netlify Bot management Web hosting platform that blocks chosen AI crawlers by user-agent string through an Edge Function extension and offers IP, geography and header-based traffic rules and rate limiting; it does not verify bot identity. 0 plan-based Homepage HTTP 200 2026-11-05 2026-11-05
Radware Bot management Application security vendor whose Bot Manager and Agent Trust Management identify Web Bot Auth-compliant and non-compliant agents, capture an agent's declared intent in conversation, and restrict actions such as login or checkout. 1 web-bot-auth sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Vercel Bot management Hosting platform whose firewall verifies known bots by IP, reverse DNS and Web Bot Auth, offers an AI Bots managed ruleset that treats AI crawlers and AI assistants separately, and sells BotID checks for high-value routes. 3 web-bot-auth, ip-list, reverse-dns included; per call Homepage HTTP 200 2026-11-05 2026-11-05
Verisoul Bot management Fake account and fraud detection platform whose Agent & Bot Detection names AI agents by signature, self-declared user agent or inference, and whose public Agent Atlas documents how each agent shows up on a site. 1 web-bot-auth sales-led, 30-day trial Homepage HTTP 200 2026-11-05 2026-11-05

Verification providers

Web Bot Auth implementers, card-network agent programmes and agent identity products that let a site check who an agent is and who sent it. The 11 verification providers on a page of their own.

Vendor Type What it does for this layer Standards supported Pricing model Our check Last read
American Express Verification provider Card network whose ACE developer kit plans an agent registration service so only verified AI agents get tokenized Amex credentials, with Agent Purchase Protection for registered agent purchases; the registration spec is still in development. 0 No price recorded Homepage HTTP 200 2026-11-05 2026-11-05
Auth0 for AI Agents Verification provider Okta's customer identity platform; its AI Agents add-on gives agents a company builds a user-bound identity, scoped tokens, a Token Vault for third-party APIs and Cross App Access, and it verifies no inbound crawlers. 0 add-on, 50% of the base plan price Homepage HTTP 200 2026-11-05 2026-11-05
Descope Verification provider Customer and agent identity platform whose Agentic Identity Hub gives AI agents and MCP clients OAuth 2.1 access, short-lived scoped credentials and an agent directory with policies, for agents calling a company's APIs rather than crawling its site. 0 free tier, then monthly plans Homepage HTTP 200 2026-11-05 2026-11-05
Mastercard Verification provider Card network whose Agent Pay registers AI agents and has them sign requests with Web Bot Auth under the tag Agent-pay-auth, so a merchant or its CDN can check the agent's key in a network directory before letting it buy. 2 web-bot-auth, mastercard-agent-pay
partial: visa-tap		 No price recorded Homepage HTTP 200, 2 probes 2026-11-05 2026-11-05
Microsoft Entra Agent ID Verification provider Microsoft's identity service for AI agents inside an Entra tenant: each agent gets its own service principal created from a blueprint, with tokens, a human sponsor and audit, for agents an organisation runs rather than agents visiting a website. 0 No price recorded Homepage HTTP 200 2026-11-05 2026-11-05
Prove Verification provider Identity verification company whose Verified Agent product binds an AI agent to a verified person or business with identity-bound tokens, checks participants against a shared trust registry and logs co-signed consent for agent purchases. 0 sales-led Homepage HTTP 200 2026-11-05 2026-11-05
Stytch IsAgent Verification provider Identity platform, part of Twilio since November 2025, that verifies Web Bot Auth signatures through its IsAgent SDK and Device Fingerprinting, and issues scoped OAuth tokens to AI agents and MCP clients through Connected Apps. 1 web-bot-auth free tier, then contact sales; per fingerprint above the included volume Homepage HTTP 200 2026-11-05 2026-11-05
Trulioo Verification provider Identity verification company whose Know Your Agent programme, in alpha, runs business and AML checks on the company behind an AI agent, binds the agent's fingerprint to it in a signed Digital Agent Profile and lets counterparties check it by API or MCP. 0 alpha, access on request Homepage HTTP 200 2026-11-05 2026-11-05
Visa Verification provider Card network whose Trusted Agent Protocol lets a merchant or its CDN verify, from RFC 9421 signatures tagged agent-browser-auth or agent-payer-auth, that a visiting agent is approved under Visa Intelligent Commerce and acting for a known consumer. 2 web-bot-auth, visa-tap No price recorded Homepage HTTP 200 2026-11-05 2026-11-05
Vouched Verification provider Identity verification company whose Know Your Agent suite detects AI agents on a site by behaviour, enforces allow, challenge, block or redirect rules, and ties each agent action to a verified person through signed delegations. 0 free; monthly fee plus per transaction Homepage HTTP 200 2026-11-05 2026-11-05
World Verification provider Proof-of-human network whose AgentKit lets a person with a verified World ID register an agent's wallet, so x402-protected sites can tell a human-backed agent from a script and limit access per unique person. 0 developer preview Homepage HTTP 200 2026-11-05 2026-11-05

Licensing intermediaries

Marketplaces, standards bodies and paywalls that turn crawler access into a licence or a price. The 8 licensing intermediaries on a page of their own.

Vendor Type What it does for this layer Standards supported Pricing model Our check Last read
Dappier Licensing intermediary Content marketplace where publishers list their content as priced RAG models and AI agents buy query access through an agent storefront that takes x402, MPP or card payments without signup. 0
partial: rfc9309		 publisher-set price per query Homepage HTTP 200 2026-11-05 2026-11-05
FetchRight Licensing intermediary Licensing service behind the open Peek-Then-Pay protocol: a site publishes /.well-known/peek.json, AI agents get a priced HTTP 203 preview, buy a JWT license per intent and present it to an edge enforcer. 0 open specification; sales-led Homepage HTTP 200 2026-11-05 2026-11-05
IAB Tech Lab Licensing intermediary Ad-industry standards body whose Content Monetization Protocols (CoMP) 1.0 define how a licensed AI crawler asks a publisher for content and gets a token, alongside a paid Spiders and Bots List and a free Agent Registry. 3 rfc9309, web-bot-auth, reverse-dns
partial: ip-list		 open specification; annual subscription; 1 more Homepage HTTP 200 2026-11-05 2026-11-05
ProRata Licensing intermediary AI attribution company that splits an AI answer into claims and measures each source's share as a basis for licensing and payment, and runs Gist Answers on a network of licensed publisher titles. 0 No price recorded Homepage HTTP 200 2026-11-05 2026-11-05
RSL Collective Licensing intermediary Nonprofit collective rights organisation and co-developer of the RSL standard, a machine-readable license file a site links from robots.txt so crawlers can see, buy and prove a license for training, AI input, search or crawling. 1 rfc9309 open standard; free to join for publishers Homepage HTTP 200 2026-11-05 2026-11-05
Skyfire Licensing intermediary Agent identity and payment network whose signed KYA and pay tokens let a website, API or bot manager admit a verified agent and charge it per request, with Skyfire vetting the agent platform and the person or business behind the agent. 0 subscription for verification (Individual or Organization) Homepage HTTP 200 2026-11-05 2026-11-05
Supertab Licensing intermediary Managed RSL licensing service: Supertab Connect hosts a publisher's license.xml and RSL license server, issues license tokens to crawlers, checks them at the CDN with its SDK and bills the use, priced from USD 99 a month. 1 rfc9309 monthly subscription by visitors and licensees Homepage HTTP 200 2026-11-05 2026-11-05
TollBit Licensing intermediary Licensing marketplace that sends AI bots, by user agent at the publisher's CDN, to an Agent Site on a tollbit subdomain, where bots holding a short-lived TollBit JWT get licensed content at publisher-set rates per 1,000 pages. 0
partial: rfc9309		 free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee Homepage HTTP 200 2026-11-05 2026-11-05

Not listed, and why

The research pass looked at these companies and did not profile them. A name taken from a roundup with no page of its own is left out the same way.

Download the list

One row per vendor: type, operator, headquarters, products with their prices, the standards assessed, our probes, what the vendor does not do, and every sourced fact in the JSON file. Edition 2026-11-05.

Cite this. Agent Ready Crawlers, "Vendors", edition 2026-11-05, https://agentreadycrawlers.com/vendors/. Free to use with this line; see the terms.

Sources

Last reviewed 2026-11-05