---
title: "Vendors: operators, bot management, licensing | Agent Ready Crawlers"
description: "The full vendor list: 73 crawler operators, bot management, verification and licensing vendors with standards, dated prices and sources. Read 2026-11-05."
canonical: https://agentreadycrawlers.com/vendors
site: "Agent Ready Crawlers"
last_updated: 2026-11-05
date: 2026-11-05
section: "Vendors"
---

# Vendors: operators, bot management, licensing | Agent Ready Crawlers

> The full vendor list: 73 crawler operators, bot management, verification and licensing vendors with standards, dated prices and sources. Read 2026-11-05.

Source: https://agentreadycrawlers.com/vendors
Site: Agent Ready Crawlers
Last reviewed: 2026-11-05

73 companies in four types

# Vendors

In short

73 companies are profiled in four types: 31 crawler operators, 23 bot management vendors, 11 agent verification providers and 8 licensing intermediaries. Each profile holds sourced facts, a standards table, dated prices where a vendor publishes them and what the vendor does not do. Read 2026-11-05: 671 facts from 331 sources.

This list profiles 73 companies that sit between a crawler and a page: the operators that run crawlers and agents, the vendors that decide at the door whether they get in, the providers that let a site check who an agent is, and the intermediaries that turn access into a licence or a price.

Crawler operators are every company behind a row of the Crawler Index (27 of them), plus 4 browser-agent and search-API companies whose own pages say how their traffic identifies itself. 15 of the vendors keep the ids and rubric pages they had before; their profiles now carry the same sections as the rest. A company is listed only when its own pages state what it does for this layer. Where a vendor's site refused the crawl's user agent, the profile cites a wire reprint, trade press or the vendor's developer documentation and says so; 12 profiles carry at least one third-party source.

671 facts come from 331 pages read on 2026-11-05, each with its source in the download and on its profile. A sentence that repeats one printed on another profile, or in a graded vendor's rubric text, appears once; 68 such sentences are kept in the download only. 50 vendors carry a dated pricing entry, and 34 of them state an amount. 32 profiles carry our own probes of key directories and IP files. Every homepage was fetched once on 2026-11-05 with its status recorded: 68 answered 200; Amazon answered 202, OpenAI answered 403, xAI answered 403, Akamai answered 403 and DataDome answered 403.

15 of these vendors are also graded on a five-question rubric. [The grades and the rubric](https://agentreadycrawlers.com/vendors/ranking/) are on their own page.

-   [Crawler operators (31)](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/)
-   [Bot management vendors (23)](https://agentreadycrawlers.com/vendors/by-type/bot-management/)
-   [Verification providers (11)](https://agentreadycrawlers.com/vendors/by-type/verification-provider/)
-   [Licensing intermediaries (8)](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/)

Standards are counted when the vendor's own pages state support, and listed by id: `rfc9309` is [RFC 9309 (robots.txt)](https://agentreadycrawlers.com/glossary/rfc-9309/); `web-bot-auth` is [Web Bot Auth](https://agentreadycrawlers.com/glossary/web-bot-auth/); `content-signals` is [Content Signals](https://agentreadycrawlers.com/glossary/content-signal/); `ip-list` is [Published IP list](https://agentreadycrawlers.com/glossary/ip-range-json/); `reverse-dns` is [Reverse DNS check](https://agentreadycrawlers.com/glossary/reverse-dns-verification/); `cloudflare-verified-bots` is [Cloudflare verified bots](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/); `visa-tap` is [Visa Trusted Agent Protocol](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol/); `mastercard-agent-pay` is [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/). Rows are alphabetical inside each type.

## Crawler operators

The companies behind the crawlers and agents in the Crawler Index, plus browser-agent and search-API companies whose own pages say how their traffic identifies itself. [The 31 crawler operators on a page of their own](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/).

| Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read |
| --- | --- | --- | --- | --- | --- | --- |
| Amazon | Crawler operator | Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 202, 4 probes 2026-11-05 | 2026-11-05 |
| Anchor Browser | Crawler operator | Cloud browser infrastructure for computer-use agents that can sign sessions with Cloudflare Web Bot Auth on its upper plans, and also sells stealth fingerprinting, proxies, CAPTCHA bypass and a pay-on-success web unlocker. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Anthropic | Crawler operator | Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 |
| Apify | Crawler operator | Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. | 0; partial: rfc9309, web-bot-auth, cloudflare-verified-bots | platform usage; subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Apple | Crawler operator | Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Block | Crawler operator | Company that built goose, an open-source local AI agent Cloudflare classed as a signed agent in 2025; Block has since donated goose to the Linux Foundation's Agentic AI Foundation and publishes no key directory for it. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Browser Use | Crawler operator | Maker of the open-source browser-use agent library and a hosted cloud browser for agents that ships with stealth fingerprinting, residential proxies and CAPTCHA solving on by default, and no published identifier for site owners. | 0 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Browserbase | Crawler operator | Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned. | 2 web-bot-auth, cloudflare-verified-bots; partial: rfc9309 | subscription plus usage | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Cohere | Crawler operator | Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page. | 1 rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Common Crawl | Crawler operator | Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests. | 3 rfc9309, ip-list, reverse-dns | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Diffbot | Crawler operator | Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. | 1 rfc9309 | subscription with credits; credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| DuckDuckGo | Crawler operator | DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Exa | Crawler operator | AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities. | 2 rfc9309, web-bot-auth | pay as you go | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Firecrawl | Crawler operator | Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends. | 1 rfc9309 | credit subscription | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Google | Crawler operator | Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic. | 2 ip-list, reverse-dns; partial: rfc9309, web-bot-auth | No price recorded | Homepage HTTP 200, 5 probes 2026-11-05 | 2026-11-05 |
| Hive | Crawler operator | AI model company, renamed from Hive to Deploy in October 2026, whose ImageSift unit runs ImagesiftBot to collect public images and page text for a reverse image search index. | 1 rfc9309 | usage-based | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Huawei | Crawler operator | Huawei runs PetalBot for the Petal search engine, whose index also feeds Huawei Assistant and AI Search; PetalBot follows robots.txt and is verified by reverse DNS under aspiegel.com or an IP lookup tool. | 2 rfc9309, reverse-dns | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Hyperbrowser | Crawler operator | Cloud browser platform for AI agents whose sessions can sign requests with Web Bot Auth under the fixed Signature-Agent www.hyperbrowser.ai, on request per team, alongside stealth, proxies and CAPTCHA solving. | 1 web-bot-auth | credits | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Kernel | Crawler operator | Cloud browser company for AI agents that runs two signed identities, Kernel Agent for user-directed browsing and KernelSearchBot for search crawling, each with its own Web Bot Auth key directory so sites can treat them separately. | 3 rfc9309, web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Manus | Crawler operator | General-purpose AI agent, owned by Meta since December 2025, that browses either in an isolated cloud browser or through an extension inside the user's own browser, and publishes no identifier, robots.txt policy or signing keys for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Meta | Crawler operator | Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt. | 0; partial: rfc9309 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Microsoft | Crawler operator | Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding. | 2 rfc9309, ip-list | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Mistral AI | Crawler operator | Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two. | 1 rfc9309; partial: ip-list | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 |
| Notte | Crawler operator | Browser platform for AI agents (sessions, agents and serverless browser functions) that says it is registered as a verified bot with Cloudflare and signs session requests with Web Bot Auth when a customer turns it on. | 2 web-bot-auth, cloudflare-verified-bots | subscription plus usage | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| OpenAI | Crawler operator | OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. | 2 web-bot-auth, ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 403, 5 probes 2026-11-05 | 2026-11-05 |
| Parallel Web Systems | Crawler operator | Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only. | 2 rfc9309, web-bot-auth; partial: ip-list | pay per request | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 |
| Perplexity | Crawler operator | Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF. | 1 ip-list; partial: rfc9309 | No price recorded | Homepage HTTP 200, 3 probes 2026-11-05 | 2026-11-05 |
| Timpi | Crawler operator | Operator of an independent search index built by a decentralised network of contributor nodes; its crawler appears in robots.txt files as Timpibot, but Timpi publishes no crawler documentation for site owners. | 0 | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| Webz.io | Crawler operator | Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training. | 1 rfc9309 | pay as you go | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| xAI | Crawler operator | xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic. | 0 | No price recorded | Homepage HTTP 403, 1 probe 2026-11-05 | 2026-11-05 |
| You.com | Crawler operator | You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | No price recorded | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |

## Bot management vendors

CDN, WAF, CAPTCHA and bot-detection vendors that decide at the door whether a crawler or agent gets in, and whether it pays. [The 23 bot management vendors on a page of their own](https://agentreadycrawlers.com/vendors/by-type/bot-management/).

| Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read |
| --- | --- | --- | --- | --- | --- | --- |
| Akamai | Bot management | CDN and security company whose Bot & Agent Control solutions judge AI agents at the edge, take identity from Visa's Trusted Agent Protocol and Know Your Agent tokens, and send AI traffic to TollBit or Skyfire for paid access. | 2 web-bot-auth, visa-tap | sales-led | Homepage HTTP 403 2026-11-05 | 2026-11-05 |
| AWS WAF Bot Control | Bot management | Cloud provider whose WAF Bot Control labels and verifies bots, checks Web Bot Auth signatures and can charge AI bots with HTTP 402, and whose AgentCore Browser signs its own agent traffic with Web Bot Auth. | 2 web-bot-auth, ip-list | per web ACL plus per request | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Arcjet | Bot management | Developer security SDK for JavaScript, Python and Go that detects bots inside application code, lets developers allow or deny named bots or categories such as CATEGORY:AI, and adds runtime guards for agent frameworks. | 2 ip-list, reverse-dns | monthly plan plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Castle | Bot management | Bot and account abuse API whose risk responses include an agent object that names AI agents and browser automation, identified by a verified Web Bot Auth signature, a browser fingerprint or a user-agent string. | 1 web-bot-auth | usage-based plans | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Centinel Analytica | Bot management | Bot and AI crawler protection vendor whose edge integration verifies declared search and AI crawlers against operators' published IP ranges and detects scrapers that run real browsers behind residential proxies. | 1 ip-list | quoted per deployment | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Cequence Security | Bot management | Application and API protection vendor whose Bot Management includes Agent Trust for personal AI shopping agents, and whose AI Gateway governs enterprise agents' tool calls behind OAuth 2.1 authentication. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Cloudflare | Bot management | Connectivity cloud whose AI Crawl Control, Bot Management and verified bot program sort crawlers by purpose, verify them by IP list or Web Bot Auth signature, and let site owners charge crawlers per request in a private beta. | 5 rfc9309, web-bot-auth, content-signals, ip-list, cloudflare-verified-bots | included in every plan; site owner sets one per-request price; 1 more | Homepage HTTP 200, 1 probe 2026-11-05 | 2026-11-05 |
| DataDome | Bot management | Bot and fraud protection vendor whose Agentic Trust layer names each AI agent, rates how strong that identification is, applies per-agent and per-endpoint policies, and hands chosen agents to TollBit for paid access. | 3 web-bot-auth, ip-list, reverse-dns | monthly, by request volume | Homepage HTTP 403 2026-11-05 | 2026-11-05 |
| F5 | Bot management | Application delivery and security company whose Distributed Cloud Bot Defense puts humans, trusted AI agents and malicious bots under one policy, accepts Web Bot Auth and Skyfire's Know Your Agent tokens, and runs on F5's platform, BIG-IP or public cloud. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Fastly | Bot management | Edge cloud vendor whose Bot Management adds edge detection (ContentGuard), TLS fingerprinting, client challenges and a verified bots signal, with separate AI crawler and AI fetcher signals for training bots and user-triggered fetchers. | 0 | contact sales | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Fingerprint | Bot management | Device intelligence vendor whose Bot Detection and Authorized AI Agent Detection signals separate good bots, bad bots and AI agents, and verify signed agents from providers such as OpenAI, AWS AgentCore and Browserbase. | 1 web-bot-auth | monthly plan plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Google Cloud Fraud Defense | Bot management | Google Cloud's bot and fraud service, the evolution of reCAPTCHA, which verifies AI agents by Web Bot Auth signature or by user agent matched to published IP ranges or reverse DNS, and reports verified and suspected agent sessions. | 3 web-bot-auth, ip-list, reverse-dns | per assessment | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| hCaptcha | Bot management | Bot detection and CAPTCHA vendor whose Enterprise Agent Controls identify AI agents, verify partner platforms that sign with Web Bot Auth, and let sites allow browsing while requiring verification before an agent reserves inventory. | 1 web-bot-auth | contact sales; monthly plus usage | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| HUMAN Security | Bot management | Bot and fraud defense vendor whose AgenticTrust module in Sightline names AI agents, grades them High, Medium or Low trust, verifies signed agents, and lets sites set what each agent may do from browsing to checkout. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Imperva | Bot management | Application security vendor owned by Thales whose Advanced Bot Protection sorts human, good bot and bad bot traffic across web, mobile and API; verified-agent controls so far ship in a controlled-availability Google Cloud offering. | 0 | sales-led, free trial | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Kasada | Bot management | Bot defense vendor founded in Sydney whose AI Agent Trust adds a verified agent directory with Web Bot Auth support and endpoint-level access rules, enforced at the edge before automated traffic reaches the application. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Known Agents | Bot management | Agent analytics and access-control service, formerly Dark Visitors, that identifies AI agents and crawlers from server-side data, keeps an automatic robots.txt current, and blocks bots that ignore it or spoof trusted identities. | 1 rfc9309 | monthly by events | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Link11 | Bot management | German security provider whose WAAP platform joins WAF, API security, Bot Management and Web DDoS protection, with a setting to monitor or block AI crawlers; the reblaze.com domain now redirects to Link11. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Netacea | Bot management | Server-side bot protection vendor that integrates at the CDN or load balancer without client code, classifies humans, good bots, bad bots and AI agents, and enforces access by intent rather than declared identity. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Netlify | Bot management | Web hosting platform that blocks chosen AI crawlers by user-agent string through an Edge Function extension and offers IP, geography and header-based traffic rules and rate limiting; it does not verify bot identity. | 0 | plan-based | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Radware | Bot management | Application security vendor whose Bot Manager and Agent Trust Management identify Web Bot Auth-compliant and non-compliant agents, capture an agent's declared intent in conversation, and restrict actions such as login or checkout. | 1 web-bot-auth | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Vercel | Bot management | Hosting platform whose firewall verifies known bots by IP, reverse DNS and Web Bot Auth, offers an AI Bots managed ruleset that treats AI crawlers and AI assistants separately, and sells BotID checks for high-value routes. | 3 web-bot-auth, ip-list, reverse-dns | included; per call | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Verisoul | Bot management | Fake account and fraud detection platform whose Agent & Bot Detection names AI agents by signature, self-declared user agent or inference, and whose public Agent Atlas documents how each agent shows up on a site. | 1 web-bot-auth | sales-led, 30-day trial | Homepage HTTP 200 2026-11-05 | 2026-11-05 |

## Verification providers

Web Bot Auth implementers, card-network agent programmes and agent identity products that let a site check who an agent is and who sent it. [The 11 verification providers on a page of their own](https://agentreadycrawlers.com/vendors/by-type/verification-provider/).

| Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read |
| --- | --- | --- | --- | --- | --- | --- |
| American Express | Verification provider | Card network whose ACE developer kit plans an agent registration service so only verified AI agents get tokenized Amex credentials, with Agent Purchase Protection for registered agent purchases; the registration spec is still in development. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Auth0 for AI Agents | Verification provider | Okta's customer identity platform; its AI Agents add-on gives agents a company builds a user-bound identity, scoped tokens, a Token Vault for third-party APIs and Cross App Access, and it verifies no inbound crawlers. | 0 | add-on, 50% of the base plan price | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Descope | Verification provider | Customer and agent identity platform whose Agentic Identity Hub gives AI agents and MCP clients OAuth 2.1 access, short-lived scoped credentials and an agent directory with policies, for agents calling a company's APIs rather than crawling its site. | 0 | free tier, then monthly plans | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Mastercard | Verification provider | Card network whose Agent Pay registers AI agents and has them sign requests with Web Bot Auth under the tag Agent-pay-auth, so a merchant or its CDN can check the agent's key in a network directory before letting it buy. | 2 web-bot-auth, mastercard-agent-pay; partial: visa-tap | No price recorded | Homepage HTTP 200, 2 probes 2026-11-05 | 2026-11-05 |
| Microsoft Entra Agent ID | Verification provider | Microsoft's identity service for AI agents inside an Entra tenant: each agent gets its own service principal created from a blueprint, with tokens, a human sponsor and audit, for agents an organisation runs rather than agents visiting a website. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Prove | Verification provider | Identity verification company whose Verified Agent product binds an AI agent to a verified person or business with identity-bound tokens, checks participants against a shared trust registry and logs co-signed consent for agent purchases. | 0 | sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Stytch IsAgent | Verification provider | Identity platform, part of Twilio since November 2025, that verifies Web Bot Auth signatures through its IsAgent SDK and Device Fingerprinting, and issues scoped OAuth tokens to AI agents and MCP clients through Connected Apps. | 1 web-bot-auth | free tier, then contact sales; per fingerprint above the included volume | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Trulioo | Verification provider | Identity verification company whose Know Your Agent programme, in alpha, runs business and AML checks on the company behind an AI agent, binds the agent's fingerprint to it in a signed Digital Agent Profile and lets counterparties check it by API or MCP. | 0 | alpha, access on request | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Visa | Verification provider | Card network whose Trusted Agent Protocol lets a merchant or its CDN verify, from RFC 9421 signatures tagged agent-browser-auth or agent-payer-auth, that a visiting agent is approved under Visa Intelligent Commerce and acting for a known consumer. | 2 web-bot-auth, visa-tap | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Vouched | Verification provider | Identity verification company whose Know Your Agent suite detects AI agents on a site by behaviour, enforces allow, challenge, block or redirect rules, and ties each agent action to a verified person through signed delegations. | 0 | free; monthly fee plus per transaction | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| World | Verification provider | Proof-of-human network whose AgentKit lets a person with a verified World ID register an agent's wallet, so x402-protected sites can tell a human-backed agent from a script and limit access per unique person. | 0 | developer preview | Homepage HTTP 200 2026-11-05 | 2026-11-05 |

## Licensing intermediaries

Marketplaces, standards bodies and paywalls that turn crawler access into a licence or a price. [The 8 licensing intermediaries on a page of their own](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary/).

| Vendor | Type | What it does for this layer | Standards supported | Pricing model | Our check | Last read |
| --- | --- | --- | --- | --- | --- | --- |
| Dappier | Licensing intermediary | Content marketplace where publishers list their content as priced RAG models and AI agents buy query access through an agent storefront that takes x402, MPP or card payments without signup. | 0; partial: rfc9309 | publisher-set price per query | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| FetchRight | Licensing intermediary | Licensing service behind the open Peek-Then-Pay protocol: a site publishes /.well-known/peek.json, AI agents get a priced HTTP 203 preview, buy a JWT license per intent and present it to an edge enforcer. | 0 | open specification; sales-led | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| IAB Tech Lab | Licensing intermediary | Ad-industry standards body whose Content Monetization Protocols (CoMP) 1.0 define how a licensed AI crawler asks a publisher for content and gets a token, alongside a paid Spiders and Bots List and a free Agent Registry. | 3 rfc9309, web-bot-auth, reverse-dns; partial: ip-list | open specification; annual subscription; 1 more | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| ProRata | Licensing intermediary | AI attribution company that splits an AI answer into claims and measures each source's share as a basis for licensing and payment, and runs Gist Answers on a network of licensed publisher titles. | 0 | No price recorded | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| RSL Collective | Licensing intermediary | Nonprofit collective rights organisation and co-developer of the RSL standard, a machine-readable license file a site links from robots.txt so crawlers can see, buy and prove a license for training, AI input, search or crawling. | 1 rfc9309 | open standard; free to join for publishers | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Skyfire | Licensing intermediary | Agent identity and payment network whose signed KYA and pay tokens let a website, API or bot manager admit a verified agent and charge it per request, with Skyfire vetting the agent platform and the person or business behind the agent. | 0 | subscription for verification (Individual or Organization) | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| Supertab | Licensing intermediary | Managed RSL licensing service: Supertab Connect hosts a publisher's license.xml and RSL license server, issues license tokens to crawlers, checks them at the CDN with its SDK and bills the use, priced from USD 99 a month. | 1 rfc9309 | monthly subscription by visitors and licensees | Homepage HTTP 200 2026-11-05 | 2026-11-05 |
| TollBit | Licensing intermediary | Licensing marketplace that sends AI bots, by user agent at the publisher's CDN, to an Agent Site on a tollbit subdomain, where bots holding a short-lived TollBit JWT get licensed content at publisher-set rates per 1,000 pages. | 0; partial: rfc9309 | free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee | Homepage HTTP 200 2026-11-05 | 2026-11-05 |

## Not listed, and why

The research pass looked at these companies and did not profile them. A name taken from a roundup with no page of its own is left out the same way.

-   **ByteDance.** No ByteDance page states a robots.txt policy, an IP list or a verification method for Bytespider, so the Bytespider row stays in the Crawler Index without a vendor profile.
-   **Steel, Browserless, Skyvern, Airtop, Bright Data, Zyte, Oxylabs and ScrapingBee.** Their documentation covers stealth, proxies and CAPTCHA solving for customers' traffic but states no user agent token, robots.txt policy or Web Bot Auth signing.
-   **Microsoft Azure WAF bot protection.** Microsoft Learn describes good-bot and bad-bot rules and says nothing about AI crawlers, agents or Web Bot Auth.
-   **Gcore.** Its robots.txt disallows the crawl user agent on the bot protection page.
-   **Friendly Captcha.** No page states how agents are detected.
-   **Peakhour, Myra Security and Barracuda.** The pages read state no AI crawler or agent feature.
-   **Sucuri, Wordfence, Prophaze and Indusface.** Not reached in this edition.
-   **GoDaddy and Persona.** Their sites answered 403 to the crawl user agent.
-   **Ping Identity and Crossmint.** No agent product page was found.
-   **Spawning.** Its site showed a maintenance page on 2026-11-05.
-   **ScalePost.** It now sells AI visibility measurement, which is not this site's subject.
-   **Copyright Clearance Center and Created by Humans.** Their licences cover internal AI use or book rights and say nothing about crawlers or site access.
-   **Coinbase x402.** Coinbase documents paid APIs and agent tool calls, not charging crawlers.
-   **Human Native.** Cloudflare announced its acquisition on 15 Jan 2026, so it is covered in the Cloudflare profile.

## Download the list

One row per vendor: type, operator, headquarters, products with their prices, the standards assessed, our probes, what the vendor does not do, and every sourced fact in the JSON file. Edition 2026-11-05.

-   [vendors.csv](https://agentreadycrawlers.com/vendors/vendors.csv) text/csv, about 55 KB
-   [vendors.json](https://agentreadycrawlers.com/vendors/vendors.json) application/json, about 412 KB

**Cite this.** Agent Ready Crawlers, "Vendors", edition 2026-11-05, https://agentreadycrawlers.com/vendors/. Free to use with this line; see [the terms](https://agentreadycrawlers.com/terms/).

## Sources

-   [Method: index, block rates, vendors (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04
-   [Amazon: amazon.com](https://www.amazon.com) read 2026-11-05
-   [Anchor Browser: anchorbrowser.io](https://anchorbrowser.io) read 2026-11-05
-   [Anthropic: anthropic.com](https://www.anthropic.com) read 2026-11-05
-   [Apify: apify.com](https://apify.com) read 2026-11-05
-   [Apple: apple.com](https://www.apple.com) read 2026-11-05
-   [Block: block.xyz](https://block.xyz) read 2026-11-05
-   [Browser Use: browser-use.com](https://browser-use.com) read 2026-11-05
-   [Browserbase: browserbase.com](https://www.browserbase.com) read 2026-11-05
-   [Cohere: cohere.com](https://cohere.com) read 2026-11-05
-   [Common Crawl: commoncrawl.org](https://commoncrawl.org) read 2026-11-05
-   [Diffbot: diffbot.com](https://www.diffbot.com) read 2026-11-05
-   [DuckDuckGo: duckduckgo.com](https://duckduckgo.com) read 2026-11-05
-   [Exa: exa.ai](https://exa.ai) read 2026-11-05
-   [Firecrawl: firecrawl.dev](https://www.firecrawl.dev) read 2026-11-05
-   [Google: google.com](https://www.google.com) read 2026-11-05
-   [Hive: deploy.com](https://deploy.com) read 2026-11-05
-   [Huawei: huawei.com](https://www.huawei.com) read 2026-11-05
-   [Hyperbrowser: hyperbrowser.ai](https://www.hyperbrowser.ai) read 2026-11-05
-   [Kernel: kernel.sh](https://www.kernel.sh) read 2026-11-05
-   [Manus: manus.im](https://manus.im) read 2026-11-05
-   [Meta: meta.com](https://www.meta.com) read 2026-11-05
-   [Microsoft: microsoft.com](https://www.microsoft.com) read 2026-11-05
-   [Mistral AI: mistral.ai](https://mistral.ai) read 2026-11-05
-   [Notte: notte.cc](https://www.notte.cc) read 2026-11-05
-   [OpenAI: openai.com](https://openai.com) read 2026-11-05
-   [Parallel Web Systems: parallel.ai](https://parallel.ai) read 2026-11-05
-   [Perplexity: perplexity.ai](https://www.perplexity.ai) read 2026-11-05
-   [Timpi: timpi.io](https://timpi.io) read 2026-11-05
-   [Webz.io: webz.io](https://webz.io) read 2026-11-05
-   [xAI: x.ai](https://x.ai) read 2026-11-05
-   [You.com: you.com](https://you.com) read 2026-11-05
-   [Akamai: akamai.com](https://www.akamai.com) read 2026-11-05
-   [AWS WAF Bot Control: aws.amazon.com](https://aws.amazon.com/waf/features/bot-control/) read 2026-11-05
-   [Arcjet: arcjet.com](https://arcjet.com) read 2026-11-05
-   [Castle: castle.io](https://castle.io) read 2026-11-05
-   [Centinel Analytica: centinelanalytica.com](https://www.centinelanalytica.com) read 2026-11-05
-   [Cequence Security: cequence.ai](https://www.cequence.ai) read 2026-11-05
-   [Cloudflare: cloudflare.com](https://www.cloudflare.com) read 2026-11-05
-   [DataDome: datadome.co](https://datadome.co) read 2026-11-05
-   [F5: f5.com](https://www.f5.com) read 2026-11-05
-   [Fastly: fastly.com](https://www.fastly.com) read 2026-11-05
-   [Fingerprint: fingerprint.com](https://fingerprint.com) read 2026-11-05
-   [Google Cloud Fraud Defense: cloud.google.com](https://cloud.google.com/security/products/recaptcha) read 2026-11-05
-   [hCaptcha: hcaptcha.com](https://www.hcaptcha.com) read 2026-11-05
-   [HUMAN Security: humansecurity.com](https://www.humansecurity.com) read 2026-11-05
-   [Imperva: imperva.com](https://www.imperva.com) read 2026-11-05
-   [Kasada: kasada.io](https://www.kasada.io) read 2026-11-05
-   [Known Agents: knownagents.com](https://knownagents.com) read 2026-11-05
-   [Link11: link11.com](https://www.link11.com) read 2026-11-05
-   [Netacea: netacea.com](https://netacea.com) read 2026-11-05
-   [Netlify: netlify.com](https://www.netlify.com) read 2026-11-05
-   [Radware: radware.com](https://www.radware.com) read 2026-11-05
-   [Vercel: vercel.com](https://vercel.com) read 2026-11-05
-   [Verisoul: verisoul.ai](https://www.verisoul.ai) read 2026-11-05
-   [American Express: americanexpress.com](https://www.americanexpress.com/en-us/company/agentic-commerce/) read 2026-11-05
-   [Auth0 for AI Agents: auth0.com](https://auth0.com/ai) read 2026-11-05
-   [Descope: descope.com](https://www.descope.com/use-cases/ai) read 2026-11-05
-   [Mastercard: developer.mastercard.com](https://developer.mastercard.com/merchant-cloud/documentation/tutorials-and-guides/agentic-commerce-guide/) read 2026-11-05
-   [Microsoft Entra Agent ID: learn.microsoft.com](https://learn.microsoft.com/en-us/entra/agent-id/agent-identities) read 2026-11-05
-   [Prove: prove.com](https://www.prove.com/solutions/agentic-suite) read 2026-11-05
-   [Stytch IsAgent: stytch.com](https://stytch.com) read 2026-11-05
-   [Trulioo: trulioo.com](https://www.trulioo.com/products/kya-verification) read 2026-11-05
-   [Visa: developer.visa.com](https://developer.visa.com/docs/trusted-agent-protocol) read 2026-11-05
-   [Vouched: vouched.id](https://www.vouched.id/know-your-agent) read 2026-11-05
-   [World: world.org](https://world.org) read 2026-11-05
-   [Dappier: dappier.com](https://dappier.com) read 2026-11-05
-   [FetchRight: fetchright.ai](https://fetchright.ai) read 2026-11-05
-   [IAB Tech Lab: iabtechlab.com](https://iabtechlab.com/standards/comp-content-monetization-protocols-initiative/) read 2026-11-05
-   [ProRata: prorata.ai](https://prorata.ai) read 2026-11-05
-   [RSL Collective: rslcollective.org](https://rslcollective.org) read 2026-11-05
-   [Skyfire: skyfire.xyz](https://skyfire.xyz) read 2026-11-05
-   [Supertab: supertab.co](https://www.supertab.co) read 2026-11-05
-   [TollBit: tollbit.com](https://tollbit.com) read 2026-11-05

Last reviewed 2026-11-05