Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Vendors by type

Licensing intermediaries

In short

8 licensing intermediaries are profiled, from marketplaces such as TollBit and Dappier to the RSL standard and IAB Tech Lab's CoMP. 3 of 8 document support for RFC 9309, 1 for Web Bot Auth and 1 for reverse DNS. Most of them price access without blocking anyone, and enforcement stays with the publisher's CDN or bot manager.

The 8 licensing intermediaries run alphabetically from Dappier to TollBit, and carry 85 sourced facts between them.

The price models on record include publisher-set price per query; open specification; sales-led; annual subscription; free with Tools Portal registration; and 5 more. None of the profiles carries a key directory or IP file probe of our own. Each profile lists what the vendor does not do next to what it does.

Vendor Type What it does for this layer Standards supported Pricing model Our check Last read
Dappier Licensing intermediary Content marketplace where publishers list their content as priced RAG models and AI agents buy query access through an agent storefront that takes x402, MPP or card payments without signup. 0
partial: rfc9309		 publisher-set price per query Homepage HTTP 200 2026-11-05 2026-11-05
FetchRight Licensing intermediary Licensing service behind the open Peek-Then-Pay protocol: a site publishes /.well-known/peek.json, AI agents get a priced HTTP 203 preview, buy a JWT license per intent and present it to an edge enforcer. 0 open specification; sales-led Homepage HTTP 200 2026-11-05 2026-11-05
IAB Tech Lab Licensing intermediary Ad-industry standards body whose Content Monetization Protocols (CoMP) 1.0 define how a licensed AI crawler asks a publisher for content and gets a token, alongside a paid Spiders and Bots List and a free Agent Registry. 3 rfc9309, web-bot-auth, reverse-dns
partial: ip-list		 open specification; annual subscription; 1 more Homepage HTTP 200 2026-11-05 2026-11-05
ProRata Licensing intermediary AI attribution company that splits an AI answer into claims and measures each source's share as a basis for licensing and payment, and runs Gist Answers on a network of licensed publisher titles. 0 No price recorded Homepage HTTP 200 2026-11-05 2026-11-05
RSL Collective Licensing intermediary Nonprofit collective rights organisation and co-developer of the RSL standard, a machine-readable license file a site links from robots.txt so crawlers can see, buy and prove a license for training, AI input, search or crawling. 1 rfc9309 open standard; free to join for publishers Homepage HTTP 200 2026-11-05 2026-11-05
Skyfire Licensing intermediary Agent identity and payment network whose signed KYA and pay tokens let a website, API or bot manager admit a verified agent and charge it per request, with Skyfire vetting the agent platform and the person or business behind the agent. 0 subscription for verification (Individual or Organization) Homepage HTTP 200 2026-11-05 2026-11-05
Supertab Licensing intermediary Managed RSL licensing service: Supertab Connect hosts a publisher's license.xml and RSL license server, issues license tokens to crawlers, checks them at the CDN with its SDK and bills the use, priced from USD 99 a month. 1 rfc9309 monthly subscription by visitors and licensees Homepage HTTP 200 2026-11-05 2026-11-05
TollBit Licensing intermediary Licensing marketplace that sends AI bots, by user agent at the publisher's CDN, to an Agent Site on a tollbit subdomain, where bots holding a short-lived TollBit JWT get licensed content at publisher-set rates per 1,000 pages. 0
partial: rfc9309		 free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee Homepage HTTP 200 2026-11-05 2026-11-05

Sources

Last reviewed 2026-11-05