Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Vendor, ranked 6 (tied) of 15

TollBit

In short

TollBit is a paywall for AI bots, charging per thousand pages: charging and reporting grade yes, while agent verification and per-purpose policy are partial because classifying visitors is left to CDN partners, for 6 of 10 points. Commerce agents are not published, and signing up costs a publisher nothing.

Platform that lets publishers see AI bot traffic, set rules and per-access rates, and send bots to a paywall where AI companies with valid TollBit tokens get content; detection relies on CDN and bot-management partners. Vendor description of itself, checked against its documents on 2026-11-04.

Alternatives to TollBit ยท tollbit.com

Type
Licensing intermediary
Homepage
tollbit.com
Last read
2026-11-05

The rubric, with evidence

QuestionGradeWhat the vendor documentsEvidence
Tells a verified agent from a spoof Partial issues access tokens to AI companies and depends on CDN or bot-manager partners for detection; Web Bot Auth support not confirmed (checked 2026-11-04) Fastly flow grants access to bots with valid TollBit tokens. Source, read 2026-11-04.
Per-purpose policy (train, search, answer, act) Partial rules and rates per bot set by the publisher Homepage: enforcing rules and rates for content access. Source, read 2026-11-04.
Charging for access Yes bot paywall with publisher-set usage-based pricing Homepage paywall section. Source, read 2026-11-04.
Lets signed commerce agents through Not published Visa TAP or Mastercard Agent Pay support (checked 2026-11-04) Not on homepage. Source, read 2026-11-04.
Cost Not graded free for publishers to start Homepage: Get started free. Source, read 2026-11-04.
Reporting Yes TollBit Analytics and portfolio analytics across sites Homepage analytics section. Source, read 2026-11-04.
Where it runs Not graded integrates at the CDN or bot-manager layer (Fastly, Akamai, DataDome, Arc XP) by redirect Partner integrations. Source, read 2026-11-04.

Cost

Free for publishers to start ('Get started free'); publishers set their own per-access rates; TollBit's fees to AI companies are not published (checked 2026-11-04). Source, read 2026-11-04.

What TollBit does not do

Documented facts

Features are as the vendor documents them; we have not tested them. A claim a vendor makes about itself is marked as a vendor claim.

Compared with

What TollBit does for this layer

From the vendor's own pages

TollBit's Agent Site lives on a tollbit subdomain of the publisher's domain, for example tollbit.yourdomain.com. (docs.tollbit.com) TollBit provisions that subdomain after the publisher proves ownership of the domain with a DNS TXT record and points NS records for the tollbit subdomain at four TollBit nameservers. (docs.tollbit.com, dated 2026-10-12) TollBit's redirect runs as edge logic at the publisher's CDN, such as Cloudflare or Akamai, which checks each request against a list of known bot user agents and sends matches to the tollbit subdomain. (docs.tollbit.com) TollBit says it concentrates on self-identified bots such as ChatGPT, Perplexity and Claude and leaves undeclared scrapers to partner tools from DataDome and HUMAN Security. (docs.tollbit.com, dated 2026-10-23)

TollBit updates its bot list about once a quarter and expects publishers to add new crawlers to their edge configuration or robots.txt by hand. (docs.tollbit.com, dated 2026-10-23) TollBit tokens are JWTs of type content or crawl that expire 5 minutes after issuance and bind the requesting user agent in the claim x.ua. (docs.tollbit.com, dated 2026-03-20) A TollBit content token carries the maximum price in micros, where 1 USD equals 1,000,000 micros, plus the currency, the license type and the license ID. (docs.tollbit.com, dated 2026-03-20) TollBit offers publishers a Summarization License and a Full Display License, each priced per 1,000 pages accessed, and a Custom License attached to a named partner's user agents. (docs.tollbit.com, dated 2026-06-25)

TollBit resolves a price by checking rates in the order bot, page, keyword, time and subdirectory; a page that matches no rate cannot be accessed by AI customers. (docs.tollbit.com, dated 2026-06-25) TollBit states it takes no percentage of publisher rates and instead charges AI customers a transaction fee on top of the rate the publisher sets. (docs.tollbit.com, dated 2026-06-25) TollBit documents integrations for Akamai, Arc XP, AWS CloudFront and ALB, Cloudflare, DataDome, Fastly, Google Cloud CDN, Azure Front Door, Vercel, WordPress VIP, Imperva and Netacea. (docs.tollbit.com) TollBit also exposes a publisher's content through NLWeb, with /ask and /mcp endpoints on the tollbit subdomain, and through Google's Agent2Agent protocol. (docs.tollbit.com, dated 2026-06-14)

After a bot presents a valid token, TollBit fetches the page through its reverse proxy at api.tollbit.com/GetContent and returns cleaned, structured content instead of the raw HTML. (docs.tollbit.com, dated 2026-10-23)

Claims the vendor makes about itself

TollBit's homepage shows publisher sites including reuters.com, apnews.com, economist.com, forbes.com and pbs.org in its customer strip. (tollbit.com)

Products and plans

ProductWhat it doesPrice modelFromAs of
Agent Site Subdomain gateway for AI agents with access controls, per-agent content and paywall enforcement. free for publishers; AI customers pay the publisher's rate plus a TollBit transaction fee transaction fee amount not published USD 0 2026-11-05
Bot Paywall Paid access terms for AI bots, priced per 1,000 pages by license type. No price recorded
TollBit Analytics Bot traffic analytics from CDN, CMS, security tool or cloud bucket log ingestion. No price recorded
Licensed RAG Access and Licensed Search API for AI developers to search, check rates and retrieve licensed content with tokens. No price recorded

Standards

StandardStatusEvidence
RFC 9309 (robots.txt) Partial TollBit relies on edge redirects by user agent and asks publishers to keep robots.txt current with new crawlers; it does not manage robots.txt itself. (docs.tollbit.com)
Web Bot Auth No statement TollBit's docs index and FAQ describe identification by user agent and partner tools, with no Web Bot Auth mention (checked 2026-11-05).
Content Signals No statement No Content-Signals mention in the docs index (checked 2026-11-05).

Our checks

CheckResultMethodDate
Homepage fetch HTTP 200 GET with the crawl user agent after a robots.txt check 2026-11-05

Sources

Last reviewed 2026-11-05