Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Crawler operator

Browserbase

In short

Browserbase hosts headless browsers for agents and publishes the Stagehand SDK under the MIT licence. Web Bot Auth signing is a beta enabled per account on request, with two Ed25519 keys served from www.browserbase.com, so some sessions arrive signed and others do not. Whether an agent obeys a site's rules is left to the customer.

Type
Crawler operator
Homepage
browserbase.com
In one line
Cloud browser platform for AI agents, maker of the Stagehand SDK, that offers Web Bot Auth signing in beta alongside Verified fingerprinted browsers, proxies and CAPTCHA solving, so its traffic may arrive signed or unsigned.
Last read
2026-11-05

What Browserbase does for this layer

From the vendor's own pages

Browserbase groups its access features under Browserbase Identity: Verified browsers, Web Bot Auth, CAPTCHA solving, residential or datacenter proxies, and human verification with AgentKit for x402 sessions. (docs.browserbase.com) Browserbase's Verified sessions use a purpose-built Chromium with "real browser fingerprints" that its bot protection partners recognise; Verified is sold on the Scale plan and defaults to a Windows operating system. (docs.browserbase.com) Browserbase offers Web Bot Auth in beta, enabled per account on request, and its identity docs say a valid signature does not grant access or sign the agent into a user account. (docs.browserbase.com) Browserbase's Web Bot Auth page lists the Signature-Agent, Signature-Input and Signature headers and says that signing @authority with created and expires parameters limits where and when a signature works without preventing all replay. (docs.browserbase.com)

Browserbase's guidance for site compliance tells customers to review a site's terms of service and to check robots.txt "when applicable" before an agent visits. (docs.browserbase.com) Browserbase's Developer plan includes 100 browser hours then USD 0.12 per browser hour and 1 GB of proxies then USD 12 per GB; the Startup plan includes 500 hours then USD 0.10 and 5 GB then USD 10 per GB. (browserbase.com) Browserbase's docs index lists AgentKit by Tools for Humanity, which lets an agent prove with a cryptographic proof-of-humanity that a real person backs it. (docs.browserbase.com) Stagehand, Browserbase's SDK for browser agents, is MIT-licensed and showed 26.0k GitHub stars on 2026-11-05. (github.com)

Stagehand combines natural-language act, extract and observe steps with Playwright-style page calls, in TypeScript, Python and Go. (docs.stagehand.dev)

Reported by a third party

Cloudflare named Browserbase in its first cohort of signed agents on 2025-08-28, alongside ChatGPT agent, goose from Block and Anchor Browser. (blog.cloudflare.com, dated 2025-08-28)

What our crawl saw

Browserbase's key directory at www.browserbase.com/.well-known/http-message-signatures-directory returned HTTP 200 with content type application/http-message-signatures-directory+json and two Ed25519 keys on 2026-11-05. (browserbase.com)

Products and plans

ProductWhat it doesPrice modelFromAs of
Browserbase cloud browsers Hosted headless browsers for agents, with Verified, proxies, CAPTCHA solving and optional Web Bot Auth. subscription plus usage Page title lists Free, USD 20, USD 99 or custom; overage USD 0.12 or 0.10 per browser hour by plan. USD 20 2026-11-05
Stagehand MIT-licensed SDK for browser agents; identity comes from the browser it runs in. No price recorded

Standards

StandardStatusEvidence
RFC 9309 (robots.txt) Partial Advice to customers to check robots.txt when applicable; no crawl policy of its own. (docs.browserbase.com)
Web Bot Auth Supports Web Bot Auth signing in beta per account; key directory served on www.browserbase.com. (docs.browserbase.com)
Published IP list No statement No egress IP list; proxies are residential or datacenter (2026-11-05).
Cloudflare verified bots Supports Listed by Cloudflare as a signed agent in the first cohort, 2025-08-28. (blog.cloudflare.com)

Our checks

CheckResultMethodDate
Homepage fetch HTTP 200 GET with the crawl user agent after a robots.txt check 2026-11-05
Web Bot Auth key directory HTTP 200, application/http-message-signatures-directory+json, 2 Ed25519 keys GET https://www.browserbase.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check 2026-11-05

What Browserbase does not do

  • Browserbase signs requests only for accounts in its Web Bot Auth beta, so a site cannot expect every Browserbase session to carry a signature.
  • Browserbase publishes no robots.txt token or crawl policy of its own; compliance is left to each customer's code.

Sources

Last reviewed 2026-11-05