Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Agentic browsers and how they identify

In short

The Index holds 23 agent rows, 20 task agents and 3 shopping agents. 8 sign their requests, 1 names a robots.txt token and 6 look like the visitor's own browser. 15 publish nothing a site can check, so a log line cannot tell them from a person.

A crawler arrives with a name. A browsing agent often arrives as a browser: the same window a person uses, driven by a model. The question for a site owner is what, if anything, in the request says an agent sent it. This table gives the answer row by row, from the operator's own pages. The difference between the two kinds of visitor is explained in the guide on crawlers and browsing agents.

How the rows identify themselves

How the agent rows in the Crawler Index identify themselvesOut of 23 task-agent and shopping-agent rows: 8 signs requests, 1 names a token, 1 full user-agent string, 6 visitor's own browser, 15 nothing a site can check. The groups overlap.0510152025Signs requests8Names a token1Full user-agent string1Visitor's own browser6Nothing a site can check15How the agent rows in the Crawler Index identify themselvesOut of 23 task-agent and shopping-agent rows: 8 signs requests, 1 names a token, 1 full user-agent string, 6 visitor's own browser, 15 nothing a site can check. The groups overlap.01020Signs requests8Names a token1Full user-agent string1Visitor's own browser6Nothing a site can check15
Out of 23 agent rows. The groups overlap: an agent can sign its requests and also appear as a browser.

Only one row, Claude Duet cloud browser, publishes a robots.txt token, so a robots.txt group can address that agent and no others. For the rest, the lever is the WAF and, where the operator signs requests, the signature.

Every agent row

Task agents and shopping agents in the Crawler Index. What the operator publishes, read 2026-11-04.
AgentOperatorTokenUser agent, as publishedVerificationrobots.txt
ChatGPT agent OpenAI none Requests carry Signature, Signature-Input and Signature-Agent: "https://chatgpt.com" headers (RFC 9421); the user agent string itself is not published as an identifier Web Bot Auth signature No statement
ChatGPT Atlas OpenAI none Ran in the user's own browser with a Chrome user agent; no agent token published Nothing published No docs
Claude in Chrome Anthropic none The user's own Chrome user agent; no agent token Nothing published No statement
Claude Duet cloud browser Anthropic Claude-Agent Linux Chrome string with a "compatible; Claude-Agent/1.0" token, as observed by Verisoul (secondary) Nothing published Reported
Claude computer use Anthropic none Whatever browser the developer runs; no Anthropic identifier Nothing published No docs
Comet Perplexity none Chrome's user agent from the user's own device; no unique identifier Nothing published No docs
Perplexity Computer Perplexity none No declared identity Nothing published No docs
Google-Agent Google none Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36 Web Bot Auth signature May not apply
Gemini in Chrome Google none The user's own Chrome user agent; no marker in local mode Nothing published No docs
Browse with Copilot Microsoft none The user's own Edge user agent; no marker documented Nothing published No docs
Buy for Me Amazon none Not published Nothing published No docs
Rufus Amazon none Not applicable to third-party sites Nothing published No docs
Amazon Bedrock AgentCore Browser Amazon Web Services none Signed with Web Bot Auth when enabled Web Bot Auth signature No statement
Muse Meta none Does not identify itself, per Verisoul and Amazon's statement reported by GeekWire Nothing published No docs
Browser Use Browser Use none Chromium driven over the DevTools Protocol; Verisoul reports it names itself only in some setups Nothing published No docs
Apify Website Content Crawler Apify none Signed with Web Bot Auth when signHttpRequests is enabled Web Bot Auth signature Setting
Browserbase cloud browsers Browserbase none Signed with Web Bot Auth for customers who enable it Web Bot Auth signature No docs
Stagehand Browserbase none Inherits the browser it runs in; signed only when run on Browserbase with Web Bot Auth Nothing published No docs
Cloudflare Browser Rendering Cloudflare none Sends signed Web Bot Auth headers Web Bot Auth signature No statement
Anchor Browser Anchor none Can sign with Web Bot Auth depending on customer setup Web Bot Auth signature No docs
Goose Block none Signs with Web Bot Auth under Cloudflare's signed agents program Cloudflare signed agent No docs
Grok Bot xAI none No declared identity since mid-October 2026, per Verisoul Nothing published No docs
Manus cloud browser Manus none Signs cloud-browser page loads, per Verisoul Nothing published No docs

What a site can do about each group

Signed agents
8 rows sign with Web Bot Auth or sit on Cloudflare's signed agent list. Verify the signature and write the rule on the verified identity. How to verify Web Bot Auth signatures.
Agents in the visitor's browser
6 rows send the user's own Chrome or Edge user agent from the user's own address. A rule written on the string would refuse people too. Account and checkout pages need authentication, since robots.txt cannot address these agents.
Everything else
9 rows publish nothing and do not look like a visitor's browser either. The page linked in each row says what the operator does and does not disclose.

Sources

Last reviewed 2026-11-05