Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Task agent: 20 rows

In short

20 rows in the Index are filed under task agent, from 14 operators. 12 publish nothing a site can verify and 19 have no robots.txt token. The crawl measured 0 of these tokens. A browsing agent that clicks, fills forms and finishes a job for a person.

A browsing agent that clicks, fills forms and finishes a job for a person.

Task agent: 20 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
OpenAI ChatGPT agent (Cloud browser, formerly Operator) no token Task agent No statement Web Bot Auth no token Profile
OpenAI ChatGPT Atlas (browser agent mode) no token Task agent No docs None published no token Profile
Anthropic Claude in Chrome no token Task agent No statement None published no token Profile
Anthropic Claude-Agent Task agent Reported None published not measured Profile
Anthropic Claude computer use (API tool) no token Task agent No docs None published no token Profile
Perplexity Comet (browser assistant) no token Task agent No docs None published no token Profile
Perplexity Perplexity Computer no token Task agent No docs None published no token Profile
Google Google-Agent no token Task agent May not apply Web Bot Auth no token Profile
Google Gemini in Chrome (auto browse) no token Task agent No docs None published no token Profile
Microsoft Browse with Copilot (formerly Copilot Actions) no token Task agent No docs None published no token Profile
Amazon Web Services Amazon Bedrock AgentCore Browser no token Task agent No statement Web Bot Auth no token Profile
Browser Use Browser Use (library and cloud) no token Task agent No docs None published no token Profile
Apify Apify Website Content Crawler no token Task agent Setting Web Bot Auth no token Profile
Browserbase Browserbase cloud browsers no token Task agent No docs Web Bot Auth no token Profile
Browserbase Stagehand (open-source framework) no token Task agent No docs None published no token Profile
Cloudflare Cloudflare Browser Rendering no token Task agent No statement Web Bot Auth no token Profile
Anchor Anchor Browser no token Task agent No docs Web Bot Auth no token Profile
Block Goose (open-source agent) no token Task agent No docs Cloudflare signed no token Profile
xAI Grok Bot (agent) no token Task agent No docs None published no token Profile
Manus Manus cloud browser no token Task agent No docs None published no token Profile

Sources

Last reviewed 2026-11-04