Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Browsing agent

In short

Software that drives a browser to complete a task for a person: clicking, typing and checking out. It may run in the operator's cloud or in the user's own browser. 20 Index rows are task agents, and 19 of them have no robots.txt token. Example: ChatGPT's Cloud browser runs in OpenAI's cloud and signs its requests.

Definition

Software that drives a browser to complete a task for a person: clicking, typing and checking out. It may run in the operator's cloud or in the user's own browser.

Sourced example

ChatGPT's Cloud browser runs in OpenAI's cloud and signs its requests; Claude in Chrome runs in the user's Chrome. Source, read 2026-11-04.

How it shows up here

20 Index rows are task agents, and 19 of them have no robots.txt token.

Previous: BotBase. Next: Content-Signal. All 41 terms.

Sources

Last reviewed 2026-11-04