Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

The difference between a crawler and a browsing agent

In short

Crawlers fetch on a schedule under their own token; browsing agents act for one person, often from that person's browser, and many carry no token at all. Cloudflare classifies bots by three controllable behaviours: Search, Training and Agent (user-directed agents visiting for a human). Meta-ExternalFetcher may bypass robots.txt because it fetches links at a user's request.

A crawler fetches on a schedule under its own name. A browsing agent acts for one person and often borrows that person's browser.

In practice

  1. Sort by behaviour, as Cloudflare does: Search, Training and Agent, where Agent means user-directed agents visiting for a human.
  2. Look for a signature. Cloudflare's signed agents are directed by end users and sign with Web Bot Auth; the first cohort was ChatGPT agent, Goose, Browserbase and Anchor Browser. Google-Agent is used by agents hosted on Google infrastructure, with ranges in user-triggered-agents.json.
  3. Expect user-triggered fetchers to differ on robots.txt. Meta-ExternalFetcher may bypass it because it fetches links at a user's request.
  4. Expect some agents to leave no mark. Verisoul reports Comet browses from the user's device with Chrome's user agent, no signature and the user's IP.
  5. Set policy per behaviour, not per name: allow or refuse the search and agent behaviours at the WAF, because a robots.txt line cannot address an agent with no token.

What the sources say

Sources

Last reviewed 2026-11-04