---
title: "Crawler versus browsing agent | Agent Ready Crawlers"
description: "The difference between a crawler and a browsing agent: schedule against task, token against none, and what that means for robots.txt."
canonical: https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
date: 2026-11-04
section: "Guides"
---

# Crawler versus browsing agent | Agent Ready Crawlers

> The difference between a crawler and a browsing agent: schedule against task, token against none, and what that means for robots.txt.

Source: https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

# The difference between a crawler and a browsing agent

In short

Crawlers fetch on a schedule under their own token; browsing agents act for one person, often from that person's browser, and many carry no token at all. Cloudflare classifies bots by three controllable behaviours: Search, Training and Agent (user-directed agents visiting for a human). Meta-ExternalFetcher may bypass robots.txt because it fetches links at a user's request.

A crawler fetches on a schedule under its own name. A browsing agent acts for one person and often borrows that person's browser.

## In practice

1.  Sort by behaviour, as Cloudflare does: Search, Training and Agent, where Agent means user-directed agents visiting for a human.
2.  Look for a signature. Cloudflare's signed agents are directed by end users and sign with Web Bot Auth; the first cohort was ChatGPT agent, Goose, Browserbase and Anchor Browser. Google-Agent is used by agents hosted on Google infrastructure, with ranges in user-triggered-agents.json.
3.  Expect user-triggered fetchers to differ on robots.txt. Meta-ExternalFetcher may bypass it because it fetches links at a user's request.
4.  Expect some agents to leave no mark. Verisoul reports Comet browses from the user's device with Chrome's user agent, no signature and the user's IP.
5.  Set policy per behaviour, not per name: allow or refuse the search and agent behaviours at the WAF, because a robots.txt line cannot address an agent with no token.

## What the sources say

-   Primary source Cloudflare classifies bots by three controllable behaviours: Search, Training and Agent (user-directed agents visiting for a human). [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04.
-   Primary source Cloudflare's signed agents are directed by end users, sign with Web Bot Auth and follow its signed agent policy; the first cohort was ChatGPT agent, Goose, Browserbase and Anchor Browser. [Source](https://blog.cloudflare.com/signed-agents/), read 2026-11-04.
-   Primary source Google-Agent is used by agents hosted on Google infrastructure to navigate and act on user request, from user-triggered-agents.json ranges. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-11-04.
-   Primary source Meta-ExternalFetcher may bypass robots.txt because it fetches links at a user's request. [Source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), read 2026-11-04.
-   Secondary source Verisoul reports Comet browses from the user's device with Chrome's user agent, no signature and the user's IP. [Source](https://www.verisoul.ai/agents/perplexity-comet), read 2026-11-04.

## Related

-   [All task agents](https://agentreadycrawlers.com/crawlers/purpose/task-agent/)
-   [Comet profile](https://agentreadycrawlers.com/crawlers/perplexity-comet/)
-   [Google-Agent profile](https://agentreadycrawlers.com/crawlers/google-agent/)
-   [Browsing agent](https://agentreadycrawlers.com/glossary/browsing-agent/)
-   [User-triggered fetcher](https://agentreadycrawlers.com/glossary/user-triggered-fetcher/)
-   [Agentic browsers and how they identify](https://agentreadycrawlers.com/crawlers/agentic-browsers/)

## Sources

-   [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04
-   [The age of agents: cryptographically recognizing agent traffic](https://blog.cloudflare.com/signed-agents/) read 2026-11-04
-   [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04
-   [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04
-   [verisoul.ai: perplexity comet](https://www.verisoul.ai/agents/perplexity-comet) read 2026-11-04

Last reviewed 2026-11-04