---
title: "Tokens versus user-agent strings | Agent Ready Crawlers"
description: "How robots.txt tokens differ from user-agent strings: what you write in robots.txt, what arrives in the request, and tokens that never appear in logs."
canonical: https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
date: 2026-11-04
section: "Guides"
---

# Tokens versus user-agent strings | Agent Ready Crawlers

> How robots.txt tokens differ from user-agent strings: what you write in robots.txt, what arrives in the request, and tokens that never appear in logs.

Source: https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

# How robots.txt tokens differ from user-agent strings

In short

The robots.txt token is what you write after User-agent:, the user-agent string is what arrives in the request; some tokens never appear in logs. Google-Extended has no HTTP user agent string; crawling uses existing Google strings and the token is a control only. Applebot-Extended does not crawl webpages; it only governs how Applebot data is used.

The word after User-agent in robots.txt is not what you see in the logs, and some tokens never appear there at all.

## In practice

1.  Write the token in robots.txt and search the logs for the string. Meta's examples use lowercase tokens such as meta-externalagent, and RFC 9309 matching of the product token is case-insensitive.
2.  Expect control tokens to be invisible. Google-Extended has no HTTP user agent string; crawling uses existing Google strings. Applebot-Extended does not crawl webpages; it only governs how Applebot data is used.
3.  Do not assume the token is in the header. Firecrawl matches the token FirecrawlAgent in robots.txt but tells site owners not to assume it is the User-Agent header on the fetch.
4.  Remember some crawlers have more than one token. Google says a rule needs to match only one of them.
5.  Use the Index's two columns: the token column is for robots.txt, the profile's user-agent string is for your logs.

## What the sources say

-   Primary source Google-Extended has no HTTP user agent string; crawling uses existing Google strings and the token is a control only. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04.
-   Primary source Applebot-Extended does not crawl webpages; it only governs how Applebot data is used. [Source](https://support.apple.com/en-us/119829), read 2026-11-04.
-   Primary source Firecrawl matches the token FirecrawlAgent in robots.txt but tells site owners not to assume it is the User-Agent header on the fetch. [Source](https://docs.firecrawl.dev/advanced-scraping-guide), read 2026-11-04.
-   Primary source Meta's robots.txt examples use lowercase tokens such as meta-externalagent; RFC 9309 matching of the product token is case-insensitive. [Source](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04.
-   Primary source Google says some crawlers have more than one token and a rule needs to match only one of them. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04.

## Related

-   [Google-Extended profile](https://agentreadycrawlers.com/crawlers/google-extended/)
-   [Applebot-Extended profile](https://agentreadycrawlers.com/crawlers/applebot-extended/)
-   [FirecrawlAgent profile](https://agentreadycrawlers.com/crawlers/firecrawl/)
-   [robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/)
-   [User-agent string](https://agentreadycrawlers.com/glossary/user-agent-string/)

## Sources

-   [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04
-   [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04
-   [docs.firecrawl.dev: advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide) read 2026-11-04
-   [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04

Last reviewed 2026-11-04