Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

How robots.txt tokens differ from user-agent strings

In short

The robots.txt token is what you write after User-agent:, the user-agent string is what arrives in the request; some tokens never appear in logs. Google-Extended has no HTTP user agent string; crawling uses existing Google strings and the token is a control only. Applebot-Extended does not crawl webpages; it only governs how Applebot data is used.

The word after User-agent in robots.txt is not what you see in the logs, and some tokens never appear there at all.

In practice

  1. Write the token in robots.txt and search the logs for the string. Meta's examples use lowercase tokens such as meta-externalagent, and RFC 9309 matching of the product token is case-insensitive.
  2. Expect control tokens to be invisible. Google-Extended has no HTTP user agent string; crawling uses existing Google strings. Applebot-Extended does not crawl webpages; it only governs how Applebot data is used.
  3. Do not assume the token is in the header. Firecrawl matches the token FirecrawlAgent in robots.txt but tells site owners not to assume it is the User-Agent header on the fetch.
  4. Remember some crawlers have more than one token. Google says a rule needs to match only one of them.
  5. Use the Index's two columns: the token column is for robots.txt, the profile's user-agent string is for your logs.

What the sources say

Sources

Last reviewed 2026-11-04