Glossary
Robots.txt token
In short
The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests, and some tokens control use rather than crawling. 47 Index rows have a token and 23 have none. 32.1 percent of top-10,000 robots.txt files name at least one AI crawler token. Example: Google-Extended has no HTTP user agent string.
Definition
The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests, and some tokens control use rather than crawling.
Sourced example
Google-Extended has no HTTP user agent string; it is a control token read against Googlebot crawls. Source, read 2026-11-04.
How it shows up here
47 Index rows have a token and 23 have none. 32.1 percent of top-10,000 robots.txt files name at least one AI crawler token.
Previous: robots.txt. Next: Search-index crawler. All 41 terms.
Sources
Last reviewed 2026-11-04