--- title: "Robots.txt token: definition | Agent Ready Crawlers" description: "Robots.txt token: The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests." canonical: https://agentreadycrawlers.com/glossary/robots-txt-token site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Glossary" --- # Robots.txt token: definition | Agent Ready Crawlers > Robots.txt token: The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests. Source: https://agentreadycrawlers.com/glossary/robots-txt-token Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Glossary # Robots.txt token In short The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests, and some tokens control use rather than crawling. 47 Index rows have a token and 23 have none. 32.1 percent of top-10,000 robots.txt files name at least one AI crawler token. Example: Google-Extended has no HTTP user agent string. ## Definition The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests, and some tokens control use rather than crawling. ## Sourced example Google-Extended has no HTTP user agent string; it is a control token read against Googlebot crawls. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. ## How it shows up here 47 Index rows have a token and 23 have none. 32.1 percent of top-10,000 robots.txt files name at least one AI crawler token. - [Tokens versus user-agent strings](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings/) - [Crawler Index](https://agentreadycrawlers.com/crawlers/) Previous: [robots.txt](https://agentreadycrawlers.com/glossary/robots-txt/). Next: [Search-index crawler](https://agentreadycrawlers.com/glossary/search-index-crawler/). [All 41 terms](https://agentreadycrawlers.com/glossary/). ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 Last reviewed 2026-11-04