Glossary
robots.txt
In short
A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Standardised as RFC 9309; it states a preference and enforces nothing. The crawl parsed a robots.txt from 79.2 percent of the 10,886 domains that answered, and 67.4 percent of top-10,000 files carry a Sitemap line.
Definition
A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Standardised as RFC 9309; it states a preference and enforces nothing.
Sourced example
Anthropic documents "User-agent: ClaudeBot" followed by "Disallow: /" to block ClaudeBot site-wide, per subdomain. Source, read 2026-11-04.
How it shows up here
The crawl parsed a robots.txt from 79.2 percent of the 10,886 domains that answered, and 67.4 percent of top-10,000 files carry a Sitemap line.
Previous: RFC 9309. Next: Robots.txt token. All 41 terms.
Sources
Last reviewed 2026-11-04