Glossary
Crawl-delay
In short
A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. RFC 9309 does not define it; support varies by operator. The Index records rate behaviour as the operator documents it, in words, because operators describe it differently; the crawl does not measure Crawl-delay. Example: Anthropic supports Crawl-delay. Related: Amazonbot profile and ClaudeBot profile.
Definition
A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. RFC 9309 does not define it; support varies by operator.
Sourced example
Anthropic supports Crawl-delay; Amazon says its bots do not; Google ignores it. Source, read 2026-11-04.
How it shows up here
The Index records rate behaviour as the operator documents it, in words, because operators describe it differently; the crawl does not measure Crawl-delay.
Previous: Content-Signal. Next: crawler-price header. All 41 terms.
Sources
Last reviewed 2026-11-04