---
title: "Crawl-delay: definition | Agent Ready Crawlers"
description: "Crawl-delay: A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. Read 2026-11-04."
canonical: https://agentreadycrawlers.com/glossary/crawl-delay
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Glossary"
---

# Crawl-delay: definition | Agent Ready Crawlers

> Crawl-delay: A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. Read 2026-11-04.

Source: https://agentreadycrawlers.com/glossary/crawl-delay
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Glossary

# Crawl-delay

In short

A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. RFC 9309 does not define it; support varies by operator. The Index records rate behaviour as the operator documents it, in words, because operators describe it differently; the crawl does not measure Crawl-delay. Example: Anthropic supports Crawl-delay. Related: Amazonbot profile and ClaudeBot profile.

## Definition

A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. RFC 9309 does not define it; support varies by operator.

## Sourced example

Anthropic supports Crawl-delay; Amazon says its bots do not; Google ignores it. [Source](https://developer.amazon.com/amazonbot), read 2026-11-04.

## How it shows up here

The Index records rate behaviour as the operator documents it, in words, because operators describe it differently; the crawl does not measure Crawl-delay.

-   [Amazonbot profile](https://agentreadycrawlers.com/crawlers/amazonbot/)
-   [ClaudeBot profile](https://agentreadycrawlers.com/crawlers/claudebot/)

Previous: [Content-Signal](https://agentreadycrawlers.com/glossary/content-signal/). Next: [crawler-price header](https://agentreadycrawlers.com/glossary/crawler-price-header/). [All 41 terms](https://agentreadycrawlers.com/glossary/).

## Sources

-   [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04

Last reviewed 2026-11-04