Training crawler from Anthropic

ClaudeBot

In short

ClaudeBot is Anthropic's training crawler. The token to write in robots.txt is ClaudeBot. Check a visitor against the operator's published IP list. Supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" then "Crawl-delay: 1". ClaudeBot: its bots honour robots.txt "do not crawl" directives; restricting ClaudeBot signals that future materials should be excluded from training datasets.

Operator

Anthropic Vendor profile: Anthropic

Purpose

Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.

User-agent string

No exact string is published. What the operator or a third party says: Contains "ClaudeBot"; Anthropic does not publish a full example string on its help page

robots.txt token

ClaudeBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Yes, per operator

Anthropic: its bots honour robots.txt "do not crawl" directives; restricting ClaudeBot signals that future materials should be excluded from training datasets. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges

claude.com/crawling/bots.json

Rate behaviour

Supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" then "Crawl-delay: 1". Anthropic says it will not attempt to bypass CAPTCHAs.

Block rate

17.6% of 5,938 top-10,000 sites with a parsed robots.txt disallow ClaudeBot at the root (1,044 sites). 14.0% name it; 3.5% refuse it only through a * group. Rank 4 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 13.1% (10.6% named). Crawl of 2026-11-04; report.

Documentation

support.claude.com, read 2026-11-04.

Note. Anthropic warns that IP blocking may not work as an opt-out because it stops its bots reading robots.txt. bots.json creationTime read: 2026-10-20 (includes 216.73.216.0/22). Help article dated 2026-04-07.

Allow and refuse lines

User-agent: ClaudeBot
Allow: /

User-agent: ClaudeBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Anthropic in the Verification Ledger

One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. All operators.

IP lists

• claude.com/crawling/bots.json

Reverse DNS

publishes nothing

Key directory

publishes nothing

Cloudflare

not confirmed on Radar (directory slug not found, 2026-11-04)

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

In the news

• 2026-10-20Anthropic refreshes its crawler IP list

Other Anthropic rows

• Claude-User Live answer fetch token Claude-User
• Claude-SearchBot Search index token Claude-SearchBot
• anthropic-ai (legacy token) Training token anthropic-ai
• Claude in Chrome Task agent no token
• Claude Duet cloud browser Task agent token Claude-Agent
• Claude computer use (API tool) Task agent no token

Last reviewed 2026-11-04