Training crawler from Anthropic

anthropic-ai (legacy token)

In short

anthropic-ai (legacy token) is Anthropic's training crawler. The token to write in robots.txt is anthropic-ai. The operator publishes nothing a site can check. 15.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, not listed on Anthropic's current crawler page, which names only ClaudeBot, Claude-User and Claude-SearchBot (checked 2026-11-04).

Operator

Anthropic Vendor profile: Anthropic

Purpose

Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.

User-agent string

No exact string is published. What the operator or a third party says: No current request string; legacy robots.txt token only

robots.txt token

anthropic-ai Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Not on current operator page

Not listed on Anthropic's current crawler page, which names only ClaudeBot, Claude-User and Claude-SearchBot (checked 2026-11-04). Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges

Not published.

Rate behaviour

Not documented.

Block rate

15.5% of 5,938 top-10,000 sites with a parsed robots.txt disallow anthropic-ai at the root (918 sites). 11.7% name it; 3.8% refuse it only through a * group. Rank 10 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 11.6% (8.9% named). Crawl of 2026-11-04; report.

Documentation

support.claude.com, read 2026-11-04.

Note. Still common in published robots.txt files; the shared crawl tests it. A Disallow for anthropic-ai alone does not cover ClaudeBot.

Allow and refuse lines

User-agent: anthropic-ai
Allow: /

User-agent: anthropic-ai
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Anthropic in the Verification Ledger

One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. All operators.

IP lists

• claude.com/crawling/bots.json

Reverse DNS

publishes nothing

Key directory

publishes nothing

Cloudflare

not confirmed on Radar (directory slug not found, 2026-11-04)

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

Other Anthropic rows

• ClaudeBot Training token ClaudeBot
• Claude-User Live answer fetch token Claude-User
• Claude-SearchBot Search index token Claude-SearchBot
• Claude in Chrome Task agent no token
• Claude Duet cloud browser Task agent token Claude-Agent
• Claude computer use (API tool) Task agent no token

Last reviewed 2026-11-04