Search index crawler from Anthropic

Claude-SearchBot

In short

Claude-SearchBot is Anthropic's search index crawler. The token to write in robots.txt is Claude-SearchBot. Check a visitor against the operator's published IP list. 7.7 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results.

Operator

Anthropic Vendor profile: Anthropic

Purpose

Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.

User-agent string

No exact string is published. What the operator or a third party says: Contains "Claude-SearchBot"; no full string published

robots.txt token

Claude-SearchBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Yes, per operator

Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges

claude.com/crawling/bots.json

Rate behaviour

Respects Crawl-delay where appropriate, per Anthropic.

Block rate

7.7% of 5,938 top-10,000 sites with a parsed robots.txt disallow Claude-SearchBot at the root (460 sites). 4.1% name it; 3.6% refuse it only through a * group. Rank 27 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 5.5% (2.9% named). Crawl of 2026-11-04; report.

Documentation

support.claude.com, read 2026-11-04.

Note. Search-index crawler, separate from training.

Allow and refuse lines

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-SearchBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Anthropic in the Verification Ledger

One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. All operators.

IP lists

• claude.com/crawling/bots.json

Reverse DNS

publishes nothing

Key directory

publishes nothing

Cloudflare

not confirmed on Radar (directory slug not found, 2026-11-04)

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

In the news

• 2026-10-20Anthropic refreshes its crawler IP list

Other Anthropic rows

• ClaudeBot Training token ClaudeBot
• Claude-User Live answer fetch token Claude-User
• anthropic-ai (legacy token) Training token anthropic-ai
• Claude in Chrome Task agent no token
• Claude Duet cloud browser Task agent token Claude-Agent
• Claude computer use (API tool) Task agent no token

Last reviewed 2026-11-04