Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Live answer fetcher from Anthropic

Claude-User

In short

Claude-User is Anthropic's live answer fetcher. The token to write in robots.txt is Claude-User. Check a visitor against the operator's published IP list. Unlike OpenAI and Perplexity, Anthropic says its user-triggered fetcher honours robots.txt. On robots.txt, Anthropic: Claude-User allows site owners to control which sites can be accessed through user-initiated requests; disabling it stops retrieval for user queries.

Operator
Anthropic Vendor profile: Anthropic
Purpose
Live answer fetch All live answer fetch rows. Fetches a page because a person asked an assistant about it, one request at a time.
User-agent string
No exact string is published. What the operator or a third party says: Contains "Claude-User"; no full string published
robots.txt token
Claude-User Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Anthropic: Claude-User allows site owners to control which sites can be accessed through user-initiated requests; disabling it stops retrieval for user queries. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges
claude.com/crawling/bots.json
Rate behaviour
Respects Crawl-delay where appropriate, per Anthropic.
Block rate

8.1% of 5,938 top-10,000 sites with a parsed robots.txt disallow Claude-User at the root (479 sites). 4.5% name it; 3.6% refuse it only through a * group. Rank 24 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 5.7% (3.1% named). Crawl of 2026-11-04; report.

Documentation
support.claude.com, read 2026-11-04.

Note. Unlike OpenAI and Perplexity, Anthropic says its user-triggered fetcher honours robots.txt.

Allow and refuse lines

To allow Claude-User

User-agent: Claude-User
Allow: /

To refuse Claude-User

User-agent: Claude-User
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Anthropic in the Verification Ledger

One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. All operators.

IP lists
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not confirmed on Radar (directory slug not found, 2026-11-04)
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

In the news

Other Anthropic rows

Sources

Last reviewed 2026-11-04