Crawler operator
Anthropic
In short
Anthropic sorts its crawling into three robots, ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests, plus a hosted browser that adds a Claude-Agent token to the browser's own user agent. Since 30 October 2026 it also reads an IETF draft no-training preference. It publishes no signing keys.
- Type
- Crawler operator
- Homepage
- anthropic.com
- In one line
- Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses.
- Last read
- 2026-11-05
What Anthropic does for this layer
From the vendor's own pages
Anthropic's help article lists three robots, ClaudeBot, Claude-User and Claude-SearchBot, and says restricting ClaudeBot signals that a site's future materials should be left out of AI training datasets. (support.claude.com) Anthropic supports the non-standard Crawl-delay extension to robots.txt and gives "Crawl-delay: 1" under "User-agent: ClaudeBot" as its example. (support.claude.com) Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs on the sites they crawl. (support.claude.com) Anthropic warns that blocking its IP addresses may not work as an opt-out, because it stops the bots from reading robots.txt, and asks for a robots.txt rule on every subdomain. (support.claude.com)
Anthropic began recognising a machine-readable "no AI training" preference on 2026-10-30, based on the IETF aipref draft draft-kelley-aipref-training-preference-signal, and is rolling it out across its services over time. (support.claude.com, dated 2026-10-30) Anthropic treats a robots.txt rule that names ClaudeBot as more specific than the general training preference, so the ClaudeBot rule wins for that path. (support.claude.com) Anthropic-hosted browsers, used by Claude Duet, send the browser's own user agent with the added token Claude-Agent, and Anthropic advises matching that token rather than the full string. (support.claude.com) Anthropic publishes the egress addresses for Claude-Agent at claude.com/.well-known/sail in the draft-kelley-webbotauth-sail-00 format and recommends updating rules from that file automatically. (support.claude.com)
What our crawl saw
Anthropic's SAIL file listed nine IPv4 addresses in an agentic-browser-use prefix set marked exclusive, modified 2026-10-22, with expires_after set to 3600 seconds, when fetched on 2026-11-05. (claude.com, dated 2026-10-22) Anthropic's bots.json, the range file for its three robots, listed 14 IPv4 prefixes with a creationTime of 2026-10-20 when fetched on 2026-11-05. (claude.com, dated 2026-10-20) Anthropic served no Web Bot Auth key directory on 2026-11-05: the well-known path returned HTTP 404 on claude.com, claude.ai and www.anthropic.com. (claude.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| ClaudeBot | Crawler that collects web content that may contribute to model training. | |||
| Claude-SearchBot | Crawler that indexes content to improve search results for Claude users. | |||
| Claude-User | Fetcher that retrieves pages when a Claude user asks a question. | |||
| Claude-Agent hosted browser | Browser on Anthropic infrastructure that Claude Duet drives for users; identified by the Claude-Agent token and the SAIL IP file. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Anthropic's bots honour robots.txt directives and Crawl-delay; Anthropic names robots.txt as the opt-out. (support.claude.com) |
| Web Bot Auth | No statement | No key directory on claude.com, claude.ai or anthropic.com (observed 404, 2026-11-05) and no mention in either help article. |
| Content Signals | No statement | Anthropic cites the IETF aipref training signal, not Content-Signals (read 2026-11-05). |
| Published IP list | Supports | bots.json for the three robots and a SAIL file for Claude-Agent. (support.claude.com) |
| Reverse DNS check | No statement | Neither help article gives a reverse DNS pattern (read 2026-11-05). |
| Cloudflare verified bots | No statement | Not mentioned in Anthropic's help articles (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 on claude.com, claude.ai and www.anthropic.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| bots.json | 14 IPv4 prefixes, creationTime 2026-10-20 | GET and count prefixes | 2026-11-05 |
| SAIL file | 9 IPv4 addresses for Claude-Agent, modified 2026-10-22, expires_after 3600 | GET and read the TOML | 2026-11-05 |
What Anthropic does not do
- Anthropic publishes no signing keys for Claude-Agent, so the token and the SAIL address file are the only checks a site can run.
- Anthropic's hosted-browser article covers Claude Duet's hosted browser only and gives no identifier for Claude in Chrome (read 2026-11-05).
Related pages on this site
Sources
- anthropic.com
- Does Anthropic crawl data from the web, and how can site owners block the crawler?
- How do Anthropic-hosted browsers identify themselves when they browse the web on a user's behalf?
- Anthropic SAIL file (Claude-Agent egress IPs)
- Anthropic bots.json IP ranges
- claude.com Web Bot Auth key directory path (GET probe)
Last reviewed 2026-11-05