--- title: "Anthropic: crawler operator profile | Agent Ready Crawlers" description: "Anthropic: crawler operator profile covering ClaudeBot, Claude-SearchBot, Claude-User and 1 more. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/anthropic site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Anthropic: crawler operator profile | Agent Ready Crawlers > Anthropic: crawler operator profile covering ClaudeBot, Claude-SearchBot, Claude-User and 1 more. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/anthropic Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Anthropic In short Anthropic sorts its crawling into three robots, ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests, plus a hosted browser that adds a Claude-Agent token to the browser's own user agent. Since 30 October 2026 it also reads an IETF draft no-training preference. It publishes no signing keys. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [anthropic.com](https://www.anthropic.com) In one line Anthropic runs ClaudeBot for training data, Claude-SearchBot for search and Claude-User for user requests under one IP file, and marks its hosted agent browsers with a Claude-Agent token and a SAIL file of egress addresses. Last read 2026-11-05 ## What Anthropic does for this layer ### From the vendor's own pages Anthropic's help article lists three robots, ClaudeBot, Claude-User and Claude-SearchBot, and says restricting ClaudeBot signals that a site's future materials should be left out of AI training datasets. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic supports the non-standard Crawl-delay extension to robots.txt and gives "Crawl-delay: 1" under "User-agent: ClaudeBot" as its example. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs on the sites they crawl. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic warns that blocking its IP addresses may not work as an opt-out, because it stops the bots from reading robots.txt, and asks for a robots.txt rule on every subdomain. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic began recognising a machine-readable "no AI training" preference on 2026-10-30, based on the IETF aipref draft draft-kelley-aipref-training-preference-signal, and is rolling it out across its services over time. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), dated 2026-10-30) Anthropic treats a robots.txt rule that names ClaudeBot as more specific than the general training preference, so the ClaudeBot rule wins for that path. ([support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) Anthropic-hosted browsers, used by Claude Duet, send the browser's own user agent with the added token Claude-Agent, and Anthropic advises matching that token rather than the full string. ([support.claude.com](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf)) Anthropic publishes the egress addresses for Claude-Agent at claude.com/.well-known/sail in the draft-kelley-webbotauth-sail-00 format and recommends updating rules from that file automatically. ([support.claude.com](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf)) ### What our crawl saw Anthropic's SAIL file listed nine IPv4 addresses in an agentic-browser-use prefix set marked exclusive, modified 2026-10-22, with expires\_after set to 3600 seconds, when fetched on 2026-11-05. ([claude.com](https://claude.com/.well-known/sail), dated 2026-10-22) Anthropic's bots.json, the range file for its three robots, listed 14 IPv4 prefixes with a creationTime of 2026-10-20 when fetched on 2026-11-05. ([claude.com](https://claude.com/crawling/bots.json), dated 2026-10-20) Anthropic served no Web Bot Auth key directory on 2026-11-05: the well-known path returned HTTP 404 on claude.com, claude.ai and www.anthropic.com. ([claude.com](https://claude.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | ClaudeBot | Crawler that collects web content that may contribute to model training. | No price recorded | | Claude-SearchBot | Crawler that indexes content to improve search results for Claude users. | No price recorded | | Claude-User | Fetcher that retrieves pages when a Claude user asks a question. | No price recorded | | Claude-Agent hosted browser | Browser on Anthropic infrastructure that Claude Duet drives for users; identified by the Claude-Agent token and the SAIL IP file. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Anthropic's bots honour robots.txt directives and Crawl-delay; Anthropic names robots.txt as the opt-out. (support.claude.com) | | Web Bot Auth | No statement | No key directory on claude.com, claude.ai or anthropic.com (observed 404, 2026-11-05) and no mention in either help article. | | Content Signals | No statement | Anthropic cites the IETF aipref training signal, not Content-Signals (read 2026-11-05). | | Published IP list | Supports | bots.json for the three robots and a SAIL file for Claude-Agent. (support.claude.com) | | Reverse DNS check | No statement | Neither help article gives a reverse DNS pattern (read 2026-11-05). | | Cloudflare verified bots | No statement | Not mentioned in Anthropic's help articles (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | 404 on claude.com, claude.ai and www.anthropic.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | bots.json | 14 IPv4 prefixes, creationTime 2026-10-20 | GET and count prefixes | 2026-11-05 | | SAIL file | 9 IPv4 addresses for Claude-Agent, modified 2026-10-22, expires_after 3600 | GET and read the TOML | 2026-11-05 | ## What Anthropic does not do - Anthropic publishes no signing keys for Claude-Agent, so the token and the SAIL address file are the only checks a site can run. - Anthropic's hosted-browser article covers Claude Duet's hosted browser only and gives no identifier for Claude in Chrome (read 2026-11-05). ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Anthropic in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/anthropic/) 7 rows - [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) token ClaudeBot - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) no token ## Sources - [anthropic.com](https://www.anthropic.com) read 2026-11-05 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-05 - [How do Anthropic-hosted browsers identify themselves when they browse the web on a user's behalf?](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf) read 2026-11-05 - [Anthropic SAIL file (Claude-Agent egress IPs)](https://claude.com/.well-known/sail) read 2026-11-05 - [Anthropic bots.json IP ranges](https://claude.com/crawling/bots.json) read 2026-11-05 - [claude.com Web Bot Auth key directory path (GET probe)](https://claude.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05