--- title: "ClaudeBot: user agent, token, block rate | Agent Ready Crawlers" description: "ClaudeBot from Anthropic: robots.txt token ClaudeBot, training purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/claudebot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # ClaudeBot: user agent, token, block rate | Agent Ready Crawlers > ClaudeBot from Anthropic: robots.txt token ClaudeBot, training purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/claudebot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from Anthropic # ClaudeBot In short ClaudeBot is Anthropic's training crawler. The token to write in robots.txt is ClaudeBot. Check a visitor against the operator's published IP list. Supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" then "Crawl-delay: 1". ClaudeBot: its bots honour robots.txt "do not crawl" directives; restricting ClaudeBot signals that future materials should be excluded from training datasets. Operator [Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string No exact string is published. What the operator or a third party says: Contains "ClaudeBot"; Anthropic does not publish a full example string on its help page robots.txt token `ClaudeBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Anthropic: its bots honour robots.txt "do not crawl" directives; restricting ClaudeBot signals that future materials should be excluded from training datasets. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://claude.com/crawling/bots.json), read 2026-11-04. IP ranges [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Rate behaviour Supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" then "Crawl-delay: 1". Anthropic says it will not attempt to bypass CAPTCHAs. Block rate **17.6%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `ClaudeBot` at the root (1,044 sites). 14.0% name it; 3.5% refuse it only through a \* group. Rank 4 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 13.1% (10.6% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. Note. Anthropic warns that IP blocking may not work as an opt-out because it stops its bots reading robots.txt. bots.json creationTime read: 2026-10-20 (includes 216.73.216.0/22). Help article dated 2026-04-07. ## Allow and refuse lines ### To allow ClaudeBot User-agent: ClaudeBot Allow: / ### To refuse ClaudeBot User-agent: ClaudeBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Anthropic in the Verification Ledger One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) Reverse DNS publishes nothing Key directory publishes nothing Cloudflare not confirmed on Radar (directory slug not found, 2026-11-04) Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) ## Other Anthropic rows - [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User - [Claude-SearchBot](https://agentreadycrawlers.com/crawlers/claude-searchbot/) Search index token Claude-SearchBot - [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai - [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token - [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent - [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token ## Sources - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 Last reviewed 2026-11-04