---
title: "Claude-SearchBot: user agent and token | Agent Ready Crawlers"
description: "Claude-SearchBot from Anthropic: robots.txt token Claude-SearchBot, search index purpose and what a site can check; no exact user-agent string is published."
canonical: https://agentreadycrawlers.com/crawlers/claude-searchbot
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Crawlers"
---

# Claude-SearchBot: user agent and token | Agent Ready Crawlers

> Claude-SearchBot from Anthropic: robots.txt token Claude-SearchBot, search index purpose and what a site can check; no exact user-agent string is published.

Source: https://agentreadycrawlers.com/crawlers/claude-searchbot
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Search index crawler from Anthropic

# Claude-SearchBot

In short

Claude-SearchBot is Anthropic's search index crawler. The token to write in robots.txt is Claude-SearchBot. Check a visitor against the operator's published IP list. 7.7 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results.

Operator

[Anthropic](https://agentreadycrawlers.com/crawlers/operator/anthropic/) Vendor profile: [Anthropic](https://agentreadycrawlers.com/vendors/anthropic/)

Purpose

Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product.

User-agent string

No exact string is published. What the operator or a third party says: Contains "Claude-SearchBot"; no full string published

robots.txt token

`Claude-SearchBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Yes, per operator

Anthropic: disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility in user search results. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. [Documentation](https://claude.com/crawling/bots.json), read 2026-11-04.

IP ranges

[claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)

Rate behaviour

Respects Crawl-delay where appropriate, per Anthropic.

Block rate

**7.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Claude-SearchBot` at the root (460 sites). 4.1% name it; 3.6% refuse it only through a \* group. Rank 27 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 5.5% (2.9% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/).

Documentation

[support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04.

Note. Search-index crawler, separate from training.

## Allow and refuse lines

### To allow Claude-SearchBot

User-agent: Claude-SearchBot
Allow: /

### To refuse Claude-SearchBot

User-agent: Claude-SearchBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

## Anthropic in the Verification Ledger

One shared IP list for all three bots; no key directory at claude.ai or claude.com (404, 2026-11-04); nothing for Claude in Chrome. [All operators](https://agentreadycrawlers.com/verification/).

IP lists

-   [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)

Reverse DNS

publishes nothing

Key directory

publishes nothing

Cloudflare

not confirmed on Radar (directory slug not found, 2026-11-04)

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

## In the news

-   2026-10-20[Anthropic refreshes its crawler IP list](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/)

## Other Anthropic rows

-   [ClaudeBot](https://agentreadycrawlers.com/crawlers/claudebot/) Training token ClaudeBot
-   [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) Live answer fetch token Claude-User
-   [anthropic-ai (legacy token)](https://agentreadycrawlers.com/crawlers/anthropic-ai/) Training token anthropic-ai
-   [Claude in Chrome](https://agentreadycrawlers.com/crawlers/claude-in-chrome/) Task agent no token
-   [Claude Duet cloud browser](https://agentreadycrawlers.com/crawlers/claude-duet/) Task agent token Claude-Agent
-   [Claude computer use (API tool)](https://agentreadycrawlers.com/crawlers/claude-computer-use/) Task agent no token

## Sources

-   [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04
-   [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04

Last reviewed 2026-11-04