robots.txt builder

In short

Choose a policy for each purpose and copy a robots.txt that names 47 tokens from the Crawler Index, one group per purpose. 23 rows in the Index have no token, so no robots.txt can address them. The search group includes Googlebot and Bingbot, so refusing it removes pages from those search results.

The default below refuses training, allows everything else and adds no sitemap or Content-Signal line. Change a purpose to Allow or Refuse and the file updates on the page. Without JavaScript the default file is still complete in the box.

Choose a policy for each purpose

Training

Collects content to train or tune a model. Refusing it is the training opt-out. 9 tokens in the Index.

Search index

Builds a search index that sends visitors back. Refusing it removes pages from that product. 15 tokens in the Index.

Live answer fetch

Fetches a page because a person asked an assistant about it, one request at a time. 7 tokens in the Index.

Task agent

A browsing agent that clicks, fills forms and finishes a job for a person. 1 token in the Index.

Shopping agent

An agent that compares, selects or buys goods for a person. 0 tokens in the Index.

Other

Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes. 15 tokens in the Index.

Sitemap URL (optional)

Add a Content-Signal line to the * group, set from the choices above

Your robots.txt

# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04.
# Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none.

# Training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Search index: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Storebot-Google
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: DuckDuckBot
User-agent: MistralAI-Index
User-agent: YouBot
User-agent: Timpibot
User-agent: PetalBot
User-agent: ExaSearchBot
Allow: /

# Live answer fetch: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /

# Task agents: allowed
User-agent: Claude-Agent
Allow: /

# Other: allowed
User-agent: OAI-AdsBot
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: MicrosoftPreview
User-agent: meta-externalads
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: FirecrawlAgent
User-agent: OraBot
User-agent: agent-readiness-crawl
Allow: /

# Everything else
User-agent: *
Allow: /

What this file cannot do

It cannot address the 23 rows with no token, such as browsing agents that run in a visitor's own browser. See crawler versus browsing agent.
It states a preference. Operators of user-triggered fetchers say robots.txt may not apply to them; see live answer fetchers, and enforce at the WAF.
Google-Extended and Applebot-Extended are control tokens: refusing them opts out of training while search crawling continues. How to allow search and refuse training.
The Content-Signal line is an extension from the Content Signals Policy, not part of RFC 9309. Crawlers that do not read it ignore it. What Content-Signals changes.

Tokens the builder writes

Training: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, MistralAI-Training
Search index: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Storebot-Google, Applebot, bingbot, Amzn-SearchBot, meta-webindexer, DuckDuckBot, MistralAI-Index, YouBot, Timpibot, PetalBot, ExaSearchBot
Live answer fetch: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, meta-externalfetcher, DuckAssistBot, MistralAI-User
Task agent: Claude-Agent
Shopping agent: None with a token
Other: OAI-AdsBot, Google-CloudVertexBot, GoogleOther, MicrosoftPreview, meta-externalads, facebookexternalhit, FacebookBot, Bytespider, cohere-ai, Diffbot, omgili, ImagesiftBot, FirecrawlAgent, OraBot, agent-readiness-crawl

The search group holds OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot and 11 more.

How the file is tested

A test in this site's repository builds the file for several combinations of choices and checks every line against the grammar of RFC 9309 section 2: product tokens, groups that start with a user-agent line, rules that begin with a slash, and an absolute sitemap URL. The two extension lines, Sitemap and Content-Signal, are checked separately. The test runs on every build.

Sources

RFC 9309: Robots Exclusion Protocol read 2026-11-04
Giving users choice with Cloudflare's new Content Signals Policy read 2026-11-04
Overview of OpenAI Crawlers read 2026-11-04
Crawler Index (this site) read 2026-11-04

Last reviewed 2026-11-04