--- title: "robots.txt builder for AI crawlers | Agent Ready Crawlers" description: "Build a robots.txt from the Crawler Index: allow or refuse training, search, answer and agent tokens, then copy the file." canonical: https://agentreadycrawlers.com/robots-builder site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Tools" --- # robots.txt builder for AI crawlers | Agent Ready Crawlers > Build a robots.txt from the Crawler Index: allow or refuse training, search, answer and agent tokens, then copy the file. Source: https://agentreadycrawlers.com/robots-builder Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # robots.txt builder In short Choose a policy for each purpose and copy a robots.txt that names 47 tokens from the Crawler Index, one group per purpose. 23 rows in the Index have no token, so no robots.txt can address them. The search group includes Googlebot and Bingbot, so refusing it removes pages from those search results. The default below refuses training, allows everything else and adds no sitemap or Content-Signal line. Change a purpose to Allow or Refuse and the file updates on the page. Without JavaScript the default file is still complete in the box. Your robots.txt \# robots.txt built at agentreadycrawlers.com from the Crawler Index, edition 2026-11-04. # Only agents that publish a robots.txt token can be addressed here. Browsing agents that run in a visitor's own browser have none. # Training: refused User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: meta-externalagent User-agent: CCBot User-agent: MistralAI-Training Disallow: / # Search index: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Storebot-Google User-agent: Applebot User-agent: bingbot User-agent: Amzn-SearchBot User-agent: meta-webindexer User-agent: DuckDuckBot User-agent: MistralAI-Index User-agent: YouBot User-agent: Timpibot User-agent: PetalBot User-agent: ExaSearchBot Allow: / # Live answer fetch: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: DuckAssistBot User-agent: MistralAI-User Allow: / # Task agents: allowed User-agent: Claude-Agent Allow: / # Other: allowed User-agent: OAI-AdsBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: MicrosoftPreview User-agent: meta-externalads User-agent: facebookexternalhit User-agent: FacebookBot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: FirecrawlAgent User-agent: OraBot User-agent: agent-readiness-crawl Allow: / # Everything else User-agent: \* Allow: / ## What this file cannot do - It cannot address the 23 rows with no token, such as browsing agents that run in a visitor's own browser. See [crawler versus browsing agent](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent/). - It states a preference. Operators of user-triggered fetchers say robots.txt may not apply to them; see [live answer fetchers](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/), and enforce at the WAF. - Google-Extended and Applebot-Extended are control tokens: refusing them opts out of training while search crawling continues. [How to allow search and refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/). - The Content-Signal line is an extension from the Content Signals Policy, not part of RFC 9309. Crawlers that do not read it ignore it. [What Content-Signals changes](https://agentreadycrawlers.com/guides/what-content-signals-change/). ## Tokens the builder writes Training GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, MistralAI-Training Search index OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Storebot-Google, Applebot, bingbot, Amzn-SearchBot, meta-webindexer, DuckDuckBot, MistralAI-Index, YouBot, Timpibot, PetalBot, ExaSearchBot Live answer fetch ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, meta-externalfetcher, DuckAssistBot, MistralAI-User Task agent Claude-Agent Shopping agent None with a token Other OAI-AdsBot, Google-CloudVertexBot, GoogleOther, MicrosoftPreview, meta-externalads, facebookexternalhit, FacebookBot, Bytespider, cohere-ai, Diffbot, omgili, ImagesiftBot, FirecrawlAgent, OraBot, agent-readiness-crawl The search group holds OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot and 11 more. ## How the file is tested A test in this site's repository builds the file for several combinations of choices and checks every line against the grammar of RFC 9309 section 2: product tokens, groups that start with a user-agent line, rules that begin with a slash, and an absolute sitemap URL. The two extension lines, Sitemap and Content-Signal, are checked separately. The test runs on every build. ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Crawler Index (this site)](https://agentreadycrawlers.com/crawlers/) read 2026-11-04 Last reviewed 2026-11-04