---
title: "Allow search, refuse training | Agent Ready Crawlers"
description: "How to allow search and answer bots while refusing training: the tokens to name, the control tokens that never crawl, and the Bing gap."
canonical: https://agentreadycrawlers.com/guides/allow-search-refuse-training
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
date: 2026-11-04
section: "Guides"
---

# Allow search, refuse training | Agent Ready Crawlers

> How to allow search and answer bots while refusing training: the tokens to name, the control tokens that never crawl, and the Bing gap.

Source: https://agentreadycrawlers.com/guides/allow-search-refuse-training
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

# How to allow search and answer bots while refusing training

In short

Name the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) in Disallow groups and leave search and user fetchers allowed. OpenAI treats OAI-SearchBot and GPTBot as independent settings: a site can allow search while disallowing training. Google-Extended controls Gemini training and grounding and does not affect Search inclusion or ranking.

Operators have split search from training into separate tokens. Most of the work is naming the training tokens and leaving the search and fetch tokens alone.

## In practice

1.  Name the training tokens in one Disallow group: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot. The robots.txt builder writes this group for you.
2.  Leave the search tokens allowed. OpenAI treats OAI-SearchBot and GPTBot as independent settings, so a site can appear in ChatGPT search while refusing training.
3.  Know which tokens never crawl. Google-Extended and Applebot-Extended are control tokens: refusing them opts out of training while pages stay in Google Search and Apple search.
4.  Split Amazon the same way. Amazonbot may train; Amzn-SearchBot and Amzn-User are not used for training.
5.  Plan for the Bing gap. Bing has no robots.txt training token yet; NOARCHIVE is today's opt-out, and Cloudflare says robots.txt support is targeted for early 2027.
6.  If you use Cloudflare, its Disallow AI Training setting (2026-09-15) writes the no-training preference into robots.txt and keeps Accountable mixed-use crawlers in search.

### Training tokens in the Index, refused

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

Generated from the Index: 9 tokens with the purpose Training. Search and fetch tokens are left out on purpose.

## What the sources say

-   Primary source OpenAI treats OAI-SearchBot and GPTBot as independent settings: a site can allow search while disallowing training. [Source](https://platform.openai.com/docs/bots), read 2026-11-04.
-   Primary source Google-Extended controls Gemini training and grounding and does not affect Search inclusion or ranking. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04.
-   Primary source Applebot-Extended does not crawl; disallowing it opts out of training while pages stay in Apple search. [Source](https://support.apple.com/en-us/119829), read 2026-11-04.
-   Primary source Amazon separates Amazonbot (may train) from Amzn-SearchBot and Amzn-User (not used for training). [Source](https://developer.amazon.com/amazonbot), read 2026-11-04.
-   Primary source Bing has no robots.txt training token yet; NOARCHIVE is today's opt-out and robots.txt support is targeted for early 2027, per Cloudflare. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04.
-   Primary source Cloudflare's Disallow AI Training (2026-09-15) writes the no-training preference into robots.txt and keeps Accountable mixed-use crawlers in search. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04.

## Related

-   [robots.txt builder](https://agentreadycrawlers.com/robots-builder/)
-   [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/)
-   [Google-Extended profile](https://agentreadycrawlers.com/crawlers/google-extended/)
-   [Disallow AI Training](https://agentreadycrawlers.com/glossary/disallow-ai-training/)
-   [Which AI bots to allow, by type of site](https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type/)
-   [Training and search policy in robots.txt](https://agentreadycrawlers.com/block-rates/training-and-search/)

## Sources

-   [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04
-   [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04
-   [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04
-   [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04
-   [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04

Last reviewed 2026-11-04