--- title: "Training crawlers and agents | Agent Ready Crawlers" description: "Training in the Crawler Index: 9 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/training site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Training crawlers and agents | Agent Ready Crawlers > Training in the Crawler Index: 9 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/training Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Training: 9 rows In short 9 rows in the Index are filed under training, from 8 operators. 5 publish nothing a site can verify and 0 have no robots.txt token. Of the 8 tokens the crawl measured, CCBot is refused by 19.4 percent of top-10,000 sites and Applebot-Extended by 14.8 percent. Collects content to train or tune a model. Refusing it is the training opt-out. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Collects content to train or tune a model. Refusing it is the training opt-out. Training: 9 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | GPTBot | Training | Yes | IP list | 18.5% | Profile | | Anthropic | ClaudeBot | Training | Yes | IP list | 17.6% | Profile | | Anthropic | anthropic-ai | Training | Legacy | None published | 15.5% | Profile | | Google | Google-Extended | Training | Yes | None published | 16.0% | Profile | | Apple | Applebot-Extended | Training | Yes | None published | 14.8% | Profile | | Amazon | Amazonbot | Training | Yes | IP list | 15.6% | Profile | | Meta | meta-externalagent | Training | Yes | None published | 16.0% | Profile | | Common Crawl | CCBot | Training | Yes | Reverse DNS | 19.4% | Profile | | Mistral AI | MistralAI-Training | Training | Yes | None published | not measured | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | CCBot | 19.4% | 15.7% | 3.7% | 14.9% | | GPTBot | 18.5% | 15.2% | 3.3% | 13.9% | | ClaudeBot | 17.6% | 14.0% | 3.5% | 13.1% | | Google-Extended | 16.0% | 12.6% | 3.4% | 12.1% | | Meta-ExternalAgent | 16.0% | 12.3% | 3.7% | 12.0% | | Amazonbot | 15.6% | 12.0% | 3.6% | 11.7% | | anthropic-ai | 15.5% | 11.7% | 3.8% | 11.6% | | Applebot-Extended | 14.8% | 11.6% | 3.1% | 11.1% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [About Applebot](https://support.apple.com/en-us/119829) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 Last reviewed 2026-11-04