Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training: 9 rows

In short

9 rows in the Index are filed under training, from 8 operators. 5 publish nothing a site can verify and 0 have no robots.txt token. Of the 8 tokens the crawl measured, CCBot is refused by 19.4 percent of top-10,000 sites and Applebot-Extended by 14.8 percent. Collects content to train or tune a model. Refusing it is the training opt-out.

Collects content to train or tune a model. Refusing it is the training opt-out.

Training: 9 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
OpenAI GPTBot Training Yes IP list 18.5% Profile
Anthropic ClaudeBot Training Yes IP list 17.6% Profile
Anthropic anthropic-ai Training Legacy None published 15.5% Profile
Google Google-Extended Training Yes None published 16.0% Profile
Apple Applebot-Extended Training Yes None published 14.8% Profile
Amazon Amazonbot Training Yes IP list 15.6% Profile
Meta meta-externalagent Training Yes None published 16.0% Profile
Common Crawl CCBot Training Yes Reverse DNS 19.4% Profile
Mistral AI MistralAI-Training Training Yes None published not measured Profile

Block rates in this purpose

Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via *" counts sites that refuse it only because their wildcard group refuses everything.

TokenRefusedNamedVia *All crawled
CCBot19.4%15.7%3.7%14.9%
GPTBot18.5%15.2%3.3%13.9%
ClaudeBot17.6%14.0%3.5%13.1%
Google-Extended16.0%12.6%3.4%12.1%
Meta-ExternalAgent16.0%12.3%3.7%12.0%
Amazonbot15.6%12.0%3.6%11.7%
anthropic-ai15.5%11.7%3.8%11.6%
Applebot-Extended14.8%11.6%3.1%11.1%

Sources

Last reviewed 2026-11-04