Glossary
Training crawler
In short
A crawler whose fetched content may be used to train AI models. Most large operators now separate it from their search crawler so sites can refuse one and keep the other. 9 Index rows are filed under Training. The most refused measured token is CCBot at 19.4 percent of top-10,000 sites. Related: All training rows and Allow search, refuse training.
Definition
A crawler whose fetched content may be used to train AI models. Most large operators now separate it from their search crawler so sites can refuse one and keep the other.
Sourced example
GPTBot crawls content that may be used in training OpenAI foundation models. Source, read 2026-11-04.
How it shows up here
9 Index rows are filed under Training. The most refused measured token is CCBot at 19.4 percent of top-10,000 sites.
Previous: Spoofed crawler. Next: User-agent string. All 41 terms.
Sources
Last reviewed 2026-11-04