---
title: "Training crawler: definition | Agent Ready Crawlers"
description: "Training crawler: A crawler whose fetched content may be used to train AI models. 9 Index rows are filed under Training."
canonical: https://agentreadycrawlers.com/glossary/training-crawler
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Glossary"
---

# Training crawler: definition | Agent Ready Crawlers

> Training crawler: A crawler whose fetched content may be used to train AI models. 9 Index rows are filed under Training.

Source: https://agentreadycrawlers.com/glossary/training-crawler
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Glossary

# Training crawler

In short

A crawler whose fetched content may be used to train AI models. Most large operators now separate it from their search crawler so sites can refuse one and keep the other. 9 Index rows are filed under Training. The most refused measured token is CCBot at 19.4 percent of top-10,000 sites. Related: All training rows and Allow search, refuse training.

## Definition

A crawler whose fetched content may be used to train AI models. Most large operators now separate it from their search crawler so sites can refuse one and keep the other.

## Sourced example

GPTBot crawls content that may be used in training OpenAI foundation models. [Source](https://platform.openai.com/docs/bots), read 2026-11-04.

## How it shows up here

9 Index rows are filed under Training. The most refused measured token is CCBot at 19.4 percent of top-10,000 sites.

-   [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/)
-   [Allow search, refuse training](https://agentreadycrawlers.com/guides/allow-search-refuse-training/)

Previous: [Spoofed crawler](https://agentreadycrawlers.com/glossary/spoofed-crawler/). Next: [User-agent string](https://agentreadycrawlers.com/glossary/user-agent-string/). [All 41 terms](https://agentreadycrawlers.com/glossary/).

## Sources

-   [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04

Last reviewed 2026-11-04