---
title: "CCBot: user agent, token, block rate | Agent Ready Crawlers"
description: "CCBot from Common Crawl: exact user-agent string, robots.txt token CCBot, training purpose and how to verify it. 19.4% of top sites block it. Read 2026-11-04."
canonical: https://agentreadycrawlers.com/crawlers/ccbot
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Crawlers"
---

# CCBot: user agent, token, block rate | Agent Ready Crawlers

> CCBot from Common Crawl: exact user-agent string, robots.txt token CCBot, training purpose and how to verify it. 19.4% of top sites block it. Read 2026-11-04.

Source: https://agentreadycrawlers.com/crawlers/ccbot
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Training crawler from Common Crawl

# CCBot

In short

CCBot is Common Crawl's training crawler. The token to write in robots.txt is CCBot. Verify it by reverse DNS, then a forward lookup. 19.4 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: CCBot/2.0 (https://commoncrawl.org/faq/). On robots.txt, Common Crawl documents "User-agent: CCBot" then "Disallow: /" to prevent crawling.

Operator

Common Crawl Vendor profile: [Common Crawl](https://agentreadycrawlers.com/vendors/common-crawl/)

Purpose

Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out.

User-agent string

`CCBot/2.0 (https://commoncrawl.org/faq/)`

robots.txt token

`CCBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Yes, per operator

Common Crawl documents "User-agent: CCBot" then "Disallow: /" to prevent crawling. [Source](https://commoncrawl.org/ccbot), read 2026-11-04.

Verification

Reverse DNS

Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://commoncrawl.org/ccbot), read 2026-11-04.

IP ranges

[index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json)

Rate behaviour

Not published on the CCBot page.

Block rate

**19.4%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `CCBot` at the root (1,151 sites). 15.7% name it; 3.7% refuse it only through a \* group. Rank 1 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 14.9% (12.2% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/).

Documentation

[commoncrawl.org](https://commoncrawl.org/ccbot), read 2026-11-04.

Note. Dedicated IP ranges with reverse DNS \*.crawl.commoncrawl.org (IPv4 only). Common Crawl warns of crawlers falsely identifying as CCBot. ccbot.json creationTime 2026-08-11. Its open dataset is widely used to train models.

## Allow and refuse lines

### To allow CCBot

User-agent: CCBot
Allow: /

### To refuse CCBot

User-agent: CCBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

## Common Crawl in the Verification Ledger

IP list with synctoken 20260811134000 and reverse DNS. [All operators](https://agentreadycrawlers.com/verification/).

IP lists

-   [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json)

Reverse DNS

\*.crawl.commoncrawl.org (IPv4 only)

Key directory

publishes nothing

Cloudflare

not checked

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

## Sources

-   [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04
-   [index.commoncrawl.org: ccbot (JSON)](https://index.commoncrawl.org/ccbot.json) read 2026-11-04

Last reviewed 2026-11-04