Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training crawler from Common Crawl

CCBot

In short

CCBot is Common Crawl's training crawler. The token to write in robots.txt is CCBot. Verify it by reverse DNS, then a forward lookup. 19.4 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: CCBot/2.0 (https://commoncrawl.org/faq/). On robots.txt, Common Crawl documents "User-agent: CCBot" then "Disallow: /" to prevent crawling.

Operator
Common Crawl Vendor profile: Common Crawl
Purpose
Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.
User-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)
robots.txt token
CCBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Common Crawl documents "User-agent: CCBot" then "Disallow: /" to prevent crawling. Source, read 2026-11-04.

Verification

Reverse DNS

Reverse-resolve the source address, then forward-resolve the name and require the same address. Documentation, read 2026-11-04.

IP ranges
index.commoncrawl.org/ccbot.json
Rate behaviour
Not published on the CCBot page.
Block rate

19.4% of 5,938 top-10,000 sites with a parsed robots.txt disallow CCBot at the root (1,151 sites). 15.7% name it; 3.7% refuse it only through a * group. Rank 1 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 14.9% (12.2% named). Crawl of 2026-11-04; report.

Documentation
commoncrawl.org, read 2026-11-04.

Note. Dedicated IP ranges with reverse DNS *.crawl.commoncrawl.org (IPv4 only). Common Crawl warns of crawlers falsely identifying as CCBot. ccbot.json creationTime 2026-08-11. Its open dataset is widely used to train models.

Allow and refuse lines

To allow CCBot

User-agent: CCBot
Allow: /

To refuse CCBot

User-agent: CCBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Common Crawl in the Verification Ledger

IP list with synctoken 20260811134000 and reverse DNS. All operators.

IP lists
Reverse DNS
*.crawl.commoncrawl.org (IPv4 only)
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04