Crawler operator
Common Crawl
In short
Common Crawl's CCBot identifies as version 2.0, follows up to four redirects, runs no JavaScript and sets no cookies, and can be verified by reverse DNS on its dedicated ranges, except over IPv6. Since 17 September 2025 the non-profit has published each legal opt-out request it receives, in what is now called the Opt-Out Ledger.
- Type
- Crawler operator
- Homepage
- commoncrawl.org
- In one line
- Common Crawl is a non-profit foundation whose CCBot builds an open web crawl archive; CCBot honours robots.txt and Crawl-delay, runs on dedicated ranges with reverse DNS, and the foundation publishes a ledger of legal opt-out requests.
- Last read
- 2026-11-05
What Common Crawl does for this layer
From the vendor's own pages
CCBot identifies itself as CCBot/2.0 with a link to commoncrawl.org/faq/; Common Crawl's older bot sent CCBot/1.0. (commoncrawl.org) Common Crawl says CCBot now runs on dedicated IP ranges with reverse DNS under crawl.commoncrawl.org, except over IPv6, where reverse DNS is not yet supported. (commoncrawl.org) Common Crawl publishes CCBot's IPv4 and IPv6 ranges as JSON at index.commoncrawl.org/ccbot.json. (commoncrawl.org) Common Crawl warns that it knows of crawlers that falsely identify themselves as CCBot and recommends verifying the user agent. (commoncrawl.org)
Common Crawl says CCBot obeys Crawl-delay in robots.txt and uses an adaptive back-off that slows requests when a server answers HTTP 429 or 5xx. (commoncrawl.org) CCBot follows up to four consecutive redirects, or five when fetching robots.txt in line with RFC 9309, and does not execute JavaScript or use cookies. (commoncrawl.org) Common Crawl's FAQ says the dataset is a sample of the web rather than whole sites, and that CCBot uses sitemaps announced in robots.txt. (commoncrawl.org) Common Crawl began publishing every legal opt-out request it had received on 2025-09-17 as the Opt-Out Registry, now renamed the Opt-Out Ledger. (commoncrawl.org, dated 2025-09-17)
Claims the vendor makes about itself
Common Crawl describes itself as a non-profit foundation that maintains an open repository of web crawl data that anyone can access and analyse. (commoncrawl.org)
What our crawl saw
commoncrawl.org returned HTTP 404 at the Web Bot Auth key directory path on 2026-11-05. (commoncrawl.org)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| CCBot | Nutch-based crawler that builds Common Crawl's open web archive. | |||
| Opt-Out Ledger | Public list of the legal opt-out requests Common Crawl has received. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | CCBot checks robots.txt before fetching, obeys Crawl-delay and cites RFC 9309 for redirect handling. (commoncrawl.org) |
| Web Bot Auth | No statement | Not on the CCBot page; key directory path 404 on commoncrawl.org (observed 2026-11-05). |
| Published IP list | Supports | Ranges published at index.commoncrawl.org/ccbot.json. (commoncrawl.org) |
| Reverse DNS check | Supports | Dedicated ranges resolve under crawl.commoncrawl.org with forward confirmation (IPv4 only). (commoncrawl.org) |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
What Common Crawl does not do
- CCBot has no reverse DNS over IPv6, so IPv6 requests can only be checked against the range file.
- A robots.txt Disallow stops future CCBot crawls; Common Crawl's FAQ does not say it removes pages from earlier crawl archives.
Reading notes
The robots.txt at index.commoncrawl.org disallows our crawl token, so we did not fetch the CCBot range file ourselves and counted no prefixes in it.
Related pages on this site
Sources
Last reviewed 2026-11-05