Crawler operator
DuckDuckGo
In short
DuckDuckGo's DuckAssistBot retrieves pages while an answer is composed, cites them, and is not used for model training; a Disallow takes 72 hours to apply. Its search crawler, DuckDuckBot, has an IP file of its own, but on 5 November 2026 the two files matched byte for byte, so addresses alone cannot separate the bots.
- Type
- Crawler operator
- Homepage
- duckduckgo.com
- In one line
- DuckDuckGo runs DuckDuckBot for its search index and DuckAssistBot for real-time fetches behind its AI-assisted answers, publishes an IP file for each, and says DuckAssistBot data is not used to train AI models.
- Last read
- 2026-11-05
What DuckDuckGo does for this layer
From the vendor's own pages
DuckDuckGo says a robots.txt Disallow for DuckAssistBot takes effect after 72 hours. (duckduckgo.com) DuckDuckGo says opting out of DuckAssistBot does not affect organic rankings or whether a site appears in DuckDuckGo search results. (duckduckgo.com) DuckAssistBot identifies itself as DuckAssistBot/1.2 with a link to duckduckgo.com/duckassistbot.html. (duckduckgo.com) DuckDuckGo says DuckDuckBot respects the rules at robotstxt.org and identifies itself as DuckDuckBot/1.1. (duckduckgo.com)
What our crawl saw
DuckDuckGo's duckassistbot.json and duckduckbot.json were byte-identical on 2026-11-05 (same SHA-1), each with 486 IPv4 prefixes and a creationTime of 2026-09-01. (duckduckgo.com, dated 2026-09-01) duckduckgo.com answered HTTP 404 for the Web Bot Auth key directory path on 2026-11-05. (duckduckgo.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| DuckDuckBot | Crawler for DuckDuckGo search results. | |||
| DuckAssistBot | Real-time crawler for AI-assisted answers; not used for model training. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Both help pages describe robots.txt control; DuckAssistBot opt-out takes 72 hours. (duckduckgo.com) |
| Web Bot Auth | No statement | Not on either help page; key directory path 404 (observed 2026-11-05). |
| Published IP list | Supports | duckassistbot.json and duckduckbot.json. (duckduckgo.com) |
| Reverse DNS check | No statement | Neither help page gives a reverse DNS pattern (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| duckassistbot.json and duckduckbot.json | identical files, 486 IPv4 prefixes each, creationTime 2026-09-01 | GET both and compare SHA-1 | 2026-11-05 |
What DuckDuckGo does not do
- DuckDuckGo's two range files were identical on 2026-11-05, so an IP check alone cannot tell DuckAssistBot from DuckDuckBot.
- DuckDuckGo publishes no reverse DNS pattern or signature for either crawler (read 2026-11-05).
Related pages on this site
Sources
Last reviewed 2026-11-05