Crawler operator
Diffbot
In short
Diffbot's own whole-web crawl feeds its Knowledge Graph and, its FAQ says, is not used to train generative AI; a second token, Diffbot-User, covers pages a person asks to see. Customers can run hosted Crawlbot jobs from USD 899 a month, and those follow robots.txt only until the customer switches that setting off.
- Type
- Crawler operator
- Homepage
- diffbot.com
- In one line
- Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets.
- Last read
- 2026-11-05
What Diffbot does for this layer
From the vendor's own pages
Diffbot's robots.txt FAQ names two user agents: Diffbot, for proactive crawling that feeds its Knowledge Graph and web search, and Diffbot-User, for requests made on behalf of a person browsing a URL with Diffbot software. (diffbot.com) Diffbot says its web crawls follow robots.txt by default, including the disallow and crawl-delay directives. (diffbot.com) Diffbot's FAQ says the Diffbot crawler "is not used for AI training" and that "Diffbot does not crawl pages for training generative AI foundation models". (diffbot.com) Diffbot states that a robots.txt instruction "can be ignored/overridden" in specific cases, typically when the customer has a partnership or agreement with the site being crawled. (diffbot.com)
Diffbot's crawl troubleshooting guide offers an "ignore robots.txt" setting in crawl settings or the Crawl API, and tells customers whose seed URL refuses connections to try turning on proxies. (diffbot.com) For Extract and Crawlbot jobs, Diffbot recommends that customers set a User-Agent naming their own organisation and notes that robots.txt adherence in Crawlbot is on by default. (diffbot.com) Diffbot's custom user agent guide shows a customer adding an agreed word to the User-Agent header for one domain, so that the site can allow that customer's requests and refuse other Diffbot customers. (diffbot.com) Diffbot's pricing page lists Free at USD 0 with 10,000 credits a month and 5 requests a minute, Startup at USD 299 a month and Plus at USD 899 a month, the first plan with access to Crawl (25 active crawls). (diffbot.com)
Diffbot charges 0 credits to spider a site for links and 2 credits instead of 1 to extract a page through a datacenter proxy, per its pricing table. (diffbot.com)
Claims the vendor makes about itself
Diffbot says it runs its own crawl of the public web, independent from Google and Bing, on custom-assembled hardware in its own data center in California. (diffbot.com) Diffbot says it has maintained a whole-web crawl "for over 20 years" and that its Global Index revisits pages on a freshness policy set per page type. (diffbot.com)
What our crawl saw
A GET for www.diffbot.com/.well-known/http-message-signatures-directory with the crawl user agent returned HTTP 404 on 2026-11-05, so Diffbot serves no Web Bot Auth key directory at that host. (diffbot.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Crawl (Crawlbot) | Hosted crawl jobs that spider a site from seed URLs and pass pages to an Extract API; robots.txt adherence on by default, customer can turn it off. | subscription with credits | USD 899 | 2026-11-05 |
| Extract APIs | Classify a page and return structured JSON; custom user agents can be set per domain. | credits | USD 0 | 2026-11-05 |
| Knowledge Graph and Web Search | Entity graph and web search built from Diffbot's own crawl of the public web. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Diffbot's crawls follow robots.txt by default, including crawl-delay; customers can override it for their own jobs. (diffbot.com) |
| Web Bot Auth | No statement | No key directory at www.diffbot.com (HTTP 404, 2026-11-05). |
| Published IP list | No statement | The robots.txt FAQ names user agents only; no IP list was found (checked 2026-11-05). |
| Reverse DNS check | No statement | No reverse DNS pattern published (checked 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 404, text/html | GET https://www.diffbot.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
What Diffbot does not do
- Diffbot publishes no IP range list, reverse DNS pattern or signature keys for its crawlers, so a request claiming to be Diffbot can be checked only against the user agent name (checked 2026-11-05).
- Diffbot does not hold its customers to robots.txt: a customer crawl job can turn adherence off.
Reading notes
The Diffbot row in the Crawler Index, read on 4 November 2026, says the full user agent string was not published. Diffbot's robots.txt FAQ, read the next day, adds the Diffbot-User token and the statement that the crawler is not used for AI training.
Related pages on this site
Sources
Last reviewed 2026-11-05