--- title: "Diffbot: crawler operator profile | Agent Ready Crawlers" description: "Diffbot: crawler operator profile covering Crawl (Crawlbot), Extract APIs and Knowledge Graph and Web Search. Sources, standards and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/diffbot site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Diffbot: crawler operator profile | Agent Ready Crawlers > Diffbot: crawler operator profile covering Crawl (Crawlbot), Extract APIs and Knowledge Graph and Web Search. Sources, standards and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/diffbot Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Diffbot In short Diffbot's own whole-web crawl feeds its Knowledge Graph and, its FAQ says, is not used to train generative AI; a second token, Diffbot-User, covers pages a person asks to see. Customers can run hosted Crawlbot jobs from USD 899 a month, and those follow robots.txt only until the customer switches that setting off. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [diffbot.com](https://www.diffbot.com) In one line Web extraction and Knowledge Graph company that runs its own whole-web crawler under the Diffbot token, a user-triggered Diffbot-User fetcher, and hosted crawl jobs whose robots.txt handling each customer sets. Last read 2026-11-05 ## What Diffbot does for this layer ### From the vendor's own pages Diffbot's robots.txt FAQ names two user agents: Diffbot, for proactive crawling that feeds its Knowledge Graph and web search, and Diffbot-User, for requests made on behalf of a person browsing a URL with Diffbot software. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot says its web crawls follow robots.txt by default, including the disallow and crawl-delay directives. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot's FAQ says the Diffbot crawler "is not used for AI training" and that "Diffbot does not crawl pages for training generative AI foundation models". ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot states that a robots.txt instruction "can be ignored/overridden" in specific cases, typically when the customer has a partnership or agreement with the site being crawled. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot's crawl troubleshooting guide offers an "ignore robots.txt" setting in crawl settings or the Crawl API, and tells customers whose seed URL refuses connections to try turning on proxies. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/crawl-not-crawling)) For Extract and Crawlbot jobs, Diffbot recommends that customers set a User-Agent naming their own organisation and notes that robots.txt adherence in Crawlbot is on by default. ([diffbot.com](https://www.diffbot.com/docs/crawl/faq/robots-txt)) Diffbot's custom user agent guide shows a customer adding an agreed word to the User-Agent header for one domain, so that the site can allow that customer's requests and refuse other Diffbot customers. ([diffbot.com](https://www.diffbot.com/docs/custom-api/faq/custom-user-agents)) Diffbot's pricing page lists Free at USD 0 with 10,000 credits a month and 5 requests a minute, Startup at USD 299 a month and Plus at USD 899 a month, the first plan with access to Crawl (25 active crawls). ([diffbot.com](https://www.diffbot.com/pricing)) Diffbot charges 0 credits to spider a site for links and 2 credits instead of 1 to extract a page through a datacenter proxy, per its pricing table. ([diffbot.com](https://www.diffbot.com/pricing)) ### Claims the vendor makes about itself Diffbot says it runs its own crawl of the public web, independent from Google and Bing, on custom-assembled hardware in its own data center in California. ([diffbot.com](https://www.diffbot.com/company/about)) Diffbot says it has maintained a whole-web crawl "for over 20 years" and that its Global Index revisits pages on a freshness policy set per page type. ([diffbot.com](https://www.diffbot.com/technology)) ### What our crawl saw A GET for www.diffbot.com/.well-known/http-message-signatures-directory with the crawl user agent returned HTTP 404 on 2026-11-05, so Diffbot serves no Web Bot Auth key directory at that host. ([diffbot.com](https://www.diffbot.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Crawl (Crawlbot) | Hosted crawl jobs that spider a site from seed URLs and pass pages to an Extract API; robots.txt adherence on by default, customer can turn it off. | subscription with credits Crawl starts on the Plus plan, USD 899 a month, 25 active crawls. | USD 899 | 2026-11-05 | | Extract APIs | Classify a page and return structured JSON; custom user agents can be set per domain. | credits Free plan 10,000 credits a month; 1 credit per page, 2 through a datacenter proxy. | USD 0 | 2026-11-05 | | Knowledge Graph and Web Search | Entity graph and web search built from Diffbot's own crawl of the public web. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Supports | Diffbot's crawls follow robots.txt by default, including crawl-delay; customers can override it for their own jobs. (diffbot.com) | | Web Bot Auth | No statement | No key directory at www.diffbot.com (HTTP 404, 2026-11-05). | | Published IP list | No statement | The robots.txt FAQ names user agents only; no IP list was found (checked 2026-11-05). | | Reverse DNS check | No statement | No reverse DNS pattern published (checked 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404, text/html | GET https://www.diffbot.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Diffbot does not do - Diffbot publishes no IP range list, reverse DNS pattern or signature keys for its crawlers, so a request claiming to be Diffbot can be checked only against the user agent name (checked 2026-11-05). - Diffbot does not hold its customers to robots.txt: a customer crawl job can turn adherence off. ## Reading notes The [Diffbot row](https://agentreadycrawlers.com/crawlers/diffbot/) in the Crawler Index, read on 4 November 2026, says the full user agent string was not published. Diffbot's robots.txt FAQ, read the next day, adds the Diffbot-User token and the statement that the crawler is not used for AI training. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Diffbot](https://agentreadycrawlers.com/crawlers/diffbot/) token Diffbot ## Sources - [diffbot.com](https://www.diffbot.com) read 2026-11-05 - [Does Diffbot respect robots.txt? | Diffbot](https://www.diffbot.com/docs/crawl/faq/robots-txt) read 2026-11-05 - [Why is my crawl not crawling (and other uncommon crawl problems)? | Diffbot](https://www.diffbot.com/docs/crawl/faq/crawl-not-crawling) read 2026-11-05 - [How to Use Custom User Agents with Extract APIs | Diffbot](https://www.diffbot.com/docs/custom-api/faq/custom-user-agents) read 2026-11-05 - [About Diffbot | Diffbot](https://www.diffbot.com/company/about) read 2026-11-05 - [Technology | Diffbot](https://www.diffbot.com/technology) read 2026-11-05 - [Plans & Pricing | Diffbot](https://www.diffbot.com/pricing) read 2026-11-05 - [Diffbot key directory probe](https://www.diffbot.com/.well-known/http-message-signatures-directory) read 2026-11-05 Last reviewed 2026-11-05