---
title: "Diffbot: user agent, token, block rate | Agent Ready Crawlers"
description: "Diffbot from Diffbot: robots.txt token Diffbot, other purpose and what a site can check; no exact user-agent string is published. 15.7% of top sites block it."
canonical: https://agentreadycrawlers.com/crawlers/diffbot
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Crawlers"
---

# Diffbot: user agent, token, block rate | Agent Ready Crawlers

> Diffbot from Diffbot: robots.txt token Diffbot, other purpose and what a site can check; no exact user-agent string is published. 15.7% of top sites block it.

Source: https://agentreadycrawlers.com/crawlers/diffbot
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Bot from Diffbot

# Diffbot

In short

Diffbot is Diffbot's bot, filed under Other. The token to write in robots.txt is Diffbot. The operator publishes nothing a site can check. Knowledge graph and extraction service; robots.txt compliance is a customer setting. Diffbot: its crawler checks robots.txt, but customers can opt to "ignore robots.txt" in crawl settings or via the Crawl API.

Operator

Diffbot Vendor profile: [Diffbot](https://agentreadycrawlers.com/vendors/diffbot/)

Purpose

Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.

User-agent string

No exact string is published. What the operator or a third party says: Contains "Diffbot"; full string not on the page read

robots.txt token

`Diffbot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Depends on a setting

Diffbot: its crawler checks robots.txt, but customers can opt to "ignore robots.txt" in crawl settings or via the Crawl API. [Source](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems), read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems), read 2026-11-04.

IP ranges

Not published.

Rate behaviour

Set per customer crawl job.

Block rate

**15.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Diffbot` at the root (932 sites). 11.9% name it; 3.8% refuse it only through a \* group. Rank 7 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 11.8% (9.1% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/).

Documentation

[docs.diffbot.com](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems), read 2026-11-04.

Note. Knowledge graph and extraction service; robots.txt compliance is a customer setting.

## Allow and refuse lines

### To allow Diffbot

User-agent: Diffbot
Allow: /

### To refuse Diffbot

User-agent: Diffbot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

## Diffbot in the Verification Ledger

Publishes nothing for verification; robots.txt can be ignored by customer setting. [All operators](https://agentreadycrawlers.com/verification/).

IP lists

publishes nothing

Reverse DNS

publishes nothing

Key directory

publishes nothing

Cloudflare

not checked

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

## Sources

-   [docs.diffbot.com: why is my crawl not crawling and other uncommon crawl problems](https://docs.diffbot.com/docs/why-is-my-crawl-not-crawling-and-other-uncommon-crawl-problems) read 2026-11-04

Last reviewed 2026-11-04