Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Bot from Diffbot

Diffbot

In short

Diffbot is Diffbot's bot, filed under Other. The token to write in robots.txt is Diffbot. The operator publishes nothing a site can check. Knowledge graph and extraction service; robots.txt compliance is a customer setting. Diffbot: its crawler checks robots.txt, but customers can opt to "ignore robots.txt" in crawl settings or via the Crawl API.

Operator
Diffbot Vendor profile: Diffbot
Purpose
Other All other rows. Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.
User-agent string
No exact string is published. What the operator or a third party says: Contains "Diffbot"; full string not on the page read
robots.txt token
Diffbot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Depends on a setting

Diffbot: its crawler checks robots.txt, but customers can opt to "ignore robots.txt" in crawl settings or via the Crawl API. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Set per customer crawl job.
Block rate

15.7% of 5,938 top-10,000 sites with a parsed robots.txt disallow Diffbot at the root (932 sites). 11.9% name it; 3.8% refuse it only through a * group. Rank 7 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 11.8% (9.1% named). Crawl of 2026-11-04; report.

Documentation
docs.diffbot.com, read 2026-11-04.

Note. Knowledge graph and extraction service; robots.txt compliance is a customer setting.

Allow and refuse lines

To allow Diffbot

User-agent: Diffbot
Allow: /

To refuse Diffbot

User-agent: Diffbot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Diffbot in the Verification Ledger

Publishes nothing for verification; robots.txt can be ignored by customer setting. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04