Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Crawler operator

You.com

In short

You.com recommends checking YouBot by signature before reverse DNS or address range, and keeps its Ed25519 key at you.com rather than on a separate host. The bot slows on HTTP 429 and pauses on 503; the 68.67.112.0/24 range is given only in prose, so it cannot be loaded from a file.

Type
Crawler operator
Homepage
you.com
In one line
You.com runs YouBot for its search index and offers three checks: Web Bot Auth signatures with keys at you.com, reverse DNS under search.you.com, and the range 68.67.112.0/24.
Last read
2026-11-05

What You.com does for this layer

From the vendor's own pages

You.com says YouBot is the crawler that powers the You.com search engine by discovering and indexing web pages. (you.com) YouBot's user agent carries YouBot/1.0, a link to docs.you.com/youbot and an env:prod environment marker inside a Chrome-style string. (you.com) You.com says YouBot signs requests with Web Bot Auth and publishes Ed25519 public keys in JWK format at you.com/.well-known/http-message-signatures-directory. (you.com) You.com names cryptographic signatures as the recommended check, ahead of reverse DNS and IP range validation. (you.com)

You.com says genuine YouBot requests originate from 68.67.112.0/24. (you.com) You.com says YouBot honours robots.txt including crawl-delay and caches robots.txt for 30 minutes. (you.com) You.com says YouBot reduces its crawl rate on HTTP 429 and pauses crawling on HTTP 503. (you.com)

What our crawl saw

you.com served a key directory with one Ed25519 key and the content type application/http-message-signatures-directory+json on 2026-11-05. (you.com)

Products and plans

ProductWhat it doesPrice modelFromAs of
YouBot Crawler for the You.com search index, signed with Web Bot Auth. No price recorded

Standards

StandardStatusEvidence
RFC 9309 (robots.txt) Supports YouBot fully respects robots.txt, including user-agent rules and crawl-delay. (you.com)
Web Bot Auth Supports Requests signed; keys at you.com/.well-known/http-message-signatures-directory. (you.com)
Published IP list Partial One /24 given in the docs, not a machine-readable range file. (you.com)
Reverse DNS check Supports youbot-{octets-with-hyphens}.search.you.com. (you.com)

Our checks

CheckResultMethodDate
Homepage fetch HTTP 200 GET with the crawl user agent after a robots.txt check 2026-11-05
Web Bot Auth key directory 200, application/http-message-signatures-directory+json, 1 key GET /.well-known/http-message-signatures-directory with the crawl user agent 2026-11-05

What You.com does not do

  • You.com publishes its YouBot range in prose on the docs page, not as a machine-readable IP file.
  • You.com's crawler page covers YouBot only and documents no separate fetcher for user-triggered requests (read 2026-11-05).

Reading notes

The YouBot row in the Crawler Index, read on 4 November 2026, said You.com published no documentation. The page cited here, you.com/docs/youbot, exists and documents the bot.

Sources

Last reviewed 2026-11-05