Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

How to verify Web Bot Auth signatures

In short

Read Signature-Agent, fetch the key directory at /.well-known/http-message-signatures-directory, verify the RFC 9421 signature, and fall back to IP checks for unsigned traffic. Web Bot Auth builds on HTTP Message Signatures (RFC 9421); the signer publishes Ed25519 public keys in a directory named by the Signature-Agent header. The IETF webbotauth working group adopted draft-ietf-webbotauth-httpsig-protocol; revision -01 is dated 2026-10-29.

Web Bot Auth lets an operator prove a request is its own with a signature. Verification is five steps and the first is reading one header.

In practice

  1. Read the Signature-Agent header, along with Signature and Signature-Input. Compare Signature-Agent exactly with the value the operator documents. OpenAI says to confirm it matches "https://chatgpt.com", quotation marks included.
  2. Fetch the key directory from the origin that header names, at /.well-known/http-message-signatures-directory. It holds the signer's Ed25519 public keys.
  3. Verify the RFC 9421 HTTP Message Signature against those keys. Cloudflare does no nonce validation today and relies on short expires values, so reject stale signatures.
  4. Keep IP and reverse DNS checks for everything unsigned. Google signs only a subset of Google-Agent requests, as https://agent.bot.goog, and says to keep them.
  5. Decide policy separately. A valid signature proves who signed, not whether the request is authorised; that stays your origin's decision.

What the sources say

Sources

Last reviewed 2026-11-04