Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

HTTP Message Signatures

In short

RFC 9421: a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. Web Bot Auth is a profile of it. A verifier library for RFC 9421 is the starting point for the checks in the guide. Example: OpenAI tells site owners to verify ChatGPT Cloud browser requests according to RFC 9421.

Definition

RFC 9421: a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. Web Bot Auth is a profile of it.

Sourced example

OpenAI tells site owners to verify ChatGPT Cloud browser requests according to RFC 9421. Source, read 2026-11-04.

How it shows up here

A verifier library for RFC 9421 is the starting point for the checks in the guide.

Previous: HTTP 402 Payment Required. Next: IP range JSON. All 41 terms.

Sources

Last reviewed 2026-11-04