Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Signature-Agent header

In short

The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys. Log it on every request: it is the only field that tells you which key directory to fetch. Example: ChatGPT's Cloud browser sends Signature-Agent: "https://chatgpt.com". Related: Verify Web Bot Auth signatures and ChatGPT agent profile.

Definition

The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys.

Sourced example

ChatGPT's Cloud browser sends Signature-Agent: "https://chatgpt.com". Source, read 2026-11-04.

How it shows up here

Log it on every request: it is the only field that tells you which key directory to fetch.

Previous: Search-index crawler. Next: Signed agent. All 41 terms.

Sources

Last reviewed 2026-11-04