Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

IP range JSON

In short

A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. An IETF draft (jafar) proposes a common format. 17 Index rows publish an IP list as JSON, the most common method, and 35 publish no method at all. Example: OpenAI's gptbot.json carried creationTime 2026-09-22 when read on 2026-11-04.

Definition

A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. An IETF draft (jafar) proposes a common format.

Sourced example

OpenAI's gptbot.json carried creationTime 2026-09-22 when read on 2026-11-04. Source, read 2026-11-04.

How it shows up here

17 Index rows publish an IP list as JSON, the most common method, and 35 publish no method at all.

Previous: HTTP Message Signatures. Next: Key directory. All 41 terms.

Sources

Last reviewed 2026-11-04