The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

User-agent string

In short

The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures. Each profile prints the string the operator documents. For the 23 rows with no robots.txt token the field says so. Example: OAI-SearchBot sends a Chrome-like string ending "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot". Related: Tokens versus user-agent strings and GPTBot profile.

Definition

The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures.

Sourced example

OAI-SearchBot sends a Chrome-like string ending "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot". Source, read 2026-11-04.

How it shows up here

Each profile prints the string the operator documents. For the 23 rows with no robots.txt token the field says so.

Previous: Training crawler. Next: User-triggered fetcher. All 41 terms.

Sources

Overview of OpenAI Crawlers read 2026-11-04

Last reviewed 2026-11-04