Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training crawler from OpenAI

GPTBot

In short

GPTBot is OpenAI's training crawler. The token to write in robots.txt is GPTBot. Check a visitor against the operator's published IP list. 18.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.

Operator
OpenAI Vendor profile: OpenAI
Purpose
Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
robots.txt token
GPTBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

OpenAI: disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges
openai.com/gptbot.json
Rate behaviour
OpenAI says changes to robots.txt take about 24 hours to reach its systems, and that if both GPTBot and OAI-SearchBot are allowed it may use one crawl for both uses. gptbot.json creationTime read: 2026-09-22.
Block rate

18.5% of 5,938 top-10,000 sites with a parsed robots.txt disallow GPTBot at the root (1,097 sites). 15.2% name it; 3.3% refuse it only through a * group. Rank 3 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 13.9% (11.5% named). Crawl of 2026-11-04; report.

Documentation
platform.openai.com, read 2026-11-04.

Note. Version 1.4 in the published example string. When fetching robots.txt it may add a "robots.txt" marker to the user agent. Listed as Verified on Cloudflare Radar (radar.cloudflare.com/bots/directory/gptbot, read 2026-11-04).

Allow and refuse lines

To allow GPTBot

User-agent: GPTBot
Allow: /

To refuse GPTBot

User-agent: GPTBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

OpenAI in the Verification Ledger

One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. All operators.

IP lists
Reverse DNS
publishes nothing
Key directory
https://chatgpt.com/.well-known/http-message-signatures-directory
Cloudflare
GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other OpenAI rows

Sources

Last reviewed 2026-11-04