Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Search index crawler from OpenAI

OAI-SearchBot

In short

OAI-SearchBot is OpenAI's search index crawler. The token to write in robots.txt is OAI-SearchBot. Check a visitor against the operator's published IP list. About 24 hours from a robots.txt change to OpenAI's systems adjusting, per OpenAI. On robots.txt, sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Operator
OpenAI Vendor profile: OpenAI
Purpose
Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.
User-agent string
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
robots.txt token
OAI-SearchBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges
openai.com/searchbot.json
Rate behaviour
About 24 hours from a robots.txt change to OpenAI's systems adjusting, per OpenAI.
Block rate

7.7% of 5,938 top-10,000 sites with a parsed robots.txt disallow OAI-SearchBot at the root (458 sites). 4.4% name it; 3.3% refuse it only through a * group. Rank 26 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 5.4% (3.0% named). Crawl of 2026-11-04; report.

Documentation
platform.openai.com, read 2026-11-04.

Note. Robots.txt fetches may carry the extra marker "; robots.txt;" in the user agent string so they can be told apart in logs that omit paths.

Allow and refuse lines

To allow OAI-SearchBot

User-agent: OAI-SearchBot
Allow: /

To refuse OAI-SearchBot

User-agent: OAI-SearchBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

OpenAI in the Verification Ledger

One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. All operators.

IP lists
Reverse DNS
publishes nothing
Key directory
https://chatgpt.com/.well-known/http-message-signatures-directory
Cloudflare
GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other OpenAI rows

Sources

Last reviewed 2026-11-04