Crawler operator
OpenAI
In short
Each of OpenAI's four robots has a robots.txt setting of its own, and a change reaches ChatGPT search after roughly a day. Only its agent browser is checkable by signature, with a key at chatgpt.com; the other robots rely on IP files, and opting out of OAI-SearchBot keeps a site out of ChatGPT answers.
- Type
- Crawler operator
- Homepage
- openai.com
- In one line
- OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com.
- Last read
- 2026-11-05
What OpenAI does for this layer
From the vendor's own pages
OpenAI's crawler page lists four user agents, OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, and says each robots.txt setting works independently of the others. (developers.openai.com) OpenAI says that when a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both uses to avoid duplicate crawling. (developers.openai.com) OpenAI says a change to a site's robots.txt can take about 24 hours to show in its search results. (developers.openai.com) OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. (developers.openai.com)
OpenAI states that OAI-AdsBot only visits landing pages submitted as ads on ChatGPT and that the data it collects is not used to train generative AI foundation models. (developers.openai.com) OpenAI says GPTBot and OAI-SearchBot may add a robots.txt marker to the user agent when fetching robots.txt files, so logs without paths can still separate those requests. (developers.openai.com)
What our crawl saw
OpenAI's chatgpt-user.json listed 235 IPv4 prefixes with a creationTime of 2026-10-27 when fetched on 2026-11-05, the largest of OpenAI's four range files. (openai.com, dated 2026-10-27) A GET to chatgpt.com/.well-known/http-message-signatures-directory on 2026-11-05 returned HTTP 200, content type application/http-message-signatures-directory+json, with one Ed25519 key. (chatgpt.com) The same well-known key path on openai.com returned HTTP 404 on 2026-11-05, so OpenAI's signing identity lives on chatgpt.com rather than its corporate domain. (openai.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| GPTBot | Crawler whose content may be used to train OpenAI's generative foundation models; disallowing it is the training opt-out. | |||
| OAI-SearchBot | Crawler for the index behind ChatGPT search answers. | |||
| OAI-AdsBot | Fetcher that checks landing pages submitted as ChatGPT ads. | |||
| ChatGPT-User | User-triggered fetcher for ChatGPT, Custom GPTs and GPT Actions. | |||
| ChatGPT agent cloud browser | Hosted browser for ChatGPT's agent mode whose requests are signed with keys published at chatgpt.com. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Partial | GPTBot and OAI-SearchBot follow robots.txt tokens; OpenAI says robots.txt rules may not apply to ChatGPT-User. (developers.openai.com) |
| Web Bot Auth | Supports | chatgpt.com serves a key directory with the Web Bot Auth media type (observed 2026-11-05). (chatgpt.com) |
| Content Signals | No statement | Not mentioned on OpenAI's crawler page (read 2026-11-05). |
| Published IP list | Supports | Four JSON range files: gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json. (developers.openai.com) |
| Reverse DNS check | No statement | OpenAI's crawler page gives no reverse DNS pattern (read 2026-11-05). |
| Cloudflare verified bots | No statement | OpenAI's crawler page does not mention Cloudflare's verified bot list (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | chatgpt.com: 200, application/http-message-signatures-directory+json, 1 key; openai.com: 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| gptbot.json | 18 IPv4 prefixes, creationTime 2026-09-22 | GET and count prefixes | 2026-11-05 |
| searchbot.json | 39 IPv4 prefixes, creationTime 2026-01-02 | GET and count prefixes | 2026-11-05 |
| chatgpt-user.json | 235 IPv4 prefixes, creationTime 2026-10-27 | GET and count prefixes | 2026-11-05 |
| adsbot.json | 2 IPv4 prefixes, creationTime 2026-05-12 | GET and count prefixes | 2026-11-05 |
What OpenAI does not do
- OpenAI publishes no reverse DNS pattern for its robots, so verification rests on the IP files or, for the agent browser, on signatures.
- Blocking ChatGPT-User in robots.txt is not a guaranteed stop, because OpenAI says robots.txt rules may not apply to user-started fetches.
- OpenAI's crawler page names no token for ChatGPT Atlas browsing (read 2026-11-05).
Reading notes
OpenAI's agent allowlisting help article returned HTTP 403 to our crawl user agent on 5 November 2026, so what this profile says about agent signing rests on the key directory we observed at chatgpt.com.
Related pages on this site
Sources
Last reviewed 2026-11-05