--- title: "OpenAI: crawler operator profile | Agent Ready Crawlers" description: "OpenAI: crawler operator profile covering GPTBot, OAI-SearchBot, OAI-AdsBot and 2 more. Sourced facts, standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/openai site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # OpenAI: crawler operator profile | Agent Ready Crawlers > OpenAI: crawler operator profile covering GPTBot, OAI-SearchBot, OAI-AdsBot and 2 more. Sourced facts, standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/openai Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # OpenAI In short Each of OpenAI's four robots has a robots.txt setting of its own, and a change reaches ChatGPT search after roughly a day. Only its agent browser is checkable by signature, with a key at chatgpt.com; the other robots rely on IP files, and opting out of OAI-SearchBot keeps a site out of ChatGPT answers. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Homepage [openai.com](https://openai.com) In one line OpenAI documents four robots (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User), each with its own robots.txt token and IP file, and signs its ChatGPT agent browser traffic with Web Bot Auth keys served from chatgpt.com. Last read 2026-11-05 ## What OpenAI does for this layer ### From the vendor's own pages OpenAI's crawler page lists four user agents, OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, and says each robots.txt setting works independently of the others. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says that when a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both uses to avoid duplicate crawling. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says a change to a site's robots.txt can take about 24 hours to show in its search results. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI states that OAI-AdsBot only visits landing pages submitted as ads on ChatGPT and that the data it collects is not used to train generative AI foundation models. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) OpenAI says GPTBot and OAI-SearchBot may add a robots.txt marker to the user agent when fetching robots.txt files, so logs without paths can still separate those requests. ([developers.openai.com](https://developers.openai.com/api/docs/bots)) ### What our crawl saw OpenAI's chatgpt-user.json listed 235 IPv4 prefixes with a creationTime of 2026-10-27 when fetched on 2026-11-05, the largest of OpenAI's four range files. ([openai.com](https://openai.com/chatgpt-user.json), dated 2026-10-27) A GET to chatgpt.com/.well-known/http-message-signatures-directory on 2026-11-05 returned HTTP 200, content type application/http-message-signatures-directory+json, with one Ed25519 key. ([chatgpt.com](https://chatgpt.com/.well-known/http-message-signatures-directory)) The same well-known key path on openai.com returned HTTP 404 on 2026-11-05, so OpenAI's signing identity lives on chatgpt.com rather than its corporate domain. ([openai.com](https://openai.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | GPTBot | Crawler whose content may be used to train OpenAI's generative foundation models; disallowing it is the training opt-out. | No price recorded | | OAI-SearchBot | Crawler for the index behind ChatGPT search answers. | No price recorded | | OAI-AdsBot | Fetcher that checks landing pages submitted as ChatGPT ads. | No price recorded | | ChatGPT-User | User-triggered fetcher for ChatGPT, Custom GPTs and GPT Actions. | No price recorded | | ChatGPT agent cloud browser | Hosted browser for ChatGPT's agent mode whose requests are signed with keys published at chatgpt.com. | No price recorded | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | GPTBot and OAI-SearchBot follow robots.txt tokens; OpenAI says robots.txt rules may not apply to ChatGPT-User. (developers.openai.com) | | Web Bot Auth | Supports | chatgpt.com serves a key directory with the Web Bot Auth media type (observed 2026-11-05). (chatgpt.com) | | Content Signals | No statement | Not mentioned on OpenAI's crawler page (read 2026-11-05). | | Published IP list | Supports | Four JSON range files: gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json. (developers.openai.com) | | Reverse DNS check | No statement | OpenAI's crawler page gives no reverse DNS pattern (read 2026-11-05). | | Cloudflare verified bots | No statement | OpenAI's crawler page does not mention Cloudflare's verified bot list (read 2026-11-05). | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | chatgpt.com: 200, application/http-message-signatures-directory+json, 1 key; openai.com: 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 | | gptbot.json | 18 IPv4 prefixes, creationTime 2026-09-22 | GET and count prefixes | 2026-11-05 | | searchbot.json | 39 IPv4 prefixes, creationTime 2026-01-02 | GET and count prefixes | 2026-11-05 | | chatgpt-user.json | 235 IPv4 prefixes, creationTime 2026-10-27 | GET and count prefixes | 2026-11-05 | | adsbot.json | 2 IPv4 prefixes, creationTime 2026-05-12 | GET and count prefixes | 2026-11-05 | ## What OpenAI does not do - OpenAI publishes no reverse DNS pattern for its robots, so verification rests on the IP files or, for the agent browser, on signatures. - Blocking ChatGPT-User in robots.txt is not a guaranteed stop, because OpenAI says robots.txt rules may not apply to user-started fetches. - OpenAI's crawler page names no token for ChatGPT Atlas browsing (read 2026-11-05). ## Reading notes [OpenAI's agent allowlisting help article](https://help.openai.com/en/articles/11845367) returned HTTP 403 to our crawl user agent on 5 November 2026, so what this profile says about agent signing rests on the key directory we observed at chatgpt.com. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [OpenAI in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/openai/) 6 rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) token GPTBot - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) no token ## Sources - [openai.com](https://openai.com) read 2026-11-05 - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) read 2026-11-05 - [OpenAI chatgpt-user.json IP ranges](https://openai.com/chatgpt-user.json) read 2026-11-05 - [OpenAI gptbot.json IP ranges](https://openai.com/gptbot.json) read 2026-11-05 - [chatgpt.com Web Bot Auth key directory path (GET probe)](https://chatgpt.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [openai.com Web Bot Auth key directory path (GET probe)](https://openai.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [OpenAI searchbot.json IP ranges](https://openai.com/searchbot.json) read 2026-11-05 - [OpenAI adsbot.json IP ranges](https://openai.com/adsbot.json) read 2026-11-05 Last reviewed 2026-11-05