--- title: "OAI-SearchBot: user agent, token, block rate | Agent Ready Crawlers" description: "OAI-SearchBot from OpenAI: exact user-agent string, robots.txt token OAI-SearchBot, search index purpose and how to verify it. 7.7% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/oai-searchbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # OAI-SearchBot: user agent, token, block rate | Agent Ready Crawlers > OAI-SearchBot from OpenAI: exact user-agent string, robots.txt token OAI-SearchBot, search index purpose and how to verify it. 7.7% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/oai-searchbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from OpenAI # OAI-SearchBot In short OAI-SearchBot is OpenAI's search index crawler. The token to write in robots.txt is OAI-SearchBot. Check a visitor against the operator's published IP list. About 24 hours from a robots.txt change to OpenAI's systems adjusting, per OpenAI. On robots.txt, sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot` robots.txt token `OAI-SearchBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://openai.com/searchbot.json), read 2026-11-04. IP ranges [openai.com/searchbot.json](https://openai.com/searchbot.json) Rate behaviour About 24 hours from a robots.txt change to OpenAI's systems adjusting, per OpenAI. Block rate **7.7%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `OAI-SearchBot` at the root (458 sites). 4.4% name it; 3.3% refuse it only through a \* group. Rank 26 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 5.4% (3.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [platform.openai.com](https://platform.openai.com/docs/bots), read 2026-11-04. Note. Robots.txt fetches may carry the extra marker "; robots.txt;" in the user agent string so they can be told apart in logs that omit paths. ## Allow and refuse lines ### To allow OAI-SearchBot User-agent: OAI-SearchBot Allow: / ### To refuse OAI-SearchBot User-agent: OAI-SearchBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [GPTBot](https://agentreadycrawlers.com/crawlers/gptbot/) Training token GPTBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: searchbot (JSON)](https://openai.com/searchbot.json) read 2026-11-04 Last reviewed 2026-11-04