--- title: "GPTBot: user agent, token, block rate | Agent Ready Crawlers" description: "GPTBot from OpenAI: exact user-agent string, robots.txt token GPTBot, training purpose and how to verify it. 18.5% of top sites block it. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/gptbot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # GPTBot: user agent, token, block rate | Agent Ready Crawlers > GPTBot from OpenAI: exact user-agent string, robots.txt token GPTBot, training purpose and how to verify it. 18.5% of top sites block it. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/gptbot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Training crawler from OpenAI # GPTBot In short GPTBot is OpenAI's training crawler. The token to write in robots.txt is GPTBot. Check a visitor against the operator's published IP list. 18.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Operator [OpenAI](https://agentreadycrawlers.com/crawlers/operator/openai/) Vendor profile: [OpenAI](https://agentreadycrawlers.com/vendors/openai/) Purpose Training [All training rows](https://agentreadycrawlers.com/crawlers/purpose/training/). Collects content to train or tune a model. Refusing it is the training opt-out. User-agent string `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot` robots.txt token `GPTBot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator OpenAI: disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. Verification Published IP list (JSON) Match the source address of a request against the operator's published list. [Documentation](https://openai.com/gptbot.json), read 2026-11-04. IP ranges [openai.com/gptbot.json](https://openai.com/gptbot.json) Rate behaviour OpenAI says changes to robots.txt take about 24 hours to reach its systems, and that if both GPTBot and OAI-SearchBot are allowed it may use one crawl for both uses. gptbot.json creationTime read: 2026-09-22. Block rate **18.5%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `GPTBot` at the root (1,097 sites). 15.2% name it; 3.3% refuse it only through a \* group. Rank 3 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 13.9% (11.5% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [platform.openai.com](https://platform.openai.com/docs/bots), read 2026-11-04. Note. Version 1.4 in the published example string. When fetching robots.txt it may add a "robots.txt" marker to the user agent. Listed as Verified on Cloudflare Radar (radar.cloudflare.com/bots/directory/gptbot, read 2026-11-04). ## Allow and refuse lines ### To allow GPTBot User-agent: GPTBot Allow: / ### To refuse GPTBot User-agent: GPTBot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## OpenAI in the Verification Ledger One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [openai.com/gptbot.json](https://openai.com/gptbot.json) - [openai.com/searchbot.json](https://openai.com/searchbot.json) - [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) - [openai.com/adsbot.json](https://openai.com/adsbot.json) Reverse DNS publishes nothing Key directory https://chatgpt.com/.well-known/http-message-signatures-directory Cloudflare GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581 Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## Other OpenAI rows - [OAI-SearchBot](https://agentreadycrawlers.com/crawlers/oai-searchbot/) Search index token OAI-SearchBot - [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/) Live answer fetch token ChatGPT-User - [OAI-AdsBot](https://agentreadycrawlers.com/crawlers/oai-adsbot/) Other token OAI-AdsBot - [ChatGPT agent (Cloud browser, formerly Operator)](https://agentreadycrawlers.com/crawlers/chatgpt-agent/) Task agent no token - [ChatGPT Atlas (browser agent mode)](https://agentreadycrawlers.com/crawlers/chatgpt-atlas/) Task agent no token ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [openai.com: gptbot (JSON)](https://openai.com/gptbot.json) read 2026-11-04 - [Cloudflare Radar: GPTBot](https://radar.cloudflare.com/bots/directory/gptbot) read 2026-11-04 Last reviewed 2026-11-04