Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

OpenAI: 6 rows

In short

OpenAI has 6 rows in the Index: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ChatGPT agent and ChatGPT Atlas. They cover 5 purposes: training, search index, live answer fetch, other, task agent. 4 of them have a robots.txt token and 1 publish nothing a site can verify. The most refused is GPTBot, disallowed at the root by 18.5 percent of top-10,000 sites.

OpenAI: 6 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
OpenAI GPTBot Training Yes IP list 18.5% Profile
OpenAI OAI-SearchBot Search index Yes IP list 7.7% Profile
OpenAI ChatGPT-User Live answer fetch May not apply IP list 9.7% Profile
OpenAI OAI-AdsBot Other No statement IP list not measured Profile
OpenAI ChatGPT agent (Cloud browser, formerly Operator) no token Task agent No statement Web Bot Auth no token Profile
OpenAI ChatGPT Atlas (browser agent mode) no token Task agent No docs None published no token Profile

Vendor profile: OpenAI, with its products, standards, our probes and what it does not do.

Verification Ledger entry

Summary
One IP JSON per crawler plus a Web Bot Auth key directory for the Cloud browser. The most complete set read.
IP lists
Reverse DNS
publishes nothing
Key directory
https://chatgpt.com/.well-known/http-message-signatures-directory
Cloudflare
GPTBot and ChatGPT agent listed Verified on Radar (read 2026-11-04); ChatGPT agent is a Cloudflare signed agent, detection ID 129220581
Visa and Mastercard
Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).

Block rates for OpenAI tokens

TokenRefused, top 10,000NamedVia *All crawled
GPTBot18.5%15.2%3.3%13.9%
ChatGPT-User9.7%6.3%3.3%6.9%
OAI-SearchBot7.7%4.4%3.3%5.4%

Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.

In the news

Sources

Last reviewed 2026-11-04