Crawler operator
xAI
In short
xAI gives Grok Bot an always-on cloud computer of its own, shared by an account's other Bots, and lets it hand control back to the user at a CAPTCHA or a passkey. A site learns little about it: neither x.ai nor grok.com serves a key directory, and a third party saw signing end in mid-October.
- Type
- Crawler operator
- Homepage
- x.ai
- In one line
- xAI operates Grok Bot, an agent product in which each Bot works on a persistent cloud computer with a browser; xAI's docs give site owners no user agent, IP list or signing keys for that traffic.
- Last read
- 2026-11-05
What xAI does for this layer
From the vendor's own pages
xAI's docs say each Grok Bot works on a persistent cloud computer with a browser, filesystem and terminal, and keeps working while the user's laptop is closed. (docs.x.ai) xAI includes Grok Bot with the Premium, Super and Ultra tiers of The X Subscription and with paid individual plans on legacy Cursor plans. (docs.x.ai) xAI says all Bots on one account share a single cloud computer, including browser cookies, signed-in sessions and files. (docs.x.ai) Grok Bot hands the computer back to the user for steps it cannot finish, which xAI lists as single sign-on, passkeys, approvals on another device and "A CAPTCHA it couldn't get past". (docs.x.ai)
Grok Bot can pay at checkout with a card saved in its Vault, a card entered in a Secure Form, or a connected wallet such as Link or X Money. (docs.x.ai) xAI's API web_search tool lets developers limit browsing with allowed_domains or excluded_domains, up to five domains each. (docs.x.ai)
Reported by a third party
Verisoul reported on 2026-11-03 that Grok Bot stopped signing requests with Web Bot Auth in mid-October 2026 and routes its cloud browser traffic through a reverse proxy on the user's own device, so sites see the user's IP. (verisoul.ai, dated 2026-11-03)
What our crawl saw
The Web Bot Auth key directory path returned HTTP 404 on both x.ai and grok.com on 2026-11-05. (x.ai)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Grok Bot | Always-on agent product; each Bot browses and acts from a persistent cloud computer. | |||
| Web Search tool (xAI API) | Server-side search and browsing tool for API callers, with domain allow and exclude filters. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | No statement | xAI's docs index lists no crawler or robots.txt page (read 2026-11-05). |
| Web Bot Auth | No statement | No key directory on x.ai or grok.com (observed 404, 2026-11-05). |
| Published IP list | No statement | No IP list in xAI's docs (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 403 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 on x.ai and grok.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
What xAI does not do
- xAI publishes no robots.txt token for any crawler or fetcher in its docs index.
Related pages on this site
Sources
Last reviewed 2026-11-05