Crawler operator
Mistral AI
In short
Mistral AI documents full user agent strings for all three of its crawlers and IP files for two of them, listing four prefixes for the user fetcher and two for the index crawler. Nothing but its name identifies the training crawler. Mistral says the data its index crawler gathers never goes into generative training.
- Type
- Crawler operator
- Homepage
- mistral.ai
- In one line
- Mistral AI documents three crawlers, MistralAI-User for Vibe user requests, MistralAI-Index for its search index and MistralAI-Training for model training data, and publishes IP files for the first two.
- Last read
- 2026-11-05
What Mistral AI does for this layer
From the vendor's own pages
Mistral's crawler page lists three user agents, MistralAI-User, MistralAI-Index and MistralAI-Training, each with its own robots.txt tag. (docs.mistral.ai) Mistral says MistralAI-User acts on user requests in Vibe and is not used for automatic crawling or to collect content for generative AI training. (docs.mistral.ai) Mistral says content crawled by MistralAI-Index, which feeds Mistral search in Vibe, is not used for generative AI training of any kind. (docs.mistral.ai) Mistral says MistralAI-Training builds datasets for training its generative models, is not used for search or live answers, and can be disallowed in robots.txt. (docs.mistral.ai)
Mistral publishes full user agent strings, such as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots). (docs.mistral.ai)
Claims the vendor makes about itself
Mistral says on its about page that the company was founded in April 2023. (mistral.ai)
What our crawl saw
Mistral's mistralai-user-ips.json held 4 IPv4 prefixes with a creationTime of 2025-02-19 on 2026-11-05. (mistral.ai, dated 2025-02-19)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| MistralAI-User | Fetcher for pages a Vibe user's question needs. | |||
| MistralAI-Index | Crawler for the search index behind Vibe. | |||
| MistralAI-Training | Crawler that gathers training data for Mistral's generative models. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Mistral uses a robots.txt tag per crawler; the MistralAI-User tag governs which sites user requests can reach, and MistralAI-Training can be disallowed. (docs.mistral.ai) |
| Web Bot Auth | No statement | Not on the crawler page; key directory path 404 (observed 2026-11-05). |
| Published IP list | Partial | IP files for MistralAI-User and MistralAI-Index; none listed for MistralAI-Training. (docs.mistral.ai) |
| Reverse DNS check | No statement | No reverse DNS pattern on the crawler page (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| mistralai-user-ips.json | 4 IPv4 prefixes, creationTime 2025-02-19 | GET and count prefixes | 2026-11-05 |
| mistralai-index-ips.json | 2 IPv4 prefixes, creationTime 2026-04-19 | GET and count prefixes | 2026-11-05 |
What Mistral AI does not do
- Mistral lists no IP file for MistralAI-Training, so the training crawler can be checked by user agent only (read 2026-11-05).
- Mistral's crawler page gives no reverse DNS pattern or signing keys for any of its three crawlers (read 2026-11-05).
Related pages on this site
Sources
Last reviewed 2026-11-05