Crawler operator
Cohere
In short
Cohere has told site owners that it currently runs no crawler gathering pages for model training, and expects any future one to be built around robots.txt, with page changes announced in its docs changelog. Fetches its products make on a user's behalf fall outside that page, and the sample opt-out it shows names Coherebot rather than cohere-ai.
- Type
- Crawler operator
- Homepage
- cohere.com
- In one line
- Cohere says it runs no bots or user agents that crawl web content to train generative models at this time, and that any future crawler will respect robots.txt and be listed on its crawler page.
- Last read
- 2026-11-05
What Cohere does for this layer
From the vendor's own pages
Cohere's crawler table listed N/A for both bot name and description when read on 2026-11-05. (docs.cohere.com) Cohere states that its policy is to require that crawlers be designed to respect robots.txt. (docs.cohere.com) Cohere's example robots.txt block for a future crawler uses the token Coherebot; the page does not mention cohere-ai. (docs.cohere.com) Cohere warns that blocking IP addresses may not give a reliable or lasting opt-out, because it prevents a crawler from reading robots.txt. (docs.cohere.com)
Cohere says changes to the crawler page will be announced in the changelog on its docs site, which offers an RSS feed. (docs.cohere.com)
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Cohere's policy requires crawlers designed to respect robots.txt. (docs.cohere.com) |
| Web Bot Auth | No statement | Key directory path 404 on cohere.com (observed 2026-11-05). |
| Published IP list | No statement | No crawler, so no IP list (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
What Cohere does not do
- Cohere publishes no user agent string, IP list or verification method, because it says it runs no training crawler.
- Cohere's crawler page covers training crawlers only and says nothing about fetches made by its products at a user's request.
Reading notes
The cohere-ai row in the Crawler Index takes its token from a third party. Cohere's own crawler page names no crawler and uses Coherebot only in an example, so the two disagree.
Related pages on this site
Sources
Last reviewed 2026-11-05