Crawler operator
Exa
In short
Exa's search API draws on its own crawler, ExaSearchBot, whose requests are signed with a single published Ed25519 key; there is no IP list. A noindex tag removes a page from the index only after ExaSearchBot's next visit, since robots.txt governs crawling, not indexing.
- Type
- Crawler operator
- Homepage
- exa.ai
- In one line
- AI search API company whose ExaSearchBot crawler signs every request with Web Bot Auth, publishes its key directory and robots.txt rules on crawler.exa.ai, and names separate ExaUser and ExaSearchBot-Images identities.
- Last read
- 2026-11-05
What Exa does for this layer
From the vendor's own pages
Exa's crawler page names three identities: ExaSearchBot for crawling, ExaUser for pages fetched on a user's request, and ExaSearchBot-Images for images on the pages it crawls. (crawler.exa.ai) ExaSearchBot sends the user agent "Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)" and its robots.txt product token is ExaSearchBot. (crawler.exa.ai) Exa signs every ExaSearchBot request with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme, carrying Signature, Signature-Input and Signature-Agent headers, and publishes its Ed25519 keys on crawler.exa.ai. (crawler.exa.ai) When no robots.txt group addresses ExaSearchBot, Exa says the crawler follows the rules a site sets for major search engines, and only otherwise the * group. (crawler.exa.ai)
Exa says robots.txt controls crawling but not indexing, and that it drops an indexed page after its next re-fetch when the page carries noindex in a robots meta tag or an X-Robots-Tag header. (crawler.exa.ai) Exa says ExaSearchBot is rate-limited per site, does not try to bypass logins, paywalls or CAPTCHAs, and submits no forms. (crawler.exa.ai) Exa's FAQ confirms the company operates ExaSearchBot to discover and refresh public pages for search and retrieval and points site owners to crawler.exa.ai for crawler controls. (exa.ai) Exa is pay-as-you-go: a Free Tier of USD 10 in credits that resets monthly, search from USD 4 per 1,000 requests, Contents at USD 1 per 1,000 pages, and an annual Enterprise plan at USD 12,000 paid upfront. (exa.ai)
Claims the vendor makes about itself
Exa says ExaSearchBot is validated automatically at the edge on Cloudflare, Akamai and other providers that support Web Bot Auth, with nothing for the site owner to configure. (crawler.exa.ai)
What our crawl saw
ExaSearchBot's key directory at crawler.exa.ai/.well-known/http-message-signatures-directory returned HTTP 200, content type application/http-message-signatures-directory+json, with one Ed25519 key on 2026-11-05. (crawler.exa.ai) Exa's own robots.txt, read on 2026-11-05, declares Content-Signal "search=yes, ai-input=yes" for the whole site and adds ai-train=yes for /docs and /llms.txt. (exa.ai)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Exa Search, Contents and Answer APIs | Web search and page retrieval for agents, built on the ExaSearchBot index. | pay as you go | USD 4 | 2026-11-05 |
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | ExaSearchBot respects the Robots Exclusion Protocol, with its own token and a fallback to search-engine rules. (crawler.exa.ai) |
| Web Bot Auth | Supports | Every ExaSearchBot request is signed; keys at crawler.exa.ai. (crawler.exa.ai) |
| Content Signals | No statement | Exa uses Content-Signal lines in its own robots.txt, but the crawler page does not say whether ExaSearchBot reads them. |
| Published IP list | No statement | The crawler page relies on signatures and says verification holds regardless of IP; no IP list published. |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key | GET https://crawler.exa.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 404 | GET https://exa.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
What Exa does not do
- Exa publishes no IP range list for ExaSearchBot; its crawler page relies on signatures instead (read 2026-11-05).
- Exa's crawler page does not say whether ExaUser requests follow robots.txt.
Reading notes
The ExaSearchBot row in the Crawler Index, read on 4 November 2026, says Exa publishes no bot documentation. The page at crawler.exa.ai, read on 5 November, now documents the bot, its keys and its robots.txt rules.
Related pages on this site
Sources
Last reviewed 2026-11-05