Crawler operator
Perplexity
In short
Perplexity splits its traffic between an index crawler and a user-triggered fetcher, ships allow-rule recipes for two WAF products, and publishes JSON range files that sites are told to treat as the source of truth. PerplexityBot is said not to feed foundation model training, while the fetcher generally disregards robots.txt.
- Type
- Crawler operator
- Homepage
- perplexity.ai
- In one line
- Perplexity documents two identities for site owners, PerplexityBot for its search index and Perplexity-User for fetches a user asks for, each with its own IP file, and publishes allow-rule recipes for Cloudflare WAF and AWS WAF.
- Last read
- 2026-11-05
What Perplexity does for this layer
From the vendor's own pages
Perplexity says PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for AI foundation models. (docs.perplexity.ai) Perplexity says Perplexity-User visits pages when users ask Perplexity a question and that, since a user requested the fetch, it "generally ignores robots.txt rules". (docs.perplexity.ai) Perplexity's Cloudflare recipe is a custom rule that matches each user agent against its own IP ranges and uses the Skip action, placed above any rule that blocks or challenges. (docs.perplexity.ai) Perplexity notes that Cloudflare's Bot Fight Mode cannot be skipped by WAF custom rules and may still challenge its bots. (docs.perplexity.ai)
Perplexity points out that Cloudflare's Free plan allows only one custom IP list, which is not enough to hold the ranges of both bots. (docs.perplexity.ai) Perplexity's AWS WAF recipe pairs an IP set match with a User-Agent string match in one AND statement whose action is Allow. (docs.perplexity.ai) Perplexity tells site owners to refresh their WAF rules from its two JSON endpoints automatically and to treat those files as the source of truth. (docs.perplexity.ai)
What our crawl saw
Perplexity's perplexitybot.json listed 13 IPv4 prefixes with a creationTime of 2026-10-28 when fetched on 2026-11-05. (perplexity.ai, dated 2026-10-28) No Web Bot Auth key directory was found at www.perplexity.ai on 2026-11-05; the well-known path returned HTTP 404. (perplexity.ai)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| PerplexityBot | Crawler for the Perplexity search index; Perplexity says it is not used for foundation model training. | |||
| Perplexity-User | Fetcher for pages a user's question needs; Perplexity says it generally ignores robots.txt. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Partial | PerplexityBot is managed through robots.txt; Perplexity-User generally ignores robots.txt rules. (docs.perplexity.ai) |
| Web Bot Auth | No statement | Not on the crawler page; key directory path 404 on www.perplexity.ai (observed 2026-11-05). |
| Published IP list | Supports | perplexitybot.json and perplexity-user.json, one file per bot. (docs.perplexity.ai) |
| Reverse DNS check | No statement | Not on the crawler page (read 2026-11-05). |
| Cloudflare verified bots | No statement | The page gives a Cloudflare WAF Skip rule recipe but says nothing about verified bot status (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| perplexitybot.json | 13 IPv4 prefixes, creationTime 2026-10-28 | GET and count prefixes | 2026-11-05 |
| perplexity-user.json | 10 IPv4 prefixes, creationTime 2026-10-28 | GET and count prefixes | 2026-11-05 |
What Perplexity does not do
- Perplexity's crawler page documents no identifier for the Comet browser or for Perplexity Computer (read 2026-11-05).
- A robots.txt rule does not reliably stop Perplexity-User, which Perplexity says generally ignores robots.txt.
Related pages on this site
Sources
Last reviewed 2026-11-05