Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Live answer fetcher from Perplexity

Perplexity-User

In short

Perplexity-User is Perplexity's live answer fetcher. Check a visitor against the operator's published IP list. 8.3 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user). Perplexity-User: since a user requested the fetch, this fetcher generally ignores robots.txt rules.

Operator
Perplexity Vendor profile: Perplexity
Purpose
Live answer fetch All live answer fetch rows. Fetches a page because a person asked an assistant about it, one request at a time.
User-agent string
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
robots.txt token
Perplexity-User Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

May not apply (user-triggered)

Perplexity: since a user requested the fetch, this fetcher generally ignores robots.txt rules. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges
perplexity.ai/perplexity-user.json
Rate behaviour
User-triggered; no crawl rate published.
Vendor rule names
  • Cloudflare: Custom rule: (http.user_agent contains "Perplexity-User" and ip.src in {ranges})
  • AWS WAF: IP set plus User-Agent string match
Block rate

8.3% of 5,938 top-10,000 sites with a parsed robots.txt disallow Perplexity-User at the root (495 sites). 4.6% name it; 3.7% refuse it only through a * group. Rank 23 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 5.9% (3.2% named). Crawl of 2026-11-04; report.

Documentation
docs.perplexity.ai, read 2026-11-04.

Note. Robots.txt lines for Perplexity-User are a preference Perplexity says it generally does not apply; enforce at the WAF if needed.

Allow and refuse lines

To allow Perplexity-User

User-agent: Perplexity-User
Allow: /

To refuse Perplexity-User

User-agent: Perplexity-User
Disallow: /

Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF.

Perplexity in the Verification Ledger

Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. All operators.

IP lists
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other Perplexity rows

Sources

Last reviewed 2026-11-04