Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Search index crawler from Perplexity

PerplexityBot

In short

PerplexityBot is Perplexity's search index crawler. The token to write in robots.txt is PerplexityBot. Check a visitor against the operator's published IP list. Changes may take up to 24 hours to reflect, per Perplexity. Perplexity recommends allowing PerplexityBot in robots.txt to appear in results and says it is not used to crawl content for AI foundation models.

Operator
Perplexity Vendor profile: Perplexity
Purpose
Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.
User-agent string
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
robots.txt token
PerplexityBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Perplexity recommends allowing PerplexityBot in robots.txt to appear in results and says it is not used to crawl content for AI foundation models. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges
perplexity.ai/perplexitybot.json
Rate behaviour
Changes may take up to 24 hours to reflect, per Perplexity.
Vendor rule names
  • Cloudflare: Custom rule: (http.user_agent contains "PerplexityBot" and ip.src in {ranges}), action Skip; Perplexity notes Bot Fight Mode cannot be skipped
  • AWS WAF: IP set plus User-Agent string match, action Allow
Block rate

11.2% of 5,938 top-10,000 sites with a parsed robots.txt disallow PerplexityBot at the root (667 sites). 7.9% name it; 3.4% refuse it only through a * group. Rank 15 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 7.9% (5.5% named). Crawl of 2026-11-04; report.

Documentation
docs.perplexity.ai, read 2026-11-04.

Note. Perplexity publishes WAF recipes for Cloudflare and AWS that pair the user agent with its IP list.

Allow and refuse lines

To allow PerplexityBot

User-agent: PerplexityBot
Allow: /

To refuse PerplexityBot

User-agent: PerplexityBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Perplexity in the Verification Ledger

Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet. All operators.

IP lists
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other Perplexity rows

Sources

Last reviewed 2026-11-04