Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Perplexity: 4 rows

In short

Perplexity has 4 rows in the Index: PerplexityBot, Perplexity-User, Comet and Perplexity Computer. They cover 3 purposes: search index, live answer fetch, task agent. 2 of them have a robots.txt token and 2 publish nothing a site can verify. The most refused is PerplexityBot, disallowed at the root by 11.2 percent of top-10,000 sites.

Perplexity: 4 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
Perplexity PerplexityBot Search index Yes IP list 11.2% Profile
Perplexity Perplexity-User Live answer fetch May not apply IP list 8.3% Profile
Perplexity Comet (browser assistant) no token Task agent No docs None published no token Profile
Perplexity Perplexity Computer no token Task agent No docs None published no token Profile

Vendor profile: Perplexity, with its products, standards, our probes and what it does not do.

Verification Ledger entry

Summary
Two IP lists and WAF recipes; no key directory (404 at www.perplexity.ai, 2026-11-04); nothing for Comet.
IP lists
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa and Mastercard
Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).

Block rates for Perplexity tokens

TokenRefused, top 10,000NamedVia *All crawled
PerplexityBot11.2%7.9%3.4%7.9%
Perplexity-User8.3%4.6%3.7%5.9%

Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.

Sources

Last reviewed 2026-11-04