Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

User-triggered fetcher

In short

A fetch made because a person asked an assistant something, such as reading a linked page. Operators often say robots.txt may not apply because a user initiated it. 8 rows are live answer fetchers. Example: Perplexity says Perplexity-User generally ignores robots.txt since a user requested the fetch. Related: All live answer fetchers and ChatGPT-User profile.

Definition

A fetch made because a person asked an assistant something, such as reading a linked page. Operators often say robots.txt may not apply because a user initiated it.

Sourced example

Perplexity says Perplexity-User generally ignores robots.txt since a user requested the fetch. Source, read 2026-11-04.

How it shows up here

8 rows are live answer fetchers. Their operators say robots.txt may not apply, yet 9.7 percent of top-10,000 sites refuse ChatGPT-User, 8.3 percent Perplexity-User and 8.5 percent Meta-ExternalFetcher.

Previous: User-agent string. Next: Verified bot. All 41 terms.

Sources

Last reviewed 2026-11-04