Glossary
Search-index crawler
In short
A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training. 15 Index rows are filed under Search index. Among measured tokens the range is 2.8 to 14.8 percent of top-10,000 sites refusing them. Related: All search index rows and OAI-SearchBot profile.
Definition
A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training.
Sourced example
PerplexityBot surfaces sites in Perplexity search results and is not used to crawl content for foundation models. Source, read 2026-11-04.
How it shows up here
15 Index rows are filed under Search index. Among measured tokens the range is 2.8 to 14.8 percent of top-10,000 sites refusing them.
Previous: Robots.txt token. Next: Signature-Agent header. All 41 terms.
Sources
Last reviewed 2026-11-04