---
title: "Search-index crawler: definition | Agent Ready Crawlers"
description: "Search-index crawler: A crawler that builds an index used to surface and cite pages in search or AI answers. Read 2026-11-04."
canonical: https://agentreadycrawlers.com/glossary/search-index-crawler
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Glossary"
---

# Search-index crawler: definition | Agent Ready Crawlers

> Search-index crawler: A crawler that builds an index used to surface and cite pages in search or AI answers. Read 2026-11-04.

Source: https://agentreadycrawlers.com/glossary/search-index-crawler
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Glossary

# Search-index crawler

In short

A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training. 15 Index rows are filed under Search index. Among measured tokens the range is 2.8 to 14.8 percent of top-10,000 sites refusing them. Related: All search index rows and OAI-SearchBot profile.

## Definition

A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training.

## Sourced example

PerplexityBot surfaces sites in Perplexity search results and is not used to crawl content for foundation models. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04.

## How it shows up here

15 Index rows are filed under Search index. Among measured tokens the range is 2.8 to 14.8 percent of top-10,000 sites refusing them.

-   [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/)
-   [OAI-SearchBot profile](https://agentreadycrawlers.com/crawlers/oai-searchbot/)

Previous: [Robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/). Next: [Signature-Agent header](https://agentreadycrawlers.com/glossary/signature-agent-header/). [All 41 terms](https://agentreadycrawlers.com/glossary/).

## Sources

-   [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04

Last reviewed 2026-11-04