Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Search index crawler from Meta

Meta-WebIndexer

In short

Meta-WebIndexer is Meta's search index crawler. The token to write in robots.txt is meta-webindexer. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure meta-webindexer. Its user-agent entry: meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). Meta-WebIndexer: allowing Meta-WebIndexer in robots.txt helps Meta cite and link content in Meta AI responses.

Operator
Meta Vendor profile: Meta
Purpose
Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.
User-agent string
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
robots.txt token
meta-webindexer Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Meta: allowing Meta-WebIndexer in robots.txt helps Meta cite and link content in Meta AI responses. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Not published.
Block rate

Not measured: the crawl checked 31 AI and search tokens, and meta-webindexer is not one of them.

Documentation
developers.facebook.com, read 2026-11-04.

Note. Search-index crawler for Meta AI.

Allow and refuse lines

To allow meta-webindexer

User-agent: meta-webindexer
Allow: /

To refuse meta-webindexer

User-agent: meta-webindexer
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Meta in the Verification Ledger

User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

In the news

Other Meta rows

Sources

Last reviewed 2026-11-04