Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Live answer fetcher from Meta

Meta-ExternalFetcher

In short

Meta-ExternalFetcher is Meta's live answer fetcher. The operator publishes nothing a site can check. 8.5 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Its user-agent entry: meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). Meta-ExternalFetcher may bypass robots.txt because it performs fetches requested by the user.

Operator
Meta Vendor profile: Meta
Purpose
Live answer fetch All live answer fetch rows. Fetches a page because a person asked an assistant about it, one request at a time.
User-agent string
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
robots.txt token
meta-externalfetcher Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

May not apply (user-triggered)

Meta: may bypass robots.txt because it performs fetches requested by the user. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
User-triggered.
Block rate

8.5% of 5,938 top-10,000 sites with a parsed robots.txt disallow Meta-ExternalFetcher at the root (505 sites). 4.7% name it; 3.8% refuse it only through a * group. Rank 22 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 6.0% (3.3% named). Crawl of 2026-11-04; report.

Documentation
developers.facebook.com, read 2026-11-04.

Note. Meta says it supports evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks.

Allow and refuse lines

To allow meta-externalfetcher

User-agent: meta-externalfetcher
Allow: /

To refuse meta-externalfetcher

User-agent: meta-externalfetcher
Disallow: /

Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF.

Meta in the Verification Ledger

User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other Meta rows

Sources

Last reviewed 2026-11-04