Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training crawler from Meta

Meta-ExternalAgent

In short

Meta-ExternalAgent is Meta's training crawler. The token to write in robots.txt is meta-externalagent. The operator publishes nothing a site can check. 16.0 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Meta documents blocking with a robots.txt disallow for the crawler; the file may be cached up to 24 hours.

Operator
Meta Vendor profile: Meta
Purpose
Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.
User-agent string
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
robots.txt token
meta-externalagent Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Meta documents blocking with a robots.txt disallow for the crawler; robots.txt may be cached up to 24 hours. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Not published.
Block rate

16.0% of 5,938 top-10,000 sites with a parsed robots.txt disallow Meta-ExternalAgent at the root (950 sites). 12.3% name it; 3.7% refuse it only through a * group. Rank 6 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 12.0% (9.4% named). Crawl of 2026-11-04; report.

Documentation
developers.facebook.com, read 2026-11-04.

Note. Crawls for training foundation AI models or indexing content directly, per Meta. Meta publishes no IP list or reverse-DNS pattern on this page.

Allow and refuse lines

To allow meta-externalagent

User-agent: meta-externalagent
Allow: /

To refuse meta-externalagent

User-agent: meta-externalagent
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Meta in the Verification Ledger

User agent strings only; the page recommends IP allowlisting but links no list. Publishes nothing for Muse. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

In the news

Other Meta rows

Sources

Last reviewed 2026-11-04