Crawler operator
Meta
In short
Meta lists five crawlers, for model training, Meta AI search, user-driven fetching, advertising and link previews. It adds that two of them may skip robots.txt, and that changes can take up to 24 hours to be seen. Five Web Bot Auth keys are served from www.meta.com, but no Meta page says which agent uses them.
- Type
- Crawler operator
- Homepage
- meta.com
- In one line
- Meta documents five crawlers, FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher, with robots.txt tokens but no IP list or verification method, and says two of them may bypass robots.txt.
- Last read
- 2026-11-05
What Meta does for this layer
From the vendor's own pages
Meta says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. (developers.facebook.com) Meta says Meta-WebIndexer crawls to improve Meta AI search results and that allowing it helps Meta cite and link to a site in Meta AI responses. (developers.facebook.com) Meta says Meta-ExternalFetcher fetches links at a user's request, including to help AI complete tasks on websites, and "may bypass robots.txt rules". (developers.facebook.com) Meta says FacebookExternalHit might bypass robots.txt when it runs security or integrity checks, such as checking for malware. (developers.facebook.com)
Meta asks site owners to allow up to 24 hours for robots.txt changes, because its crawlers may cache the file for that long. (developers.facebook.com) Meta's page says it prefers robots.txt over non-standard formats such as NoAI tags for publishers to state their preferences. (developers.facebook.com) Meta says FacebookExternalHit needs Open Graph properties within the first 1 MB of a page and servers that support gzip and deflate encodings. (developers.facebook.com)
Reported by a third party
Verisoul reported on 2026-11-01 that Meta publishes no user agent token, signature or IP list for its Muse agent, which browses from Meta's cloud as desktop Chrome on Linux. (verisoul.ai, dated 2026-11-01)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Meta-ExternalAgent | Crawler for AI model training and direct indexing. | |||
| Meta-WebIndexer | Crawler for Meta AI search results and citations. | |||
| Meta-ExternalFetcher | User-requested fetcher that may bypass robots.txt. | |||
| Meta-ExternalAds | Crawler for advertising and business products. | |||
| FacebookExternalHit | Link preview crawler for content shared in Meta's apps. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Partial | Meta asks publishers to use robots.txt; Meta-ExternalFetcher and FacebookExternalHit may bypass it. (developers.facebook.com) |
| Web Bot Auth | No statement | Meta's crawler page says nothing about signatures, although www.meta.com served a key directory on 2026-11-05 (observed). |
| Published IP list | No statement | Meta's crawler page links no IP range list (read 2026-11-05). |
| Reverse DNS check | No statement | No reverse DNS pattern on the crawler page (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | www.meta.com: 200, application/http-message-signatures-directory+json, 5 keys (undocumented) | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
What Meta does not do
- Meta's crawler page links no IP range list, so its documented tokens cannot be checked against addresses from that page.
- Meta's crawler page does not say which Meta agent, if any, signs with the keys served at www.meta.com.
- Meta's current crawler page no longer lists FacebookBot (read 2026-11-05).
Reading notes
Meta's robots.txt at www.facebook.com disallows our crawl token for the .well-known path, so we did not probe it for a key directory; the five keys at www.meta.com were found instead.
Related pages on this site
Sources
Last reviewed 2026-11-05