Crawler operator
Apple
In short
Applebot gathers pages for Spotlight, Siri and Safari search, and the same crawl can be used when Apple trains its foundation models. Publishers who want out of training aim a Disallow at the Applebot-Extended token, which leaves ordinary search crawling alone. Genuine visits resolve through reverse DNS to applebot.apple.com or fall inside 24 published CIDR prefixes.
- Type
- Crawler operator
- Homepage
- apple.com
- In one line
- Apple runs Applebot for Spotlight, Siri and Safari search and lets publishers refuse generative model training through the Applebot-Extended token, with verification by reverse DNS under applebot.apple.com or a CIDR file.
- Last read
- 2026-11-05
What Apple does for this layer
From the vendor's own pages
Apple says Applebot data powers search in Spotlight, Siri and Safari, and may also help train Apple foundation models behind Apple Intelligence, Services and Developer Tools. (support.apple.com) Apple's training opt-out is a robots.txt Disallow for Applebot-Extended, which leaves normal Applebot crawling for search in place. (support.apple.com) Apple says publishers can keep content out of AI-generated broad world knowledge answers in Siri and Search by applying the nosnippet meta tag to that content. (support.apple.com) Apple says Applebot traffic resolves by reverse DNS into the applebot.apple.com domain and can also be matched against the CIDR prefixes in applebot.json. (support.apple.com)
Apple says that when robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. (support.apple.com) Apple states Applebot does not follow crawl-delay and instead slows its crawl rate automatically when a site slows down or returns errors. (support.apple.com) Apple's iTMS user agent, which crawls URLs tied to registered Apple Podcasts content from applebot.apple.com hosts, does not follow robots.txt. (support.apple.com) Apple warns that Applebot may render pages in a browser, so blocking JavaScript, CSS or XHR resources in robots.txt can stop it rendering content properly. (support.apple.com)
What our crawl saw
Apple's applebot.json listed 24 IPv4 prefixes with a creationTime of 2026-09-15 when fetched on 2026-11-05. (search.developer.apple.com, dated 2026-09-15) www.apple.com returned HTTP 404 for the Web Bot Auth key directory path on 2026-11-05. (apple.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Applebot | Crawler and renderer for Spotlight, Siri and Safari search, whose data may also train Apple foundation models. | |||
| Applebot-Extended | Robots.txt token whose Disallow opts content out of Apple's generative model training. | |||
| iTMS | Special crawler for URLs registered with Apple Podcasts; does not follow robots.txt. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Applebot respects standard robots.txt directives targeted at it in general search crawls; iTMS is the exception. (support.apple.com) |
| Web Bot Auth | No statement | Not on the Applebot page; key directory path 404 on www.apple.com (observed 2026-11-05). |
| Content Signals | No statement | Not mentioned on the Applebot page (read 2026-11-05). |
| Published IP list | Supports | applebot.json lists the CIDR prefixes. (support.apple.com) |
| Reverse DNS check | Supports | Hostnames in *.applebot.apple.com with a forward lookup check. (support.apple.com) |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| applebot.json | 24 IPv4 prefixes, creationTime 2026-09-15 | GET and count prefixes | 2026-11-05 |
What Apple does not do
- Disallowing Applebot-Extended does not stop Applebot from crawling; Apple says the content stays discoverable in Spotlight, Siri and Safari.
- Applebot ignores crawl-delay, so a site that needs a slower crawl cannot ask for one in robots.txt.
Related pages on this site
Sources
Last reviewed 2026-11-05