Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training crawler from Apple

Applebot-Extended (control token)

In short

Applebot-Extended (control token) is Apple's training crawler. The token to write in robots.txt is Applebot-Extended. The operator publishes nothing a site can check. 14.8 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, disallowing Applebot-Extended opts content out of training generative foundation models; pages can still appear in search.

Operator
Apple Vendor profile: Apple
Purpose
Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.
User-agent string
No exact string is published. What the operator or a third party says: None: Applebot-Extended does not crawl webpages
robots.txt token
Applebot-Extended Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Apple: disallowing Applebot-Extended opts content out of training generative foundation models; pages can still appear in search. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
No crawl of its own.
Block rate

14.8% of 5,938 top-10,000 sites with a parsed robots.txt disallow Applebot-Extended at the root (877 sites). 11.6% name it; 3.1% refuse it only through a * group. Rank 13 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 11.1% (8.8% named). Crawl of 2026-11-04; report.

Documentation
support.apple.com, read 2026-11-04.

Note. Apple says it only determines how data crawled by Applebot is used.

Allow and refuse lines

To allow Applebot-Extended

User-agent: Applebot-Extended
Allow: /

To refuse Applebot-Extended

User-agent: Applebot-Extended
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Apple in the Verification Ledger

IP list and reverse DNS. All operators.

IP lists
Reverse DNS
*.applebot.apple.com
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other Apple rows

Sources

Last reviewed 2026-11-04