Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Apple: 2 rows

In short

Apple has 2 rows in the Index: Applebot and Applebot-Extended. They cover 2 purposes: search index, training. 2 of them have a robots.txt token and 1 publish nothing a site can verify. The most refused is Applebot, disallowed at the root by 14.8 percent of top-10,000 sites. IP list and reverse DNS.

Apple: 2 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
Apple Applebot Search index Yes Reverse DNS 14.8% Profile
Apple Applebot-Extended Training Yes None published 14.8% Profile

Vendor profile: Apple, with its products, standards, our probes and what it does not do.

Verification Ledger entry

Summary
IP list and reverse DNS.
IP lists
Reverse DNS
*.applebot.apple.com
Key directory
publishes nothing
Cloudflare
not checked
Visa and Mastercard
Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).

Block rates for Apple tokens

TokenRefused, top 10,000NamedVia *All crawled
Applebot14.8%11.6%3.1%11.1%
Applebot-Extended14.8%11.6%3.1%11.1%

Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.

In the news

Sources

Last reviewed 2026-11-04