Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Applebot-Extended

In short

Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself. 14.8 percent of top-10,000 sites refuse it, 11.6 percent by naming it. Example: Apple says pages that disallow Applebot-Extended can still be included in search results. Related: Applebot-Extended profile and Allow search, refuse training.

Definition

Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself.

Sourced example

Apple says pages that disallow Applebot-Extended can still be included in search results. Source, read 2026-11-04.

How it shows up here

14.8 percent of top-10,000 sites refuse it, 11.6 percent by naming it.

Previous: AIPREF training signal. Next: Bot management. All 41 terms.

Sources

Last reviewed 2026-11-04