Glossary
Mixed-use crawler
In short
A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. Refusal in the crawl differs widely: Googlebot 2.8 percent, Bingbot 3.1 percent, Applebot 14.8 percent of top-10,000 sites, so the cost of refusing a mixed-use crawler is known to most sites. Related: Googlebot profile and Applebot profile.
Definition
A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search.
Sourced example
Cloudflare names Applebot, Bingbot and Googlebot as mixed-use crawlers in its 2026-09-15 post. Source, read 2026-11-04.
How it shows up here
Refusal in the crawl differs widely: Googlebot 2.8 percent, Bingbot 3.1 percent, Applebot 14.8 percent of top-10,000 sites, so the cost of refusing a mixed-use crawler is known to most sites.
Previous: Mastercard Agent Pay. Next: noarchive. All 41 terms.
Sources
Last reviewed 2026-11-04