---
title: "Mixed-use crawler: definition | Agent Ready Crawlers"
description: "Mixed-use crawler: A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search."
canonical: https://agentreadycrawlers.com/glossary/mixed-use-crawler
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Glossary"
---

# Mixed-use crawler: definition | Agent Ready Crawlers

> Mixed-use crawler: A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search.

Source: https://agentreadycrawlers.com/glossary/mixed-use-crawler
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Glossary

# Mixed-use crawler

In short

A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. Refusal in the crawl differs widely: Googlebot 2.8 percent, Bingbot 3.1 percent, Applebot 14.8 percent of top-10,000 sites, so the cost of refusing a mixed-use crawler is known to most sites. Related: Googlebot profile and Applebot profile.

## Definition

A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search.

## Sourced example

Cloudflare names Applebot, Bingbot and Googlebot as mixed-use crawlers in its 2026-09-15 post. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04.

## How it shows up here

Refusal in the crawl differs widely: Googlebot 2.8 percent, Bingbot 3.1 percent, Applebot 14.8 percent of top-10,000 sites, so the cost of refusing a mixed-use crawler is known to most sites.

-   [Googlebot profile](https://agentreadycrawlers.com/crawlers/googlebot/)
-   [Applebot profile](https://agentreadycrawlers.com/crawlers/applebot/)
-   [Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/)

Previous: [Mastercard Agent Pay](https://agentreadycrawlers.com/glossary/mastercard-agent-pay/). Next: [noarchive](https://agentreadycrawlers.com/glossary/noarchive/). [All 41 terms](https://agentreadycrawlers.com/glossary/).

## Sources

-   [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04

Last reviewed 2026-11-04