---
title: "Cloudflare adds Disallow AI Training | Agent Ready Crawlers"
description: "Cloudflare adds Disallow AI Training and the Accountable label: a no-training line in robots.txt that keeps mixed-use crawlers in search. Dated 2026-09-15."
canonical: https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
date: 2026-09-15
section: "News"
---

# Cloudflare adds Disallow AI Training | Agent Ready Crawlers

> Cloudflare adds Disallow AI Training and the Accountable label: a no-training line in robots.txt that keeps mixed-use crawlers in search. Dated 2026-09-15.

Source: https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

News from 2026-09-15

# Cloudflare adds Disallow AI Training and the Accountable label

In short

15 September 2026: Cloudflare adds Disallow AI Training and the Accountable label. Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. Block now applies to mixed-use crawlers too, Block AI Bots is deprecated, and Managed Robots.txt becomes Bot Preference Sync.

Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. Block now applies to mixed-use crawlers too, Block AI Bots is deprecated, and Managed Robots.txt becomes Bot Preference Sync.

Primary source [blog.cloudflare.com/accountable-mixed-use-ai-crawlers/](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04.

## Where it stands in the Index

[Cloudflare](https://agentreadycrawlers.com/vendors/cloudflare/)

Vendor. CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta.

[Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/)

Google, search index. Token `Googlebot`. Verification: reverse dns. Refused by 2.8% of top-10,000 sites.

[Applebot](https://agentreadycrawlers.com/crawlers/applebot/)

Apple, search index. Token `Applebot`. Verification: reverse dns. Refused by 14.8% of top-10,000 sites.

[Bingbot](https://agentreadycrawlers.com/crawlers/bingbot/)

Microsoft, search index. Token `bingbot`. Verification: reverse dns. Refused by 3.1% of top-10,000 sites.

[All news](https://agentreadycrawlers.com/news/).

## Sources

-   [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04

Last reviewed 2026-11-04