News from 2026-09-15

Cloudflare adds Disallow AI Training and the Accountable label

In short

15 September 2026: Cloudflare adds Disallow AI Training and the Accountable label. Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. Block now applies to mixed-use crawlers too, Block AI Bots is deprecated, and Managed Robots.txt becomes Bot Preference Sync.

Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. Block now applies to mixed-use crawlers too, Block AI Bots is deprecated, and Managed Robots.txt becomes Bot Preference Sync.

Primary source blog.cloudflare.com/accountable-mixed-use-ai-crawlers/, read 2026-11-04.

Where it stands in the Index

Cloudflare

Vendor. CDN and security network that sorts crawlers and agents by purpose (search, training, agent), verifies them by IP or Web Bot Auth signatures, and offers per-request charging for AI crawlers in private beta.

Googlebot

Google, search index. Token Googlebot. Verification: reverse dns. Refused by 2.8% of top-10,000 sites.

Applebot

Apple, search index. Token Applebot. Verification: reverse dns. Refused by 14.8% of top-10,000 sites.

Bingbot

Microsoft, search index. Token bingbot. Verification: reverse dns. Refused by 3.1% of top-10,000 sites.

All news.

Last reviewed 2026-11-04