The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Bot Preference Sync

In short

The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves, replacing Managed Robots.txt from 2026-09-15. It replaced Managed Robots.txt on 2026-09-15, so a Cloudflare-served robots.txt may differ from the file in your repository. Example: Customers who enabled Managed Robots.txt migrate to Bot Preference Sync, per Cloudflare.

Definition

The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves, replacing Managed Robots.txt from 2026-09-15.

Sourced example

Customers who enabled Managed Robots.txt migrate to Bot Preference Sync, per Cloudflare. Source, read 2026-11-04.

How it shows up here

It replaced Managed Robots.txt on 2026-09-15, so a Cloudflare-served robots.txt may differ from the file in your repository.

Previous: Bot paywall. Next: Bot score. All 41 terms.

Sources

Have it both ways: stay discoverable in search while disallowing AI training read 2026-11-04

Last reviewed 2026-11-04