Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

What Content-Signals changes

In short

Content-Signal lines in robots.txt state search, ai-input and ai-train preferences; they express a preference and enforce nothing. The Content Signals Policy defines three signals: search, ai-input and ai-train, written as comma-delimited yes or no values. Example from Cloudflare: Content-Signal: search=yes, ai-train=no under User-Agent: *. Google intends to support the AIPREF training signal written as user-agent: __usage_ai-training in early 2027.

Content-Signal lines add a vocabulary for what a site allows after it has been crawled. They state a preference and enforce nothing.

In practice

  1. Know the three signals: search, ai-input and ai-train, written as comma-delimited yes or no values. Cloudflare's example is Content-Signal: search=yes, ai-train=no under User-Agent: *.
  2. Expect defaults to differ. Cloudflare updated managed robots.txt for over 3.8 million domains to serve search=yes and ai-train=no, and serves no ai-input value for them.
  3. Do not treat the line as a block. Cloudflare says a robots.txt directive cannot identify who is crawling or stop a crawler that ignores it.
  4. Watch the next vocabulary. Google intends to support the AIPREF training signal, written as user-agent: __usage_ai-training, in early 2027.
  5. Add the line with the robots.txt builder if you want it; it sets the three values from your choices.

What the sources say

Sources

Last reviewed 2026-11-04