Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Content-Signal

In short

A robots.txt line from the Content Signals Policy that states yes or no for three uses: search, ai-input and ai-train. 6.4 percent of top-10,000 robots.txt files (380 of 5,938) carry a Content-Signal line. Example: Cloudflare's example: "Content-Signal: search=yes, ai-train=no". Related: What Content-Signals changes and robots.txt builder.

Definition

A robots.txt line from the Content Signals Policy that states yes or no for three uses: search, ai-input and ai-train.

Sourced example

Cloudflare's example: "Content-Signal: search=yes, ai-train=no". Source, read 2026-11-04.

How it shows up here

6.4 percent of top-10,000 robots.txt files (380 of 5,938) carry a Content-Signal line.

Previous: Browsing agent. Next: Crawl-delay. All 41 terms.

Sources

Last reviewed 2026-11-04