Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

noarchive

In short

A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as "do not use the page for model training". It matters for Bing, which has no robots.txt training token yet; the guide on training opt-outs covers the gap. Example: Amazon says its bots honour noarchive as do not use the page for model.

Definition

A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as "do not use the page for model training".

Sourced example

Amazon says its bots honour noarchive as do not use the page for model training. Source, read 2026-11-04.

How it shows up here

It matters for Bing, which has no robots.txt training token yet; the guide on training opt-outs covers the gap.

Previous: Mixed-use crawler. Next: nosnippet. All 41 terms.

Sources

Last reviewed 2026-11-04