Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Training crawler from Amazon

Amazonbot

In short

Amazonbot is Amazon's training crawler. The token to write in robots.txt is Amazonbot. Check a visitor against the operator's published IP list. 15.6 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Amazonbot: automated crawling respects the Robots Exclusion Protocol, honouring user-agent and allow/disallow directives; robots.txt may be cached for 30 days.

Operator
Amazon Vendor profile: Amazon
Purpose
Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
robots.txt token
Amazonbot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Amazon: automated crawling respects the Robots Exclusion Protocol, honouring user-agent and allow/disallow directives; robots.txt may be cached for 30 days. Source, read 2026-11-04.

Verification

Published IP list (JSON)

Match the source address of a request against the operator's published list. Documentation, read 2026-11-04.

IP ranges
developer.amazon.com/amazonbot/ip-addresses/
Rate behaviour
Does not support Crawl-delay. Honours rel=nofollow and robots meta noarchive, noindex and none.
Block rate

15.6% of 5,938 top-10,000 sites with a parsed robots.txt disallow Amazonbot at the root (926 sites). 12.0% name it; 3.6% refuse it only through a * group. Rank 9 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 11.7% (9.1% named). Crawl of 2026-11-04; report.

Documentation
developer.amazon.com, read 2026-11-04.

Note. Amazon says Amazonbot improves products and services and may be used to train Amazon AI models. Changes take about 24 hours.

Allow and refuse lines

To allow Amazonbot

User-agent: Amazonbot
Allow: /

To refuse Amazonbot

User-agent: Amazonbot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Amazon in the Verification Ledger

Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus. All operators.

IP lists
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Other Amazon rows

Sources

Last reviewed 2026-11-04