Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Browsing agent from Amazon Web Services

Amazon Bedrock AgentCore Browser

In short

Amazon Bedrock AgentCore Browser is Amazon Web Services' browsing agent. Amazon Bedrock AgentCore Browser has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. Amazon Bedrock AgentCore Browser: site owners keep control and may block, monitor or rate-limit agent traffic regardless of signatures; AWS states no robots.txt behaviour.

Operator
Amazon Web Services
Purpose
Task agent All task agent rows. A browsing agent that clicks, fills forms and finishes a job for a person.
User-agent string
No exact string is published. What the operator or a third party says: Signed with Web Bot Auth when enabled
robots.txt token
None. No robots.txt line can address this agent.
Respects robots.txt

No statement

AWS: site owners keep control and may block, monitor or rate-limit agent traffic regardless of signatures; AWS states no robots.txt behaviour. Source, read 2026-11-04.

Verification

Web Bot Auth signature

Requests carry an HTTP message signature that can be checked against the operator's published key directory. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Set by the developer running the agent.
Block rate

None: the agent has no token, so a robots.txt block rate does not exist.

Documentation
docs.aws.amazon.com, read 2026-11-04.

Note. Web Bot Auth support is in preview and off by default (browserSigning setting), per AWS docs and WorkOS.

Allow and refuse lines

This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity.

Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF.

Amazon Web Services in the Verification Ledger

Signs on behalf of customers' agents when enabled. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
Web Bot Auth signing in preview, opt-in (browserSigning)
Cloudflare
listed by AWS as recognised by Cloudflare, Akamai and others
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04