Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Browsing agent from Apify

Apify Website Content Crawler

In short

Apify Website Content Crawler is Apify's browsing agent. Apify Website Content Crawler has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. On robots.txt, respectRobotsTxtFile is an input option; the Console prefills it, the API does not.

Operator
Apify Vendor profile: Apify
Purpose
Task agent All task agent rows. A browsing agent that clicks, fills forms and finishes a job for a person.
User-agent string
No exact string is published. What the operator or a third party says: Signed with Web Bot Auth when signHttpRequests is enabled
robots.txt token
None. No robots.txt line can address this agent.
Respects robots.txt

Depends on a setting

Apify: respectRobotsTxtFile is an input option; the Console prefills it, the API does not. Source, read 2026-11-04.

Verification

Web Bot Auth signature

Requests carry an HTTP message signature that can be checked against the operator's published key directory. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Set by the Actor run.
Block rate

None: the agent has no token, so a robots.txt block rate does not exist.

Documentation
blog.apify.com, read 2026-11-04.

Note. No key directory at apify.com/.well-known (404, 2026-11-04); verification relies on the signature-agent named in each signed request.

Allow and refuse lines

This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity.

Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF.

Apify in the Verification Ledger

Signing option documented; directory location not found. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
signHttpRequests option; no key directory at apify.com (404, 2026-11-04)
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04