Crawler operator
Apify
In short
Apify runs a scraping platform where customers' Actors do the crawling. A run signs its traffic with Web Bot Auth only when its input asks for that, and an API caller who keeps the defaults also skips robots.txt. The signing key sits on api.apify.com rather than the main site, and a free tier covers light use.
- Type
- Crawler operator
- Based in
- Czech Republic
- Homepage
- apify.com
- In one line
- Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option.
- Last read
- 2026-11-05
What Apify does for this layer
From the vendor's own pages
Apify's Website Content Crawler has an experimental signHttpRequests input that signs all HTTP requests with the crawler's Web Bot Auth private key, which the input schema says is needed to use the crawler as a Cloudflare Signed Agent. (apify.com) Website Content Crawler's respectRobotsTxtFile option consults robots.txt before each page, but the input schema says the crawler "does not use any specific user agent identifier" and does not yet support crawl-delay. (apify.com) Apify's blog says the Apify Console prefills respectRobotsTxtFile while the API does not, and that the measured run in the post used the API default, which is off. (blog.apify.com) Website Content Crawler also offers a useLlmsTxt input that crawls /llms.txt and Markdown files. (apify.com)
Apify's blog says that since June 2026 an agent can pay for Actor calls through x402 by buying a prepaid USDC token with a wallet, limited to Actors with pay-per-event pricing. (blog.apify.com) Apify's pricing page lists Free at USD 0 with USD 5 of monthly usage at USD 0.2 per compute unit, Starter at USD 19, Scale at USD 199 (USD 0.16 per compute unit) and Business at USD 999 a month (USD 0.13 per compute unit). (apify.com)
Claims the vendor makes about itself
Apify says it was launched in 2015 from the Y Combinator Fellowship in Mountain View and moved to the Czech Republic in 2016; its About page counts 110,737 ready-made Actors in Apify Store. (apify.com)
What our crawl saw
Apify's key directory at api.apify.com/.well-known/http-message-signatures-directory returned HTTP 200 with content type application/http-message-signatures-directory+json, one Ed25519 key and a "purpose" value of "rag" on 2026-11-05. (api.apify.com) The same path on apify.com returned HTTP 404 on 2026-11-05, so Apify's signing keys live on the api.apify.com host only. (apify.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Website Content Crawler | Actor that crawls sites into text or Markdown for RAG; optional robots.txt adherence and Web Bot Auth signing. | platform usage | USD 0.2 | 2026-11-05 |
| Apify platform and Store | Runs Actors (scrapers and agents) by API, MCP or schedule. | subscription plus usage | USD 0 | 2026-11-05 |
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Partial | respectRobotsTxtFile is an option, off by API default, with no own token and no crawl-delay support. (apify.com) |
| Web Bot Auth | Partial | Experimental signHttpRequests input on Website Content Crawler; key directory served on api.apify.com. (apify.com) |
| Published IP list | No statement | No egress IP list for Actor traffic found (2026-11-05). |
| Cloudflare verified bots | Partial | The input schema says signing is required to run as a Cloudflare Signed Agent. (apify.com) |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key, purpose rag | GET https://api.apify.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 404 | GET https://apify.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
What Apify does not do
- Apify's Website Content Crawler names no robots.txt token of its own, so a site can address it only through the * group (input schema read 2026-11-05).
- Apify does not sign requests unless the run turns signHttpRequests on, and robots.txt is ignored when the API caller leaves respectRobotsTxtFile at its default.
Reading notes
The Crawler Index row for the Website Content Crawler recorded a 404 for the key directory at apify.com on 4 November 2026; the directory is served from api.apify.com, as the checks above show.
Related pages on this site
Sources
Last reviewed 2026-11-05