--- title: "Apify: crawler operator profile | Agent Ready Crawlers" description: "Apify: crawler operator profile covering Website Content Crawler and Apify platform and Store. Standards, dated prices and limits, read 2026-11-05." canonical: https://agentreadycrawlers.com/vendors/apify site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Vendors" --- # Apify: crawler operator profile | Agent Ready Crawlers > Apify: crawler operator profile covering Website Content Crawler and Apify platform and Store. Standards, dated prices and limits, read 2026-11-05. Source: https://agentreadycrawlers.com/vendors/apify Site: Agent Ready Crawlers Last reviewed: 2026-11-05 Crawler operator # Apify In short Apify runs a scraping platform where customers' Actors do the crawling. A run signs its traffic with Web Bot Auth only when its input asks for that, and an API caller who keeps the defaults also skips robots.txt. The signing key sits on api.apify.com rather than the main site, and a free tier covers light use. Type [Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) Based in Czech Republic Homepage [apify.com](https://apify.com) In one line Scraping and automation platform whose Actors crawl on behalf of customers; its Website Content Crawler can sign requests with Web Bot Auth from a key directory on api.apify.com, and robots.txt adherence is an input option. Last read 2026-11-05 ## What Apify does for this layer ### From the vendor's own pages Apify's Website Content Crawler has an experimental signHttpRequests input that signs all HTTP requests with the crawler's Web Bot Auth private key, which the input schema says is needed to use the crawler as a Cloudflare Signed Agent. ([apify.com](https://apify.com/apify/website-content-crawler/input-schema)) Website Content Crawler's respectRobotsTxtFile option consults robots.txt before each page, but the input schema says the crawler "does not use any specific user agent identifier" and does not yet support crawl-delay. ([apify.com](https://apify.com/apify/website-content-crawler/input-schema)) Apify's blog says the Apify Console prefills respectRobotsTxtFile while the API does not, and that the measured run in the post used the API default, which is off. ([blog.apify.com](https://blog.apify.com/ai-agent-infrastructure/)) Website Content Crawler also offers a useLlmsTxt input that crawls /llms.txt and Markdown files. ([apify.com](https://apify.com/apify/website-content-crawler/input-schema)) Apify's blog says that since June 2026 an agent can pay for Actor calls through x402 by buying a prepaid USDC token with a wallet, limited to Actors with pay-per-event pricing. ([blog.apify.com](https://blog.apify.com/ai-agent-infrastructure/)) Apify's pricing page lists Free at USD 0 with USD 5 of monthly usage at USD 0.2 per compute unit, Starter at USD 19, Scale at USD 199 (USD 0.16 per compute unit) and Business at USD 999 a month (USD 0.13 per compute unit). ([apify.com](https://apify.com/pricing)) ### Claims the vendor makes about itself Apify says it was launched in 2015 from the Y Combinator Fellowship in Mountain View and moved to the Czech Republic in 2016; its About page counts 110,737 ready-made Actors in Apify Store. ([apify.com](https://apify.com/about)) ### What our crawl saw Apify's key directory at api.apify.com/.well-known/http-message-signatures-directory returned HTTP 200 with content type application/http-message-signatures-directory+json, one Ed25519 key and a "purpose" value of "rag" on 2026-11-05. ([api.apify.com](https://api.apify.com/.well-known/http-message-signatures-directory)) The same path on apify.com returned HTTP 404 on 2026-11-05, so Apify's signing keys live on the api.apify.com host only. ([apify.com](https://apify.com/.well-known/http-message-signatures-directory)) ## Products and plans | Product | What it does | Price model | From | As of | | --- | --- | --- | --- | --- | | Website Content Crawler | Actor that crawls sites into text or Markdown for RAG; optional robots.txt adherence and Web Bot Auth signing. | platform usage Per compute unit on Free and Starter; lower per-unit rates on Scale and Business. | USD 0.2 | 2026-11-05 | | Apify platform and Store | Runs Actors (scrapers and agents) by API, MCP or schedule. | subscription plus usage Free USD 0 with USD 5 usage; Starter USD 19, Scale USD 199, Business USD 999 a month. | USD 0 | 2026-11-05 | ## Standards | Standard | Status | Evidence | | --- | --- | --- | | RFC 9309 (robots.txt) | Partial | respectRobotsTxtFile is an option, off by API default, with no own token and no crawl-delay support. (apify.com) | | Web Bot Auth | Partial | Experimental signHttpRequests input on Website Content Crawler; key directory served on api.apify.com. (apify.com) | | Published IP list | No statement | No egress IP list for Actor traffic found (2026-11-05). | | Cloudflare verified bots | Partial | The input schema says signing is required to run as a Cloudflare Signed Agent. (apify.com) | ## Our checks | Check | Result | Method | Date | | --- | --- | --- | --- | | Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 1 Ed25519 key, purpose rag | GET https://api.apify.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | | Web Bot Auth key directory | HTTP 404 | GET https://apify.com/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 | ## What Apify does not do - Apify's Website Content Crawler names no robots.txt token of its own, so a site can address it only through the \* group (input schema read 2026-11-05). - Apify does not sign requests unless the run turns signHttpRequests on, and robots.txt is ignored when the API caller leaves respectRobotsTxtFile at its default. ## Reading notes The Crawler Index row for the [Website Content Crawler](https://agentreadycrawlers.com/crawlers/apify/) recorded a 404 for the key directory at apify.com on 4 November 2026; the directory is served from api.apify.com, as the checks above show. ## Related pages on this site - [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles - [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types - [Apify Website Content Crawler](https://agentreadycrawlers.com/crawlers/apify/) no token ## Sources - [apify.com](https://apify.com) read 2026-11-05 - [Input · Website Content Crawler · Apify](https://apify.com/apify/website-content-crawler/input-schema) read 2026-11-05 - [Apify key directory (api.apify.com)](https://api.apify.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [Apify key directory probe (apify.com)](https://apify.com/.well-known/http-message-signatures-directory) read 2026-11-05 - [AI agent infrastructure stack: from web data to retrieval | Apify Blog](https://blog.apify.com/ai-agent-infrastructure/) read 2026-11-05 - [Apify pricing - flexible plan + pay as you go · Apify](https://apify.com/pricing) read 2026-11-05 - [About · Apify](https://apify.com/about) read 2026-11-05 Last reviewed 2026-11-05