Crawler operator
Parallel Web Systems
In short
Parallel Web Systems feeds its search and extraction APIs from two signed identities, documented from 24 October 2026. ShapBot builds the index and has a robots.txt token and an IP list; Shap-User fetches pages at a user's direction and offers no robots.txt control. Search requests are priced from USD 0.001.
- Type
- Crawler operator
- Homepage
- parallel.ai
- In one line
- Web search and research API company for AI agents that crawls as ShapBot and fetches for users as Shap-User, both signed with Web Bot Auth, with an IP list for ShapBot and a robots.txt token for the crawler only.
- Last read
- 2026-11-05
What Parallel Web Systems does for this layer
From the vendor's own pages
Parallel Web Systems documents two identities on a bots page effective 2026-10-24: ShapBot, which builds Parallel's search index, and Shap-User, which fetches pages at users' direction. (parallel.ai, dated 2026-10-24) ShapBot's full user agent is "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0", and Parallel tells site owners to manage it with a User-agent: ShapBot group in robots.txt. (parallel.ai) Shap-User sends a user agent ending "compatible; Shap-User/0.1.0", and Parallel says Shap-User "is intended to provide visibility rather than to be managed through automated controls". (parallel.ai) Parallel signs both ShapBot and Shap-User requests with Web Bot Auth and publishes keys at shapbot.parallel.ai and shap-user.parallel.ai under /.well-known/http-message-signatures-directory. (parallel.ai)
Parallel warns that restricting ShapBot "prevents our system from indexing your content", which it says may reduce a site's discoverability through its web APIs and search features. (parallel.ai) Parallel invites content owners with questions about its index to explore joining "Index by Parallel" through a dedicated contact address. (parallel.ai) Parallel's pricing page allows up to 5,000 requests a month free and prices the Search API at USD 0.001 to USD 0.005 for 10 results and the Extract API at USD 0.001 per request. (parallel.ai)
What our crawl saw
Parallel's ShapBot IP list at docs.parallel.ai/resources/shapbot.json had a creationTime of 2026-07-14 and 10 single-address IPv4 prefixes when read on 2026-11-05. (docs.parallel.ai) The ShapBot and Shap-User key directories both returned HTTP 200 with content type application/http-message-signatures-directory+json and two Ed25519 keys each on 2026-11-05. (shapbot.parallel.ai)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Parallel Search, Extract and Task APIs | Web search, page extraction and research tasks for agents, built on the ShapBot index. | pay per request | USD 0.001 | 2026-11-05 |
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | ShapBot is managed with a User-agent: ShapBot robots.txt group. (parallel.ai) |
| Web Bot Auth | Supports | ShapBot and Shap-User sign requests; key directories on two hosts. (parallel.ai) |
| Published IP list | Partial | IP list published for ShapBot only, not for Shap-User. (docs.parallel.ai) |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 2 Ed25519 keys | GET https://shapbot.parallel.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 200, application/http-message-signatures-directory+json, 2 Ed25519 keys | GET https://shap-user.parallel.ai/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
| ShapBot IP list | creationTime 2026-07-14, 10 IPv4 /32 prefixes | GET https://docs.parallel.ai/resources/shapbot.json with the crawl user agent | 2026-11-05 |
What Parallel Web Systems does not do
- Parallel gives Shap-User no robots.txt control; its bots page says user-initiated fetches are for visibility, not automated management.
Reading notes
ShapBot and Shap-User from Parallel are not in the Crawler Index yet.
Related pages on this site
Sources
Last reviewed 2026-11-05