Crawler operator
Webz.io
In short
Webz.io, headquartered in Israel, resells crawled web data through APIs for news, blogs, forums and reviews, with separate dark web and breach feeds quoted on request. Webzio-extended exists only to label each collected item as usable or not for AI training, and the older omgili bot was succeeded in July 2024. Custom tiers reach back ten years of archive.
- Type
- Crawler operator
- Based in
- Israel
- Homepage
- webz.io
- In one line
- Web data vendor that crawls news, blogs, forums and reviews under the Webzio and Webzio-extended tokens (successors to the omgili bot) and sells the results as APIs, tagging which data may be used for AI training.
- Last read
- 2026-11-05
What Webz.io does for this layer
From the vendor's own pages
Webz.io introduced two crawlers on 2024-07-17, Webzio and Webzio-extended, as the successors to its Omgilibot, and its webz.io/bot.html address now redirects to that announcement. (webz.io, dated 2024-07-17) Webz.io says the Webzio crawler collects and structures content, while Webzio-extended tags the collected data as usable or not usable for AI/ML training. (webz.io, dated 2024-07-17) Webz.io's 2017-12-28 post gives the Omgili bot's user agent as "omgili/0.5 +https://omgili.com" and says the bot was first built for the now discontinued Omgili search engine. (webz.io, dated 2017-12-28) The Omgili bot post lists news sites, blogs, reviews, ecommerce and message boards as the bot's focus and says site owners can refuse it through robots.txt or by telling Webz.io directly. (webz.io, dated 2017-12-28)
Webz.io's pricing page lists a Free plan at USD 0 a month with USD 5 of usage credits every month, a pay-as-you-go Scale plan, and custom High Volume pricing with up to 10 years of archive data. (webz.io) Webz.io's About page states that the company is headquartered in Israel. (webz.io)
Claims the vendor makes about itself
Webz.io says the two newer crawlers adhere to robots.txt exclusions and that it only provides data tagged as permitted for AI purposes, a rule it says is reflected in its Terms of Service. (webz.io, dated 2024-07-17) Webz.io sells News, Blogs, Online Forums, Reviews, Dark Web and Breach APIs, and says its News API covers 350K+ sources in 170+ languages. (webz.io)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Open Web APIs (News, Blogs, Forums, Reviews, Search) | Structured feeds built from Webz.io's crawl of the open web. | pay as you go | USD 0 | 2026-11-05 |
| Dark Web API and Breach API | Data from deep and dark web sources; contact for pricing. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Webz.io says Webzio and Webzio-extended adhere to robots.txt exclusions. (webz.io) |
| Web Bot Auth | No statement | No key directory at webz.io (HTTP 404, 2026-11-05). |
| Published IP list | No statement | No IP list found on the pages read (2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 404, text/html | GET https://webz.io/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
What Webz.io does not do
- Webz.io publishes no IP list, reverse DNS pattern or signature keys for Webzio, Webzio-extended or omgili (checked 2026-11-05).
- Webz.io's 2024 announcement names the two new tokens but gives no full user agent strings for them.
Related pages on this site
Sources
Last reviewed 2026-11-05