What is Webzio-Extended? AI crawler guide
Webzio-Extended by Webz.io: Webz.io token covering whether crawled content may be included in the datasets it resells for AI and machine learning use. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Webz.io token covering whether crawled content may be included in the datasets it resells for AI and machine learning use.
What is Webzio-Extended?
Webzio-Extended is a web crawler operated by Webz.io. It replaces the older omgili and omgilibot tokens. The crawler collects publicly available web content, and its operator states that the data may be included in datasets resold for AI and machine learning purposes. Webz.io publishes no exact user-agent string, so only the token Webzio-Extended is recorded. The crawler honors robots.txt directives, meaning it respects the Disallow rules set by site owners. Its activity is tied to the commercial resale of training data, making it distinct from search indexing bots or analytics crawlers.
What it's for
If you allow Webzio-Extended to crawl your site, your content could be packaged into datasets that Webz.io sells for AI and machine learning training. This means your text, images, or other public material may end up in models built by third parties. Blocking the crawler signals that you do not want your content used in this way. For site owners who monetize their own data or want to control how their content is used downstream, understanding this crawler's purpose is important.
How to handle Webzio-Extended
To prevent Webzio-Extended from crawling your site, add a Disallow rule for the user-agent token Webzio-Extended in your robots.txt file. Since the crawler honors robots.txt, this will stop it from accessing any disallowed paths. If you later decide to permit crawling, you can remove or adjust the rule.
robots.txt rule
User-agent: Webzio-Extended Disallow: /
Blocking cost
Blocking Webzio-Extended may prevent your content from being included in AI training datasets sold by Webz.io, but it does not affect your visibility in search engines or other web services.
Examples
- A news website adds a robots.txt Disallow rule for Webzio-Extended to keep its articles out of commercial AI training datasets.
- An e-commerce site allows Webzio-Extended to crawl product descriptions, which then appear in a dataset used to train a product recommendation model.
- A blog author notices Webzio-Extended in server logs and blocks it after learning that the crawler feeds data into resold machine learning datasets.
Related bots
- Applebot-Extended: Also tracked as a training crawler.
- ClaudeBot: Also tracked as a training crawler.
- img2dataset: Also tracked as a training crawler.
- Google-Extended: Also tracked as a training crawler.
- VelenPublicWebCrawler: Also tracked as a training crawler.
- LAIONDownloader: Also tracked as a training crawler.
- GPTBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- AI2Bot: Also tracked as a training crawler.
- AI Training Opt-Out: Webzio-Extended is a training crawler tied to this policy decision.
- Robots.txt: Robots.txt is the control file used to allow or block Webzio-Extended.
Frequently Asked Questions
What does Webzio-Extended do?
Webzio-Extended is a crawler from Webz.io that collects web content. The operator states that crawled content may be included in datasets resold for AI and machine learning training.
Does Webzio-Extended follow robots.txt rules?
Yes, Webzio-Extended honors robots.txt directives. If you disallow it, the crawler will not access the specified paths.
How can I block Webzio-Extended?
Add a Disallow rule for the user-agent token Webzio-Extended in your robots.txt file. The crawler respects this standard exclusion method.
Will blocking Webzio-Extended affect my search rankings?
No, blocking Webzio-Extended does not impact your visibility in search engines. It only prevents your content from being included in Webz.io's AI training datasets.
Is Webzio-Extended the same as the old omgili crawler?
Webzio-Extended replaces the older omgili and omgilibot tokens. It serves a similar purpose but under a new identifier.
Data & Sources
- Webz.io documentation - Primary source for Webzio-Extended crawler details.