Websites blocking Webzio-Extended
Webz.io's opt-out token for AI training uses of its crawled data. Webz.io sells web datasets to AI companies.
Operated by Webz.io · Training crawler
4.50%
block Webzio-Extended
1.11 million of 24.68 million domains with a known robots.txt policy
1.10%
block it by name
271K domains, the rest inherit a blanket
* block
2.26%
restrict it by name
559K domains name it and close some of its paths
3.39%
name it in robots.txt
836K domains give it rules of their own
Data for Oct 2026. All four percentages
divide by the same 24.68 million domains. Blocked counts both
explicit User-agent: Webzio-Extended blocks and blanket
* blocks the crawler inherits.
Blocking trend
2%
4%
6%
Aug 2026
Sep 2026
Oct 2026
Share of domains with a known robots.txt policy
blocking Webzio-Extended in each monthly crawl. The denominator moves with each crawl and is
defined in the methodology.
Historical mentions, HTTP Archive
1%
2%
3%
4%
Jul 2021
Oct 2022
Feb 2024
Jun 2025
Sep 2026
Share of sites in each month's
HTTP Archive crawl whose robots.txt named
Webzio-Extended (or a legacy alias of it). This series counts
mentions only (the archive keeps no rule text), comes from a different population than our own
crawl above, and the two are never merged.
High-authority sites blocking Webzio-Extended by name
| Domain | OPROpen Page Rank | Blocking since |
|---|---|---|
| cnn.com | 9.16/10 | Oct 2026 |
| amazon.co.jp | 9.16/10 | Sep 2026 |
| amazon.de | 9.13/10 | Oct 2026 |
| usatoday.com | 9.08/10 | Sep 2026 |
| wired.com | 9.07/10 | Oct 2026 |
| fastcompany.com | 9.00/10 | Oct 2026 |
| upenn.edu | 9.00/10 | Oct 2026 |
| healthline.com | 8.98/10 | Aug 2026 |
| uol.com.br | 8.92/10 | Oct 2026 |
| amazon.ca | 8.91/10 | Oct 2026 |
| newyorker.com | 8.90/10 | Oct 2026 |
| airbnb.com | 8.88/10 | Sep 2026 |
| mashable.com | 8.87/10 | Sep 2026 |
| francetvinfo.fr | 8.87/10 | Sep 2026 |
| proofpoint.com | 8.85/10 | Sep 2026 |
| arstechnica.com | 8.84/10 | Aug 2026 |
| medicalnewstoday.com | 8.83/10 | Aug 2026 |
| unc.edu | 8.83/10 | Oct 2026 |
| gizmodo.com | 8.82/10 | Aug 2026 |
| wikihow.com | 8.81/10 | Aug 2026 |
| pcmag.com | 8.80/10 | Aug 2026 |
| nikkei.com | 8.79/10 | Oct 2026 |
| welt.de | 8.78/10 | Oct 2026 |
| fliphtml5.com | 8.75/10 | Oct 2026 |
| iop.org | 8.75/10 | Oct 2026 |
| verywellmind.com | 8.72/10 | Aug 2026 |
| metro.co.uk | 8.72/10 | Aug 2026 |
| bostonglobe.com | 8.71/10 | Oct 2026 |
| people.com | 8.70/10 | Aug 2026 |
| bild.de | 8.70/10 | Oct 2026 |
Sites whose robots.txt names Webzio-Extended (or a legacy
alias) with a full block, ranked by Open Page Rank. Sites blocking every crawler with a blanket
* rule are counted above but not listed here. Since dates start at our first
observation of the rule.
Restricting some paths
- statista.com
- npr.org
- hbr.org
- vox.com
- nzz.ch
- ccm19.de
- lifehacker.com
- ign.com
- everydayhealth.com
- eater.com
- foreignpolicy.com
- curbed.com
Also matched via
the legacy token omgilibot,
which some sites still use for this crawler.