Websites blocking AI2Bot
The Allen Institute for AI's crawler, used to build open datasets such as Dolma that train its open language models.
Operated by Allen Institute for AI · Training crawler · operator documentation
3.76%
block AI2Bot
938K of 24.91 million domains with a known robots.txt policy
0.27%
block it by name
67,573 domains, the rest inherit a blanket
* block
2.30%
restrict it by name
574K domains name it and close some of its paths
2.65%
name it in robots.txt
660K domains give it rules of their own
Data for Aug 2026. All four percentages
divide by the same 24.91 million domains. Blocked counts both
explicit User-agent: AI2Bot blocks and blanket
* blocks the crawler inherits.
Historical mentions, HTTP Archive
1%
2%
3%
Sep 2024
Mar 2025
Aug 2025
Feb 2026
Jul 2026
Share of sites in each month's
HTTP Archive crawl whose robots.txt named
AI2Bot (or a legacy alias of it). This series counts
mentions only (the archive keeps no rule text), comes from a different population than our own
crawl above, and the two are never merged.
High-authority sites blocking AI2Bot by name
| Domain | OPROpen Page Rank | Blocking since |
|---|---|---|
| usatoday.com | 9.08/10 | Aug 2026 |
| congress.gov | 9.05/10 | Aug 2026 |
| msn.com | 9.04/10 | Aug 2026 |
| mashable.com | 8.87/10 | Aug 2026 |
| thelancet.com | 8.85/10 | Aug 2026 |
| gizmodo.com | 8.82/10 | Aug 2026 |
| wikihow.com | 8.81/10 | Aug 2026 |
| pcmag.com | 8.80/10 | Aug 2026 |
| slate.com | 8.80/10 | Aug 2026 |
| techradar.com | 8.78/10 | Aug 2026 |
| amazon.it | 8.78/10 | Aug 2026 |
| adweek.com | 8.77/10 | Aug 2026 |
| oreilly.com | 8.76/10 | Aug 2026 |
| fliphtml5.com | 8.75/10 | Aug 2026 |
| rfi.fr | 8.73/10 | Aug 2026 |
| metro.co.uk | 8.72/10 | Aug 2026 |
| tagesspiegel.de | 8.71/10 | Aug 2026 |
| mainichi.jp | 8.69/10 | Aug 2026 |
| france24.com | 8.68/10 | Aug 2026 |
| cell.com | 8.68/10 | Aug 2026 |
| nos.nl | 8.67/10 | Aug 2026 |
| livescience.com | 8.66/10 | Aug 2026 |
| launchpad.net | 8.63/10 | Aug 2026 |
| computerworld.com | 8.63/10 | Aug 2026 |
| vermont.gov | 8.63/10 | Aug 2026 |
| fedoraproject.org | 8.62/10 | Aug 2026 |
| nintendo.com | 8.62/10 | Aug 2026 |
| francebleu.fr | 8.61/10 | Aug 2026 |
| svt.se | 8.59/10 | Aug 2026 |
| allocine.fr | 8.58/10 | Aug 2026 |
Sites whose robots.txt names AI2Bot (or a legacy
alias) with a full block, ranked by Open Page Rank. Sites blocking every crawler with a blanket
* rule are counted above but not listed here. Since dates start at our first
observation of the rule.
Restricting some paths
- statista.com
- hbr.org
- squarespace.com
- frontiersin.org
- theverge.com
- vox.com
- newsweek.com
- buzzfeed.com
- youradchoices.ca
- ccm19.de
- ign.com
- datareportal.com
Also matched via
the legacy token ai2bot-dolma,
which some sites still use for this crawler.