Websites blocking GPTBot
OpenAI's web crawler for collecting training data for its foundation models, including the GPT series. Announced in August 2023, it quickly became the most widely blocked AI crawler token.
Operated by OpenAI · Training crawler · operator documentation
* block
2.89%
restrict it by name
712K domains name it and close some of its paths
6.30%
name it in robots.txt
1.55 million domains give it rules of their own
Data for Sep 2026. All four percentages
divide by the same 24.63 million domains. Blocked counts both
explicit User-agent: GPTBot blocks and blanket
* blocks the crawler inherits.
Blocking trend
Historical mentions, HTTP Archive
High-authority sites blocking GPTBot by name
| Domain | OPROpen Page Rank | Blocking since |
|---|---|---|
| linkedin.com | 9.82/10 | Aug 2026 |
| vimeo.com | 9.54/10 | Oct 2026 |
| whatsapp.com | 9.52/10 | Oct 2026 |
| cloudfront.net | 9.47/10 | Oct 2026 |
| fbcdn.net | 9.40/10 | Aug 2026 |
| nytimes.com | 9.28/10 | Aug 2026 |
| yahoo.com | 9.27/10 | Oct 2026 |
| line.me | 9.23/10 | Oct 2026 |
| amazon.co.jp | 9.16/10 | Sep 2026 |
| hbr.org | 9.14/10 | Oct 2026 |
| substack.com | 9.13/10 | Oct 2026 |
| cookielaw.org | 9.10/10 | Aug 2026 |
| bandcamp.com | 9.06/10 | Aug 2026 |
| tandfonline.com | 9.05/10 | Aug 2026 |
| lnkd.in | 9.05/10 | Aug 2026 |
| ed.gov | 9.05/10 | Sep 2026 |
| pexels.com | 9.03/10 | Aug 2026 |
| pixabay.com | 9.02/10 | Oct 2026 |
| apnews.com | 9.02/10 | Sep 2026 |
| cbsnews.com | 9.02/10 | Aug 2026 |
| naver.com | 8.99/10 | Oct 2026 |
| amazon.fr | 8.99/10 | Sep 2026 |
| healthline.com | 8.98/10 | Aug 2026 |
| nhk.or.jp | 8.98/10 | Aug 2026 |
| media-amazon.com | 8.95/10 | Aug 2026 |
| science.org | 8.95/10 | Oct 2026 |
| mastodon.social | 8.93/10 | Sep 2026 |
| wa.gov | 8.93/10 | Oct 2026 |
| quora.com | 8.92/10 | Aug 2026 |
| autoriteitpersoonsgegevens.nl | 8.92/10 | Oct 2026 |
Sites whose robots.txt names GPTBot (or a legacy
alias) with a full block, ranked by Open Page Rank. Sites blocking every crawler with a blanket
* rule are counted above but not listed here. Since dates start at our first
observation of the rule.
Restricting some paths
- gravatar.com
- medium.com
- reddit.com
- linktr.ee
- statista.com
- npr.org
- jotform.com
- wsj.com
- xing.com
- squarespace.com
- mckinsey.com
- usatoday.com
Crawler first seen in the wild around Aug 2023. How to block it and what each state means is on the methodology page.