Websites blocking Google-Extended
A robots.txt control token rather than a separate crawler. Disallowing Google-Extended tells Google not to use a site's content for Gemini model training and grounding, without affecting Google Search indexing.
Operated by Google · Training crawler · operator documentation
* block
2.85%
restrict it by name
703K domains name it and close some of its paths
5.77%
name it in robots.txt
1.42 million domains give it rules of their own
Data for Oct 2026. All four percentages
divide by the same 24.68 million domains. Blocked counts both
explicit User-agent: Google-Extended blocks and blanket
* blocks the crawler inherits.
Blocking trend
Historical mentions, HTTP Archive
High-authority sites blocking Google-Extended by name
| Domain | OPROpen Page Rank | Blocking since |
|---|---|---|
| linkedin.com | 9.82/10 | Aug 2026 |
| whatsapp.com | 9.52/10 | Oct 2026 |
| cloudfront.net | 9.47/10 | Oct 2026 |
| fbcdn.net | 9.40/10 | Aug 2026 |
| nytimes.com | 9.28/10 | Aug 2026 |
| tumblr.com | 9.27/10 | Aug 2026 |
| yahoo.com | 9.27/10 | Oct 2026 |
| amazon.co.jp | 9.16/10 | Sep 2026 |
| substack.com | 9.13/10 | Oct 2026 |
| amazon.de | 9.13/10 | Oct 2026 |
| cookielaw.org | 9.10/10 | Aug 2026 |
| giphy.com | 9.07/10 | Oct 2026 |
| sagepub.com | 9.07/10 | Oct 2026 |
| wired.com | 9.07/10 | Oct 2026 |
| bandcamp.com | 9.06/10 | Aug 2026 |
| lnkd.in | 9.05/10 | Aug 2026 |
| cbc.ca | 9.03/10 | Oct 2026 |
| pixabay.com | 9.02/10 | Oct 2026 |
| naver.com | 8.99/10 | Oct 2026 |
| academia.edu | 8.99/10 | Sep 2026 |
| healthline.com | 8.98/10 | Oct 2026 |
| nhk.or.jp | 8.98/10 | Aug 2026 |
| science.org | 8.95/10 | Oct 2026 |
| elsevier.com | 8.93/10 | Oct 2026 |
| si.edu | 8.92/10 | Oct 2026 |
| huffpost.com | 8.91/10 | Aug 2026 |
| amazon.ca | 8.91/10 | Oct 2026 |
| newyorker.com | 8.90/10 | Oct 2026 |
| ssrn.com | 8.88/10 | Oct 2026 |
| claude.ai | 8.87/10 | Sep 2026 |
Sites whose robots.txt names Google-Extended (or a legacy
alias) with a full block, ranked by Open Page Rank. Sites blocking every crawler with a blanket
* rule are counted above but not listed here. Since dates start at our first
observation of the rule.
Restricting some paths
- gravatar.com
- vimeo.com
- reddit.com
- statista.com
- npr.org
- hbr.org
- squarespace.com
- usatoday.com
- time.com
- go.com
- tripadvisor.com
- bbb.org
Crawler first seen in the wild around Sep 2023. How to block it and what each state means is on the methodology page.