Websites blocking meta-externalagent
Meta's crawler for collecting web data used to train its AI models, including the Llama family, and to improve Meta products.
Operated by Meta · Training crawler · operator documentation
* block
2.62%
restrict it by name
646K domains name it and close some of its paths
4.97%
name it in robots.txt
1.23 million domains give it rules of their own
Data for Oct 2026. All four percentages
divide by the same 24.68 million domains. Blocked counts both
explicit User-agent: meta-externalagent blocks and blanket
* blocks the crawler inherits.
Blocking trend
Historical mentions, HTTP Archive
High-authority sites blocking meta-externalagent by name
| Domain | OPROpen Page Rank | Blocking since |
|---|---|---|
| linkedin.com | 9.82/10 | Aug 2026 |
| nytimes.com | 9.28/10 | Aug 2026 |
| tumblr.com | 9.27/10 | Aug 2026 |
| cnn.com | 9.16/10 | Oct 2026 |
| amazon.co.jp | 9.16/10 | Sep 2026 |
| substack.com | 9.13/10 | Oct 2026 |
| amazon.de | 9.13/10 | Oct 2026 |
| cookielaw.org | 9.10/10 | Aug 2026 |
| usatoday.com | 9.08/10 | Sep 2026 |
| mhlw.go.jp | 9.08/10 | Sep 2026 |
| giphy.com | 9.07/10 | Oct 2026 |
| wired.com | 9.07/10 | Oct 2026 |
| bandcamp.com | 9.06/10 | Aug 2026 |
| lnkd.in | 9.05/10 | Aug 2026 |
| naver.com | 8.99/10 | Oct 2026 |
| healthline.com | 8.98/10 | Aug 2026 |
| ebay.com | 8.95/10 | Oct 2026 |
| indiatimes.com | 8.95/10 | Aug 2026 |
| coursera.org | 8.94/10 | Oct 2026 |
| uol.com.br | 8.92/10 | Oct 2026 |
| si.edu | 8.92/10 | Oct 2026 |
| huffpost.com | 8.91/10 | Aug 2026 |
| amazon.ca | 8.91/10 | Oct 2026 |
| newyorker.com | 8.90/10 | Oct 2026 |
| entrepreneur.com | 8.89/10 | Oct 2026 |
| heise.de | 8.88/10 | Sep 2026 |
| mashable.com | 8.87/10 | Sep 2026 |
| francetvinfo.fr | 8.87/10 | Aug 2026 |
| smh.com.au | 8.86/10 | Sep 2026 |
| proofpoint.com | 8.85/10 | Sep 2026 |
Sites whose robots.txt names meta-externalagent (or a legacy
alias) with a full block, ranked by Open Page Rank. Sites blocking every crawler with a blanket
* rule are counted above but not listed here. Since dates start at our first
observation of the rule.
Restricting some paths
- medium.com
- reddit.com
- www.gov.uk
- statista.com
- npr.org
- hbr.org
- jotform.com
- squarespace.com
- go.com
- tripadvisor.com
- nbcnews.com
- fastcompany.com
Crawler first seen in the wild around Aug 2024. How to block it and what each state means is on the methodology page.