Who blocks AI crawlers
How websites respond to AI crawlers in robots.txt, measured by our own monthly crawl of 44.73 million domains. Data for Sep 2026.
*
1.15 million of 20.47 million with robots.txt
Percentages divide by the 24.63 million domains whose robots.txt policy we could determine this month, except where a tile names a different base. Full definitions in the methodology.
Emerging licensing standards
Two newer robots.txt directives let a site state terms rather than just allow or block:
RSL (Really Simple Licensing) adds a
License: line pointing at machine-readable licence terms, and the IETF AIPREF draft adds
Content-Usage: preferences. We record both verbatim on every robots.txt we read and count
domains that publish at least one.
License: line
1,517 of 20.47 million domains with a robots.txt
<0.01%
publish an AIPREF Content-Usage: line
805 of 20.47 million domains with a robots.txt
Every AI crawler we track
Blocked means the domain's robots.txt shuts the crawler out entirely, either by naming it
or via a blanket * rule. Restricted by name means the file names the crawler and
closes some of its paths. Named means the crawler appears in the file with its own rules,
whatever they say.
Training crawlers
| Crawler | Operator | Blocked | Restricted by name | NamedNamed in robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | 6.04% (1.49M) | 712K | 1.55M |
| CCBot | Common Crawl | 6.03% (1.48M) | 639K | 1.35M |
| Bytespider | ByteDance | 5.94% (1.46M) | 608K | 1.25M |
| ClaudeBot | Anthropic | 5.76% (1.42M) | 691K | 1.47M |
| Google-Extended | 5.51% (1.36M) | 678K | 1.35M | |
| Applebot-Extended | Apple | 5.41% (1.33M) | 646K | 1.2M |
| meta-externalagent | Meta | 5.23% (1.29M) | 641K | 1.18M |
| Webzio-Extended | Webz.io | 4.67% (1.15M) | 578K | 846K |
| cohere-training-data-crawler | Cohere | 4.64% (1.14M) | 598K | 922K |
| ImagesiftBot | Hive | 4.53% (1.12M) | 24,761 | 202K |
| Diffbot | Diffbot | 4.50% (1.11M) | 31,526 | 188K |
| FacebookBot | Meta | 4.47% (1.1M) | 596K | 805K |
| AI2Bot | Allen Institute for AI | 4.30% (1.06M) | 576K | 719K |
| img2dataset | Open source (rom1504) | 4.19% (1.03M) | 575K | 674K |
| GoogleOther | 4.19% (1.03M) | 588K | 722K | |
| PanguBot | Huawei | 4.16% (1.02M) | 2,146 | 35,480 |
| Google-CloudVertexBot | 4.11% (1.01M) | 6,175 | 31,267 | |
| SBIntuitionsBot | SB Intuitions | 4.10% (1.01M) | 1,025 | 20,930 |
Search and AI assistants
| Crawler | Operator | Blocked | Restricted by name | NamedNamed in robots.txt |
|---|---|---|---|---|
| PetalBot | Huawei | 9.12% (2.25M) | 39,351 | 1.33M |
| Amazonbot | Amazon | 5.83% (1.44M) | 625K | 1.25M |
| PerplexityBot | Perplexity | 4.46% (1.1M) | 138K | 537K |
| YouBot | You.com | 4.46% (1.1M) | 588K | 827K |
| DuckAssistBot | DuckDuckGo | 4.18% (1.03M) | 573K | 705K |
| Applebot | Apple | 4.14% (1.02M) | 75,304 | 199K |
| OAI-SearchBot | OpenAI | 4.13% (1.02M) | 119K | 341K |
| Claude-SearchBot | Anthropic | 4.08% (1M) | 60,465 | 149K |
| Kagibot | Kagi | 4.03% (992K) | 1,639 | 2,878 |
User-request fetchers
| Crawler | Operator | Blocked | Restricted by name | NamedNamed in robots.txt |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | 4.51% (1.11M) | 137K | 522K |
| meta-externalfetcher | Meta | 4.13% (1.02M) | 34,711 | 96,844 |
| GoogleAgent-Mariner | 4.09% (1.01M) | 837 | 17,933 | |
| Perplexity-User | Perplexity | 4.09% (1.01M) | 61,501 | 156K |
| MistralAI-User | Mistral AI | 4.08% (1M) | 15,071 | 42,677 |
| Claude-User | Anthropic | 4.06% (1M) | 68,574 | 159K |
Blocking by segment
The same measures cut four ways. High-authority and high-traffic sites block AI crawlers far more often than the long tail, and platform defaults decide the numbers for millions of sites at once.
By authority (Open Page Rank)
| Segment | Robots.txt known | GPTBot | ClaudeBot | Name any AI crawler | Block allBlock all crawlers |
|---|---|---|---|---|---|
| OPR 9.0 and up | 334 | 13.17% | 13.17% | 19.76% | 8.47% |
| OPR 8.0 to 8.9 | 6,760 | 15.59% | 14.72% | 18.79% | 8.69% |
| OPR 7.0 to 7.9 | 34,168 | 13.21% | 12.04% | 15.68% | 6.75% |
| OPR 6.0 to 6.9 | 57,118 | 11.01% | 10.17% | 14.27% | 4.99% |
| OPR 5.0 to 5.9 | 115K | 11.05% | 10.43% | 14.15% | 6.08% |
By traffic rank (Chrome UX Report)
| Segment | Robots.txt known | GPTBot | ClaudeBot | Name any AI crawler | Block allBlock all crawlers |
|---|---|---|---|---|---|
| Top 1K | 610 | 16.23% | 14.75% | 21.80% | 6.91% |
| Top 10K | 5,568 | 16.47% | 15.79% | 20.19% | 6.69% |
| Top 100K | 53,147 | 13.79% | 13.30% | 16.76% | 6.33% |
| Top 1M | 532K | 9.94% | 9.63% | 12.98% | 4.61% |
| Top 10M | 5.43M | 4.98% | 4.69% | 11.43% | 2.06% |
By primary audience country
| Segment | Robots.txt known | GPTBot | ClaudeBot | Name any AI crawler | Block allBlock all crawlers |
|---|---|---|---|---|---|
| United States | 1.51M | 4.26% | 3.89% | 23.21% | 1.61% |
| Japan | 711K | 1.70% | 1.28% | 5.86% | 0.63% |
| India | 542K | 5.00% | 4.84% | 10.09% | 2.13% |
| Brazil | 538K | 7.99% | 7.85% | 18.48% | 5.58% |
| Germany | 517K | 3.20% | 2.56% | 8.65% | 1.23% |
| United Kingdom | 385K | 4.70% | 4.30% | 18.70% | 1.26% |
| France | 384K | 3.12% | 2.96% | 11.45% | 1.14% |
| Russia | 302K | 3.78% | 3.38% | 4.13% | 2.38% |
| Canada | 261K | 4.98% | 4.45% | 22.04% | 1.51% |
| Indonesia | 254K | 14.96% | 14.46% | 15.39% | 1.92% |
| Spain | 244K | 4.81% | 4.71% | 9.41% | 2.46% |
| Poland | 221K | 4.36% | 4.17% | 7.77% | 1.03% |
| Australia | 214K | 4.93% | 4.55% | 18.57% | 1.34% |
| Italy | 204K | 3.78% | 3.64% | 10.58% | 1.52% |
| Netherlands | 176K | 3.75% | 3.44% | 9.49% | 1.44% |
| Türkiye | 156K | 5.48% | 5.41% | 11.76% | 0.86% |
| Argentina | 135K | 5.21% | 4.91% | 9.16% | 1.60% |
| Belgium | 118K | 3.73% | 3.26% | 13.59% | 1.27% |
| Mexico | 113K | 3.96% | 3.93% | 12.99% | 1.63% |
| Vietnam | 108K | 7.09% | 7.03% | 8.48% | 2.61% |
By platform
| Segment | Robots.txt known | GPTBot | ClaudeBot | Name any AI crawler | Block allBlock all crawlers |
|---|---|---|---|---|---|
| Cloudflare | 5.75M | 9.33% | 9.01% | 13.74% | 2.73% |
| WordPress | 5.03M | 3.00% | 2.80% | 4.32% | 0.70% |
| Shopify | 688K | 2.76% | 2.69% | 2.43% | 2.05% |
| Wix | 540K | 0.40% | 0.37% | 95.77% | 0.09% |
| Squarespace | 297K | 2.53% | 2.52% | 97.17% | 0.94% |
| Joomla | 147K | 2.79% | 2.62% | 5.50% | 1.07% |
| Webflow | 124K | 2.81% | 2.50% | 5.46% | 1.10% |
| Drupal | 104K | 4.58% | 4.24% | 6.07% | 1.46% |
| Magento | 46,547 | 12.93% | 12.38% | 21.89% | 1.68% |
| Ghost | 6,563 | 3.35% | 3.11% | 5.53% | 0.81% |
Robots.txt known counts the segment's domains whose robots.txt policy we could determine; blocked and named percentages divide by that number, and the block-all column divides by the segment's domains with a robots.txt. Authority tiers come from Open Page Rank scores, traffic tiers from the newest Chrome UX Report month, country from where a site's Chrome users are, and platform from our technology detections.
Beyond robots.txt
How to read these numbers
Of the 44.73 million domains in this month's crawl: 31.21 million resolved in DNS, 29.14 million answered our fetch, and 24.63 million gave a definitive robots.txt answer (a readable file or a clean 404, which means everything is allowed). Percentages on this page use that last number unless labeled otherwise.
A robots.txt rule is a published preference, not an enforcement mechanism. We record what sites declare; we cannot see which crawlers comply. Method details on the methodology page.