Websites blocking OAI-SearchBot

OpenAI's search crawler. It builds the index behind ChatGPT search features and is separate from GPTBot, so sites can allow search visibility while blocking model training.

Operated by OpenAI · Search and AI assistant · operator documentation

3.98% block OAI-SearchBot 981K of 24.68 million domains with a known robots.txt policy 0.22% block it by name 53,688 domains, the rest inherit a blanket * block 0.61% restrict it by name 151K domains name it and close some of its paths 1.62% name it in robots.txt 400K domains give it rules of their own

Data for Oct 2026. All four percentages divide by the same 24.68 million domains. Blocked counts both explicit User-agent: OAI-SearchBot blocks and blanket * blocks the crawler inherits.

Blocking trend

Share of domains with a known robots.txt policy blocking OAI-SearchBot in each monthly crawl. The denominator moves with each crawl and is defined in the methodology.

Historical mentions, HTTP Archive

Share of sites in each month's HTTP Archive crawl whose robots.txt named OAI-SearchBot. This series counts mentions only (the archive keeps no rule text), comes from a different population than our own crawl above, and the two are never merged.

High-authority sites blocking OAI-SearchBot by name

Domain OPROpen Page Rank Blocking since
nytimes.com 9.28/10 Aug 2026
bbc.co.uk 9.18/10 Nov 2026
cnn.com 9.16/10 Nov 2026
msn.com 9.04/10 Nov 2026
hotjar.com 8.96/10 Nov 2026
theconversation.com 8.96/10 Nov 2026
huffpost.com 8.91/10 Aug 2026
amazon.ca 8.91/10 Nov 2026
claude.ai 8.87/10 Sep 2026
mashable.com 8.87/10 Sep 2026
francetvinfo.fr 8.87/10 Aug 2026
moz.com 8.83/10 Nov 2026
radiofrance.fr 8.82/10 Aug 2026
wikihow.com 8.81/10 Aug 2026
mirror.co.uk 8.73/10 Nov 2026
metro.co.uk 8.72/10 Aug 2026
ctvnews.ca 8.72/10 Aug 2026
chicagotribune.com 8.70/10 Nov 2026
overheid.nl 8.68/10 Oct 2026
newscientist.com 8.65/10 Nov 2026
express.co.uk 8.65/10 Nov 2026
nhk.jp 8.64/10 Aug 2026
launchpad.net 8.63/10 Nov 2026
asahi.com 8.62/10 Nov 2026
deadline.com 8.62/10 Nov 2026
francebleu.fr 8.61/10 Aug 2026
cargocollective.com 8.60/10 Aug 2026
franceculture.fr 8.58/10 Aug 2026
archives-ouvertes.fr 8.58/10 Aug 2026
towardsdatascience.com 8.57/10 Aug 2026

Sites whose robots.txt names OAI-SearchBot (or a legacy alias) with a full block, ranked by Open Page Rank. Sites blocking every crawler with a blanket * rule are counted above but not listed here. Since dates start at our first observation of the rule.

Restricting some paths

Crawler first seen in the wild around Jul 2024. How to block it and what each state means is on the methodology page.

Look up any website for free

No paywall, no lead-gen, no account. Paste a domain and see what powers it.

No account neededHistory since 2018