Methodology

How PoweredBy detects, counts, and tracks web technologies, where the authority, traffic, and performance numbers next to them come from, and the choices we made.

What we count

Everything on PoweredBy counts registrable domains (per the Public Suffix List, including its private section): nike.com is one site, and www.nike.com folds into it, while someone.github.io counts as its own site. Subdomains are tracked underneath each domain and shown on its page, but market-share numbers never count a domain twice.

Where detections come from

Detections are produced by the open-source Wappalyzer ruleset maintained by HTTP Archive, the same fingerprints HTTP Archive runs in real Chrome browsers for its monthly crawl of the web's homepages. We pin an exact snapshot of that ruleset, so our numbers are consistent with the underlying crawl by construction. Our history is built from HTTP Archive's monthly datasets (mobile crawl, root pages), which we normalize, quality-check, and load into our own database.

Corpus history, read this before comparing months

Apr 2018 to Jul 2018: the crawl targeted roughly 450K sites (Alexa-based list), and technology detection was in its rollout period, about 12% of pages produced no detections at all. We treat those as failed crawls, not as evidence a site dropped its stack.
Aug 2018 onward: the corpus switched to the Chrome UX Report (CrUX) origin list and jumped to 1.3M+ pages, growing steadily since (5M+ by 2019, 15M+ today). Detection quality stabilized (~1% empty pages).
CrUX popularity ranks exist from May 2021 onward. We use them internally (ordering long lists, choosing which pages search engines should index); the authority figures shown on pages are Open Page Rank.

When do we say a site "dropped" a technology?

Only after two consecutive successful render-grade crawls of the same host both fail to detect it, and every host of the domain that carried the technology has independently confirmed the absence. Blocked, errored, or robots-disallowed fetches never count as absence, and a host disappearing from the crawl corpus doesn't either. This is deliberately conservative: false "X dropped Y!" stories are worse than a few days of staleness.

Historical technology names

Fingerprint rulesets rename and retire technologies over the years (Piwik became Matomo; IBM WebSphere Commerce became HCL Commerce; Google+ died). We maintain a curated mapping so history stays continuous under today's names, and genuinely discontinued products keep their own pages - their adoption and disappearance are part of the record.

Authority scores: Open Page Rank

The authority score shown for domains is Open Page Rank: 0 to 10, computed from referring domains across the open web link graph, with an exact rank among more than 121 million scored domains. We load the full dataset and refresh it as new releases land. It also orders every "websites using X" list: highest authority first, then Chrome UX popularity, then everything else. A domain without a score simply shows no authority block.

Audience and Core Web Vitals: Chrome UX Report

Audience countries, device splits, and Core Web Vitals come from the Chrome UX Report (CrUX), Google's monthly dataset of real Chrome usage, which covers about 12 million domains. Three things to keep in mind:

Countries are where a site's Chrome visitors browse from, not where its servers sit. CrUX reports a site's popularity rank in each country rather than visitor counts, so the audience list is ordered by an estimate that weighs the size of each country's Chrome web against the site's rank there. A rank-500K site in the United States outranks a rank-100K site in a market a tenth the size, while a local site still leads its home country.
Web Vitals (LCP, INP, CLS, TTFB) are p75 values from real users, not lab scores.
Per-technology vitals are the median site among that technology's CrUX-covered sites. They describe the sites that chose the technology, not overhead the technology causes.

Search traffic, keywords, and backlinks

Estimated monthly organic visits, ranking-keyword counts (Google top 30), top keywords, and top backlinks come from Keywords Everywhere, for the US Google market. These are estimates from a search index, not the site's own analytics: treat them as order-of-magnitude signals. We refresh them in the background roughly every three months, prioritized by site authority, with full keyword and backlink panels for higher-authority domains and visit estimates for the rest, so a figure can be up to about a quarter behind the live index. Keyword and backlink lists show the top 25 each, backlinks one per source domain; technology pages show the same visit estimates for the sites they list.

Page facts

Language, CDN, page weight and request counts, WordPress themes, and structured-data flags come from the same monthly HTTP Archive crawl as our detections, so they describe the site's homepage as that crawl fetched it. Page weight is what the crawl's mobile Chrome downloaded for the root page, which can differ from what your browser loads.

AI crawlers

When a site is scanned live (the Rescan button), PoweredByBot also fetches its robots.txt, llms.txt, llms-full.txt, and ai.txt and records which AI crawlers the robots.txt blocks. The domain page's "AI Crawlers" section shows the result: a crawler is Blocked when every probe path we test against the site's own rules is denied for its user-agent token, Restricted when some paths are denied and some allowed, and Allowed when nothing denies it. We parse robots.txt with the same matching rules Google's open source parser uses, and we test every Allow carve-out a rule group declares, so a site that blocks everything except a few paths counts as Restricted, not Blocked. When in doubt we round toward Allowed: we never claim a crawler is blocked without proof.

A solid label means the site names that crawler's token (or a legacy alias of it) in its own rule group; an outlined label means the verdict is inherited from the catch-all User-agent: * group or is simply the default. Declared policy is not enforcement: a site can block a crawler in robots.txt while its CDN admits it, or admit it on paper while blocking it at the network level. We report what the files say. Checks run per scan, and a policy file that temporarily fails to load never changes a site's recorded policy; only a repeated, confirmed change does.

The aggregate numbers on the AI Crawlers pages come from our own monthly batch crawl of every domain we have ever tracked, run from the crawl addresses published on the bot page with the same parser and the same rounding toward Allowed. Every percentage names its denominator. The bases, from widest to narrowest: the universe is every domain in that month's crawl; resolved domains answered DNS; reachable domains answered our fetch; domains with a known robots.txt policy gave a definitive robots.txt answer, meaning a readable file or a clean 404 (no file means nothing is blocked, so a 404 is an answer, not a failure). Blocked counts add explicit by-name rules and blanket User-agent: * blocks the crawler inherits; where a page separates the two it says so. Domains that timed out, returned server errors, or answered with a bot-challenge page stay out of the known-policy base rather than being guessed at.

Charts labeled HTTP Archive use a separate historical series: how many sites in each month's HTTP Archive crawl named a token in robots.txt, from July 2021 onward. The archive keeps counts, not rule text, so that series measures mentions only (it cannot say whether the mention blocked anything), covers the archive's own site population rather than ours, and is drawn separately from our crawl series, never merged with it. llms.txt, llms-full.txt, and ai.txt adoption counts every domain that answers with a file at the standard path; some responses are empty or boilerplate, so adoption totals are upper bounds on deliberate use. Content-Signal counts read the machine-readable search, ai-train, and ai-input preferences some robots.txt files carry; a large share arrive through CDN-managed files rather than hand edits.

Freshness

HTTP Archive publishes monthly; we ingest each new month shortly after it lands. Any site can be rescanned live from its own page (the Rescan button), and a site we have never seen is scanned the moment someone looks it up. Every domain page shows when its data was last seen. Authority, audience, and search figures refresh on their own cadences, noted in their sections above.

Known limitations

Homepage-only detection: technologies used solely on inner pages can be missed.
Server-side stacks with no client-visible fingerprint (headers, cookies, markup) are undetectable by anyone using this approach.
Version numbers exist only where a fingerprint exposes them; absence of a version is not absence of the technology.
Detections before Aug 2018 cover a much smaller corpus (see above), treat early-2018 first-seen dates as "at least since".
Authority, audience, vitals, and search figures cover only domains present in their source datasets; a missing block means no data, not zero.
Search traffic, keyword, and backlink figures are estimates for the US Google market, not measurements from the site itself.

Attribution

Crawl data: HTTP Archive (thank you). Popularity ranks: Chrome UX Report. Domain rules: the Public Suffix List. Fingerprints: HTTP Archive's Wappalyzer fork (GPL-3.0, used server-side). Authority scores: Open Page Rank. Search estimates: Keywords Everywhere. Cite us as: "PoweredBy, the free technology profiler from Keywords Everywhere."