PoweredByBot & data collection policy

How PoweredBy gathers data, what our bot does, and how to opt out.

Current data source

Today, PoweredBy's data comes from the HTTP Archive public dataset, we don't crawl your site ourselves yet. On-demand rescans and our own crawler (PoweredByBot) are rolling out; this page is their standing policy.

PoweredByBot behavior

• User-Agent: PoweredByBot (+https://poweredby.keywordseverywhere.com/bot)
• Fetches homepages only, never deep-crawls your site.
• Respects robots.txt everywhere, including user-requested rescans, which is stricter than most scanners. If your robots.txt disallows us (or all bots), we don't fetch, even when a user clicks "rescan".
• Low frequency: popular sites at most weekly, others monthly or quarterly.
• No personal data: we record technology fingerprints, never content, emails, or contacts.

Opting out

Stop future crawling: disallow PoweredByBot (or all bots) in your robots.txt. We honor it automatically, existing data freezes at its last-known state and only updates if HTTP Archive's own crawl still covers your site.

User-agent: PoweredByBot
Disallow: /

Hide your site's pages here: verified owners can request removal of their domain's lookup and history pages from public view (the domain still counts in aggregate statistics, that keeps the stats honest). You prove control with a DNS TXT record; it takes a minute.

Contact

Abuse, questions, removals: Keywords Everywhere support, mention PoweredBy.