What powers robotstxt.org?
Last seen Jun 2026 · 8 technologies across 7 categories
Advertising
Cookie compliance
Font scripts
Operating systems
Security
Tag managers
Web servers
Top search keywords
| Keyword | Position | Est. monthly visits |
|---|---|---|
| robots.txt disallow all | #2 | 260 |
| robots.txt disallow everything | #3 | 160 |
| web robots | #2 | 42 |
| robotstxt.org | #1 | 27 |
| web robot | #3 | 25 |
| robots.txt meaning | #5 | 23 |
| robots nofollow | #4 | 17 |
| robots.txt allow | #6 | 16 |
| meta robots | #12 | 14 |
| robots txt block all | #4 | 11 |
| robots.txt validator | #12 | 10 |
| robot txt | #7 | 9 |
| site robots | #3 | 9 |
| meta tag robots | #5 | 9 |
| web robots.txt | #3 | 7 |
| robots txt allow and disallow | #4 | 7 |
| robots txt allow all | #5 | 7 |
| what is robots.txt | #18 | 6 |
| robots txt protocol | #4 | 6 |
| robots txt code | #9 | 6 |
| robots.txt file | #37 | 5 |
| robots disallow | #10 | 5 |
| disallow robots | #7 | 5 |
| meta robotic | #80 | 4 |
| robots.txt validator | #46 | 3 |
The site's top keywords in Google (US), with the visits each one is estimated to bring per month.
Top backlinks
Top links pointing at the site, one per source domain, strongest first. The anchor text opens the page carrying the link.
Audience
- 71% desktop
- 29% phone
India
United States
Countries where the site has Chrome traffic, strongest markets first.
Open Page Rank
8.14 / 10 · ranked #5,626 of all domains
1,208 referring domains · OpenPageRank
Search traffic
740 est. monthly organic visits (US)
173 keywords in Google's top 30
Core Web Vitals
| LCP | 1.8 s |
| INP | 175 ms |
| CLS | 0 |
| TTFB | 1000 ms |
Real user p75, Chrome UX Report, Jun 2026.
Page
- Weight: 0.9 MB · 72 requests
Homepage as crawled by the HTTP Archive, Jul 2026.
Full technology history →
Report an incorrect detection
Recent changes
- detected Google Tag Manager Feb 2026
- dropped Google Tag Manager Jan 2026
- detected Google Font API Nov 2025
- dropped Snowplow Analytics Nov 2025
- detected Snowplow Analytics Aug 2025