What powers crawlproof.com?
Last seen Aug 2026 · 15 technologies across 10 categories
CDN
Cloudflare
since Jun 2026
Development
Turbopack
since Jun 2026
Amazon SES
since Jun 2026
JavaScript frameworks
JavaScript graphics
Recharts
since Jun 2026
Miscellaneous
PaaS
Amazon Web Services
since Jun 2026
Performance
Priority Hints
since Jun 2026
Programming languages
Node.js
since Jun 2026
UI frameworks
Tailwind CSS
since Jun 2026
AI Crawlers
Restricts 14 of 33 AI crawlers we track by name in its robots.txt. Checked Sep 2026.
Training crawlers
- Applebot-Extended Restricted
- Bytespider Restricted
- CCBot Restricted
- ClaudeBot Restricted
- FacebookBot Restricted
- Google-Extended Restricted
- GPTBot Restricted
- meta-externalagent Restricted
Search and AI assistants
- Claude-SearchBot Restricted
- OAI-SearchBot Restricted
- PerplexityBot Restricted
User-request fetchers
- ChatGPT-User Restricted
- Claude-User Restricted
- Perplexity-User Restricted
The other 19 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt found
- ai.txt not found
- Sep 2026 chatgpt-user added to robots.txt
- Sep 2026 claude-searchbot added to robots.txt
- Sep 2026 claude-user added to robots.txt
- Sep 2026 ccbot: Restricted (was Allowed)
- Sep 2026 *: Restricted (was Allowed)
- Sep 2026 anthropic-ai added to robots.txt
AI Crawlers
14 of 33 AI crawlers restricted
Standards: llms.txt
Open Page Rank
0.80 / 10 · ranked #16,562,541 of all domains
3 referring domains · OpenPageRank
Infrastructure
- Hosted on RAILWAY (69.46.46.37)
- DNS: Porkbun
- Email: Forward Email
- Email security: no SPF record, no DMARC record
- SSL: Let's Encrypt, ECDSA P256, expires Dec 2026
- Common Crawl link graph: rank 28,069,756
DNS records as swept Sep 2026, certificate as seen Sep 2026. How the web is hosted