What powers webclaw.io?
Last seen Aug 2026 · 18 technologies across 12 categories
Affiliate programs
Analytics
CDN
Development
JavaScript frameworks
Miscellaneous
Operating systems
Performance
Programming languages
Security
UI frameworks
Web servers
AI Crawlers
Restricts 15 of 33 AI crawlers we track by name in its robots.txt. Checked Aug 2026.
Training crawlers
- Applebot-Extended Restricted
- Bytespider Restricted
- CCBot Restricted
- ClaudeBot Restricted
- cohere-training-data-crawler Restricted
- Google-Extended Restricted
- GPTBot Restricted
- meta-externalagent Restricted
Search and AI assistants
- Amazonbot Restricted
- DuckAssistBot Restricted
- OAI-SearchBot Restricted
- PerplexityBot Restricted
- YouBot Restricted
User-request fetchers
- ChatGPT-User Restricted
- Perplexity-User Restricted
The other 18 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt found
- llms-full.txt found
- ai.txt not found
Content Signals:
search=yes ai-train=no ai-input=yes declared in robots.txt
AI Crawlers
15 of 33 AI crawlers restricted
Standards: llms.txt, llms-full.txt, Content Signals
Open Page Rank
1.65 / 10 · ranked #5,315,275 of all domains
10 referring domains · OpenPageRank
Page
- Language: English
- Weight: 1.0 MB · 81 requests
- Structured data: JSON-LD, OpenGraph, Twitter Cards
Homepage as crawled by the HTTP Archive, Sep 2026.
Full technology history →
Report an incorrect detection
Recent changes
- detected Base UI Aug 2026
- detected jsDelivr Jul 2026
- detected Snowplow Analytics Jul 2026
- detected Tolt Jul 2026
- detected HSTS Jun 2026