What powers thi.ng?
Last seen Jul 2026 · 10 technologies across 7 categories
Analytics
Miscellaneous
PaaS
Security
UI frameworks
Web servers
AI Crawlers
Blocks 25 of 33 AI crawlers we track by name in its robots.txt. Checked Sep 2026.
Training crawlers
- AI2Bot Blocked
- Applebot-Extended Blocked
- Bytespider Blocked
- CCBot Blocked
- ClaudeBot Blocked
- cohere-training-data-crawler Blocked
- Diffbot Blocked
- FacebookBot Blocked
- Google-Extended Blocked
- GoogleOther Blocked
- GPTBot Blocked
- ImagesiftBot Blocked
- img2dataset Blocked
- meta-externalagent Blocked
- PanguBot Blocked
- Webzio-Extended Blocked
Search and AI assistants
- Amazonbot Blocked
- Applebot Blocked
- DuckAssistBot Blocked
- OAI-SearchBot Blocked
- PerplexityBot Blocked
- PetalBot Blocked
- YouBot Blocked
User-request fetchers
- ChatGPT-User Blocked
- meta-externalfetcher Blocked
The other 8 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt not checked
- ai.txt not checked
Top backlinks
Top links pointing at the site, one per source domain, strongest first. The anchor text opens the page carrying the link.
AI Crawlers
25 of 33 AI crawlers blocked
Open Page Rank
6.41 / 10 · ranked #81,069 of all domains
54 referring domains · OpenPageRank
Full technology history →
Report an incorrect detection
Recent changes
- dropped Amazon CloudFront Jul 2026
- dropped Amazon S3 Jul 2026