What powers busdex.io?
Last seen Jul 2026 · 9 technologies across 6 categories
Advertising
Google AdSense
since Jul 2026
CDN
Cloudflare
since Jul 2026
Miscellaneous
Programming languages
Security
Web frameworks
AI Crawlers
Restricts 13 of 33 AI crawlers we track by name in its robots.txt. Checked Aug 2026.
Training crawlers
- Applebot-Extended Restricted
- Bytespider Restricted
- CCBot Restricted
- ClaudeBot Restricted
- cohere-training-data-crawler Restricted
- Google-Extended Restricted
- GoogleOther Restricted
- GPTBot Restricted
Search and AI assistants
- Amazonbot Restricted
- OAI-SearchBot Restricted
- PerplexityBot Restricted
User-request fetchers
- ChatGPT-User Restricted
- Perplexity-User Restricted
The other 20 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt not found
- ai.txt not found