What powers aoagents.dev?
Last seen Aug 2026 · 19 technologies across 11 categories
Analytics
Caching
CDN
Development
JavaScript frameworks
Miscellaneous
PaaS
Performance
Programming languages
Security
UI frameworks
AI Crawlers
Blocks 2 of 33 AI crawlers we track by name in its robots.txt. Checked Oct 2026.
Training crawlers
- Bytespider Blocked
- CCBot Blocked
- Applebot-Extended Allowed
- Google-Extended Allowed
- GoogleOther Allowed
- GPTBot Allowed
- meta-externalagent Allowed
Search and AI assistants
- Claude-SearchBot Allowed
- DuckAssistBot Allowed
- OAI-SearchBot Allowed
- PerplexityBot Allowed
User-request fetchers
- ChatGPT-User Allowed
- Claude-User Allowed
The other 20 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt found
- llms-full.txt found
- ai.txt not found
Content Signals:
search=yes ai-train=yes ai-input=yes declared in robots.txt
AI Crawlers
2 of 33 AI crawlers blocked
Standards: llms.txt, llms-full.txt, Content Signals
Infrastructure
- Hosted on Cloudflare (104.21.78.22, IPv6)
- DNS: Cloudflare DNS
- Email: pkarnal.com
- Email security: SPF (hard fail), DMARC monitor only
- SSL: Google Trust Services, ECDSA P256, expires Dec 2026
- Common Crawl link graph: rank 28,977,791
DNS records as swept Oct 2026, certificate as seen Oct 2026. How the web is hosted