What powers developress.io?
Last seen Jul 2026 · 28 technologies across 18 categories
Advertising
Analytics
CDN
CMS
Cookie compliance
Databases
JavaScript libraries
Miscellaneous
PaaS
Page builders
Performance
Programming languages
Reviews
Security
Tag managers
Web servers
WordPress plugins
AI Crawlers
Blocks 8 of 33 AI crawlers we track by name in its robots.txt. Checked Aug 2026.
Training crawlers
- Applebot-Extended Blocked
- Bytespider Blocked
- CCBot Blocked
- ClaudeBot Blocked
- Google-Extended Blocked
- GPTBot Blocked
- meta-externalagent Blocked
Search and AI assistants
- Amazonbot Blocked
- Applebot Allowed
- Claude-SearchBot Allowed
- OAI-SearchBot Allowed
- PerplexityBot Allowed
User-request fetchers
- ChatGPT-User Allowed
- Claude-User Allowed
- Perplexity-User Allowed
The other 18 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt not found
- ai.txt not found
Content Signals:
search=yes ai-train=no ai-input=yes declared in robots.txt
AI Crawlers
8 of 33 AI crawlers blocked
Standards: Content Signals
Open Page Rank
2.69 / 10 · ranked #2,176,703 of all domains
19 referring domains · OpenPageRank
Full technology history →
Report an incorrect detection
Recent changes
- dropped Cart Functionality Dec 2025
- dropped jQuery Dec 2025
- dropped Priority Hints Dec 2025
- dropped Sentry Dec 2025
- dropped Google PageSpeed Oct 2025
- dropped GSAP Oct 2025
- dropped Hotjar Oct 2025