What powers nycdatascience.edu?
Last seen Jul 2026 · 19 technologies across 13 categories
Advertising
Affiliate programs
Analytics
CDN
Cookie compliance
Ecommerce
JavaScript frameworks
JavaScript libraries
Miscellaneous
PaaS
Tag managers
Webmail
Widgets
AI Crawlers
Blocks 8 of 33 AI crawlers we track by name in its robots.txt. Checked Sep 2026. A recent check saw a policy file missing; awaiting confirmation.
Training crawlers
- Applebot-Extended Blocked
- Bytespider Blocked
- CCBot Blocked
- ClaudeBot Blocked
- Google-Extended Blocked
- GPTBot Blocked
- meta-externalagent Blocked
Search and AI assistants
- Amazonbot Blocked
The other 25 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt found
- ai.txt not found
Content Signals:
search=yes ai-train=no declared in robots.txt
AI Crawlers
8 of 33 AI crawlers blocked
Standards: llms.txt, Content Signals
Open Page Rank
4.00 / 10 · ranked #565,578 of all domains
42 referring domains · OpenPageRank