What powers web-scraping-guide.com?
Last seen Oct 2026 · 9 technologies across 9 categories
Advertising
Google AdSense
since Jul 2026
Analytics
Google Analytics
since Jul 2026
Caching
Varnish
since Jul 2026
CDN
Fastly
since Jul 2026
Font scripts
Google Font API
since Jul 2026
Miscellaneous
Open Graph
since Jul 2026
PaaS
GitHub Pages
since Jul 2026
Performance
Priority Hints
since Sep 2026
Security
reCAPTCHA
since Jul 2026
AI Crawlers
Does not block any of the 33 AI crawlers we track in its robots.txt. Checked Oct 2026.
Training crawlers
- Applebot-Extended Allowed
- Bytespider Allowed
- ClaudeBot Allowed
- cohere-training-data-crawler Allowed
- Google-Extended Allowed
- GPTBot Allowed
- meta-externalagent Allowed
Search and AI assistants
- Amazonbot Allowed
- Applebot Allowed
- Claude-SearchBot Allowed
- DuckAssistBot Allowed
- OAI-SearchBot Allowed
- PerplexityBot Allowed
- YouBot Allowed
User-request fetchers
- ChatGPT-User Allowed
- Claude-User Allowed
The other 17 AI crawlers we track are not mentioned and stay allowed by default.
- robots.txt found
- llms.txt found
- ai.txt not found
Infrastructure
- Hosted on Fastly (185.199.108.153)
- DNS: Google Cloud DNS
- Email: no MX record returned
- Email security: SPF (hard fail), DMARC reject, DNSSEC signed
- SSL: Let's Encrypt, RSA 2048, expires Dec 2026
- Common Crawl link graph: rank 63,095,854
- Verified with: Google
DNS records as swept Oct 2026, certificate as seen Oct 2026. How the web is hosted
Full technology history →
Report an incorrect detection
Recent changes
- detected Priority Hints Sep 2026