What powers
webscraping.ai?
Last seen Oct 2026 · 34 technologies across 18 categories
Analytics
CDN
Documentation
Font scripts
Issue trackers
JavaScript frameworks
JavaScript libraries
Miscellaneous
PaaS
Payment processors
Programming languages
RUM
Search engines
Security
UI frameworks
Web frameworks
Webmail
AI Crawlers
Restricts 12 of 33 AI crawlers we track by name in its robots.txt. Checked Nov 2026.
Training crawlers
- Applebot-Extended Restricted
- CCBot Restricted
- ClaudeBot Restricted
- Google-Extended Restricted
- GPTBot Restricted
- meta-externalagent Restricted
Search and AI assistants
- Claude-SearchBot Restricted
- OAI-SearchBot Restricted
- PerplexityBot Restricted
User-request fetchers
- ChatGPT-User Restricted
- Claude-User Restricted
- Perplexity-User Restricted
The other 21 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.
- robots.txt found
- llms.txt found
- llms-full.txt found
- ai.txt not found
Content Signals:
search=yes ai-train=yes ai-input=yes declared in robots.txt
- Sep 2026 oai-searchbot added to robots.txt
- Sep 2026 chatgpt-user added to robots.txt
- Sep 2026 claude-searchbot added to robots.txt
- Sep 2026 ccbot added to robots.txt
- Sep 2026 llms.txt appeared
- Sep 2026 llms-full.txt appeared
AI Crawlers
12 of 33 AI crawlers restricted
Standards: llms.txt, llms-full.txt, Content Signals
Open Page Rank
4.69 / 10 · ranked #321,694 of all domains
67 referring domains · OpenPageRank
Page
- Language: English
- CDN: Cloudflare
- Weight: 1.2 MB · 88 requests
- Structured data: JSON-LD, OpenGraph, Twitter Cards
Homepage as crawled by the HTTP Archive, Oct 2026.
Organisation
- WebScraping.AI
- Industry: Software and apps (declares
SoftwareApplication) - Profiles: GitHub, X, LinkedIn
From the homepage's own structured data (Oct 2026).
Infrastructure
- Registered Oct 2019 (7 years) via Porkbun LLC, renews Oct 2029
- Hosted on Cloudflare (104.26.10.196)
- DNS: Cloudflare DNS
- Email: Google Workspace
- Email security: SPF (hard fail), DMARC reject
- SSL: Google Trust Services, ECDSA P256, wildcard, expires Dec 2026
- Common Crawl link graph: rank 2,086,553, 3 hosts
- Verified with: Google, Postman
DNS records as swept Nov 2026, certificate as seen Nov 2026, registration from the registry's RDAP service. How the web is hosted
Subdomains
| Host | Techs |
|---|---|
| webscraping.ai | 24 |
| docs.webscraping.ai | 18 |
Full technology history →
Report an incorrect detection
Recent changes
- dropped Cloudflare Browser Insights Feb 2026
- detected cdnjs Jan 2026
- detected Font Awesome Jan 2026
- detected jQuery CDN Jan 2026
- detected jsDelivr Jan 2026
- detected Popper Jan 2026