What powers webscraping.ai?

Last seen Oct 2026 · 34 technologies across 18 categories

Analytics

Google Analytics
since Jul 2022
PostHog
since Jul 2023

CDN

cdnjs
since Jan 2026
Cloudflare
since Jul 2022
jQuery CDN
since Jan 2026
jsDelivr
since Jan 2026
Unpkg
since May 2023

Documentation

ReadMe
since May 2023

Email

Amazon SES
since Aug 2022

Font scripts

Font Awesome v6.5.1
since Jan 2026
Google Font API
since Jul 2022

Issue trackers

Sentry
since May 2023

JavaScript frameworks

AngularJS
since May 2023
React
since May 2023
Stimulus
since Jul 2022

JavaScript libraries

AOS
since Feb 2023
core-js v3.45.1
since Jul 2022
jQuery v3.7.1
since Jul 2022
Loadable-Components
since May 2023
Lodash v4.17.21
since May 2023

Miscellaneous

HTTP/3
since May 2023
Open Graph
since May 2023
Popper
since Jan 2026
Prism
since Jul 2022

PaaS

Amazon Web Services
since Aug 2022
Heroku
since Jul 2022

Payment processors

Stripe v3
since Jul 2022

Programming languages

Ruby
since Jul 2022

RUM

New Relic
since Aug 2024

Search engines

Algolia
since May 2023

Security

HSTS
since Sep 2022

UI frameworks

Bootstrap v4.6.2
since Jul 2022

Web frameworks

Ruby on Rails
since Jul 2022

Webmail

Google Workspace
since Aug 2022

AI Crawlers

Restricts 12 of 33 AI crawlers we track by name in its robots.txt. Checked Nov 2026.

Training crawlers

  • Applebot-Extended Restricted
  • CCBot Restricted
  • ClaudeBot Restricted
  • Google-Extended Restricted
  • GPTBot Restricted
  • meta-externalagent Restricted

Search and AI assistants

  • Claude-SearchBot Restricted
  • OAI-SearchBot Restricted
  • PerplexityBot Restricted

User-request fetchers

  • ChatGPT-User Restricted
  • Claude-User Restricted
  • Perplexity-User Restricted

The other 21 AI crawlers we track are not mentioned and only inherit the catch-all rules that apply to every crawler.

Content Signals: search=yes ai-train=yes ai-input=yes declared in robots.txt

  • Sep 2026 oai-searchbot added to robots.txt
  • Sep 2026 chatgpt-user added to robots.txt
  • Sep 2026 claude-searchbot added to robots.txt
  • Sep 2026 ccbot added to robots.txt
  • Sep 2026 llms.txt appeared
  • Sep 2026 llms-full.txt appeared

See who blocks each AI crawler across the web

AI Crawlers

12 of 33 AI crawlers restricted

Standards: llms.txt, llms-full.txt, Content Signals

Full breakdown

Open Page Rank

4.69 / 10 · ranked #321,694 of all domains

67 referring domains · OpenPageRank

Page

  • Language: English
  • CDN: Cloudflare
  • Weight: 1.2 MB · 88 requests
  • Structured data: JSON-LD, OpenGraph, Twitter Cards

Homepage as crawled by the HTTP Archive, Oct 2026.

Organisation

  • WebScraping.AI
  • Industry: Software and apps (declares SoftwareApplication)
  • Profiles: GitHub, X, LinkedIn

From the homepage's own structured data (Oct 2026).

Infrastructure

  • Registered Oct 2019 (7 years) via Porkbun LLC, renews Oct 2029
  • Hosted on Cloudflare (104.26.10.196)
  • DNS: Cloudflare DNS
  • Email: Google Workspace
  • Email security: SPF (hard fail), DMARC reject
  • SSL: Google Trust Services, ECDSA P256, wildcard, expires Dec 2026
  • Common Crawl link graph: rank 2,086,553, 3 hosts
  • Verified with: Google, Postman

DNS records as swept Nov 2026, certificate as seen Nov 2026, registration from the registry's RDAP service. How the web is hosted

Subdomains

HostTechs
webscraping.ai 24
docs.webscraping.ai 18

Full technology history →
Report an incorrect detection

Recent changes

Look up any website for free

No paywall, no lead-gen, no account. Paste a domain and see what powers it.

No account neededHistory since 2018