🕵️‍♂️ All-in-one OSINT tool for analysing any website
Go to file
2024-06-15 20:04:19 +01:00
.github docs: Updates contributors list 2024-05-05 03:01:51 +01:00
api JSON parsing in API endpoints 2024-05-18 15:01:52 +01:00
public Updates to new icon 2024-06-03 23:37:48 +01:00
src Builds footer, and updates new homepage 2024-06-08 15:23:25 +01:00
.env docs: Updates samples 2024-06-15 20:04:19 +01:00
.gitignore Ignore built files 2024-05-07 12:03:28 +01:00
astro.config.mjs chore: Redirect /about to /check/about 2024-06-08 15:20:14 +01:00
docker-compose.yml Update exposed ports 2023-08-12 11:27:22 +02:00
Dockerfile Update start command, and remove --prod flag from dev install script in Dockerfile 2024-05-08 21:19:20 +01:00
fly.toml Adds a fly.toml for deploying to fly.io 2024-05-11 23:13:25 +01:00
LICENSE Adds MIT license 2023-07-09 01:39:48 +01:00
netlify.toml Update publish directory for Netlify 2024-05-07 12:38:56 +01:00
package.json dep: Adds Font Awesome, and builds an Icon component 2024-06-08 15:20:39 +01:00
server.js Fix catch-all when GUI is disabled 2024-05-20 22:43:53 +01:00
svelte.config.js Adds Svelte, Framer and starts work on homepage 2024-05-26 22:15:29 +01:00
tsconfig.json Adds aliases for styles and assets 2024-05-20 22:44:29 +01:00
vercel.json Updates configration for Vercel and Typescript 2024-05-05 17:45:23 +01:00
vite.config.js Configuration for Vite 2024-05-05 17:44:14 +01:00
yarn.lock dep: Adds Font Awesome, and builds an Icon component 2024-06-08 15:20:39 +01:00

Web-Check


Comprehensive, on-demand open source intelligence for any website
🌐 web-check.xyz


Kindly supported by:
Terminal Trove
The $HOME of all things in the terminal.

Find your next CLI / TUI tool and more at Terminal Trove,
Get updates on new tools on our newsletter.


Contents


About

Get an insight into the inner-workings of a given website: uncover potential attack vectors, analyse server architecture, view security configurations, and learn what technologies a site is using.

Currently the dashboard will show: IP info, SSL chain, DNS records, cookies, headers, domain info, search crawl rules, page map, server location, redirect ledger, open ports, traceroute, DNS security extensions, site performance, trackers, associated hostnames, carbon footprint. Stay tuned, as I'll add more soon!

The aim is to help you easily understand, optimize and secure your website.

Screenshot

Expand Screenshot

Screenshot

Screenshot

Live Demo

A hosted version can be accessed at: web-check.as93.net

Mirror

The source for this repo is mirrored to CodeBerg, available at: codeberg.org/alicia/web-check

Status

Build & Deploys: Netlify Status Vercel Status 🐳 Build + Publish Docker Image 🚀 Deploy to AWS
Repo Management & Miscellaneous: 🪞 Mirror to Codeberg 💓 Inserts Contributors & Sponsors

Features

Click to expand / collapse section

Note this list needs updating, many more jobs have been added since...

The following section outlines the core features, and briefly explains why this data might be useful for you to know, as well as linking to further resources for learning more.

IP Info
Description

An IP address (Internet Protocol address) is a numerical label assigned to each device connected to a network / the internet. The IP associated with a given domain can be found by querying the Domain Name System (DNS) for the domain's A (address) record.

Use Cases

Finding the IP of a given server is the first step to conducting further investigations, as it allows us to probe the server for additional info. Including creating a detailed map of a target's network infrastructure, pinpointing the physical location of a server, identifying the hosting service, and even discovering other domains that are hosted on the same IP address.

SSL Chain
Description

SSL certificates are digital certificates that authenticate the identity of a website or server, enable secure encrypted communication (HTTPS), and establish trust between clients and servers. A valid SSL certificate is required for a website to be able to use the HTTPS protocol, and encrypt user + site data in transit. SSL certificates are issued by Certificate Authorities (CAs), which are trusted third parties that verify the identity and legitimacy of the certificate holder.

Use Cases

SSL certificates not only provide the assurance that data transmission to and from the website is secure, but they also provide valuable OSINT data. Information from an SSL certificate can include the issuing authority, the domain name, its validity period, and sometimes even organization details. This can be useful for verifying the authenticity of a website, understanding its security setup, or even for discovering associated subdomains or other services.

DNS Records
Description

This task involves looking up the DNS records associated with a specific domain. DNS is a system that translates human-readable domain names into IP addresses that computers use to communicate. Various types of DNS records exist, including A (address), MX (mail exchange), NS (name server), CNAME (canonical name), and TXT (text), among others.

Use Cases

Extracting DNS records can provide a wealth of information in an OSINT investigation. For example, A and AAAA records can disclose IP addresses associated with a domain, potentially revealing the location of servers. MX records can give clues about a domain's email provider. TXT records are often used for various administrative purposes and can sometimes inadvertently leak internal information. Understanding a domain's DNS setup can also be useful in understanding how its online infrastructure is built and managed.

Cookies
Description

The Cookies task involves examining the HTTP cookies set by the target website. Cookies are small pieces of data stored on the user's computer by the web browser while browsing a website. They hold a modest amount of data specific to a particular client and website, such as site preferences, the state of the user's session, or tracking information.

Use Cases

Cookies can disclose information about how the website tracks and interacts with its users. For instance, session cookies can reveal how user sessions are managed, and tracking cookies can hint at what kind of tracking or analytics frameworks are being used. Additionally, examining cookie policies and practices can offer insights into the site's security settings and compliance with privacy regulations.

Crawl Rules
Description

Robots.txt is a file found (usually) at the root of a domain, and is used to implement the Robots Exclusion Protocol (REP) to indicate which pages should be ignored by which crawlers and bots. It's good practice to avoid search engine crawlers from over-loading your site, but should not be used to keep pages out of search results (use the noindex meta tag or header instead).

Use Cases

It's often useful to check the robots.txt file during an investigation, as it can sometimes disclose the directories and pages that the site owner doesn't want to be indexed, potentially because they contain sensitive information, or reveal the existence of otherwise hidden or unlinked directories. Additionally, understanding crawl rules may offer insights into a website's SEO strategies.

Headers
Description

The Headers task involves extracting and interpreting the HTTP headers sent by the target website during the request-response cycle. HTTP headers are key-value pairs sent at the start of an HTTP response, or before the actual data. Headers contain important directives for how to handle the data being transferred, including cache policies, content types, encoding, server information, security policies, and more.

Use Cases

Analyzing HTTP headers can provide significant insights in an OSINT investigation. Headers can reveal specific server configurations, chosen technologies, caching directives, and various security settings. This information can help to determine a website's underlying technology stack, server-side security measures, potential vulnerabilities, and general operational practices.

Quality Metrics
Description

Using Lighthouse, the Quality Metrics task measures the performance, accessibility, best practices, and SEO of the target website. This returns a simple checklist of 100 core metrics, along with a score for each category, to gauge the overall quality of a given site.

Use Cases

Useful for assessing a site's technical health, SEO issues, identify vulnerabilities, and ensure compliance with standards.

Server Location
Description

The Server Location task determines the physical location of the server hosting a given website based on its IP address. This is done by looking up the IP in a location database, which maps the IP to a lat + long of known data centers and ISPs. From the latitude and longitude, it's then possible to show additional contextual info, like a pin on the map, along with address, flag, time zone, currency, etc.

Use Cases

Knowing the server location is a good first step in better understanding a website. For site owners this aids in optimizing content delivery, ensuring compliance with data residency requirements, and identifying potential latency issues that may impact user experience in specific geographical regions. And for security researcher, assess the risk posed by specific regions or jurisdictions regarding cyber threats and regulations.

Associated Hosts
Description

This task involves identifying and listing all domains and subdomains (hostnames) that are associated with the website's primary domain. This process often involves DNS enumeration to discover any linked domains and hostnames, as well as looking at known DNS records.

Use Cases

During an investigation, understanding the full scope of a target's web presence is critical. Associated domains could lead to uncovering related projects, backup sites, development/test sites, or services linked to the main site. These can sometimes provide additional information or potential security vulnerabilities. A comprehensive list of associated domains and hostnames can also give an overview of the organization's structure and online footprint.

Redirect Chain
Description

This task traces the sequence of HTTP redirects that occur from the original URL to the final destination URL. An HTTP redirect is a response with a status code that advises the client to go to another URL. Redirects can occur for several reasons, such as URL normalization (directing to the www version of the site), enforcing HTTPS, URL shorteners, or forwarding users to a new site location.

Use Cases

Understanding the redirect chain can be useful for several reasons. From a security perspective, long or complicated redirect chains can be a sign of potential security risks, such as unencrypted redirects in the chain. Additionally, redirects can impact website performance and SEO, as each redirect introduces additional round-trip-time (RTT). For OSINT, understanding the redirect chain can help identify relationships between different domains or reveal the use of certain technologies or hosting providers.

TXT Records
Description

TXT records are a type of DNS record that provides text information to sources outside your domain. They can be used for a variety of purposes, such as verifying domain ownership, ensuring email security, and even preventing unauthorized changes to your website.

Use Cases

The TXT records often reveal which external services and technologies are being used with a given domain. They may reveal details about the domain's email configuration, the use of specific services like Google Workspace or Microsoft 365, or security measures in place such as SPF and DKIM. Understanding these details can give an insight into the technologies used by the organization, their email security practices, and potential vulnerabilities.

Server Status
Description

Checks if a server is online and responding to requests.

Use Cases
Open Ports
Description

Open ports on a server are endpoints of communication which are available for establishing connections with clients. Each port corresponds to a specific service or protocol, such as HTTP (port 80), HTTPS (port 443), FTP (port 21), etc. The open ports on a server can be determined using techniques such as port scanning.

Use Cases

Knowing which ports are open on a server can provide information about the services running on that server, useful for understanding the potential vulnerabilities of the system, or for understanding the nature of the services the server is providing.

Traceroute
Description

Traceroute is a network diagnostic tool used to track in real-time the pathway taken by a packet of information from one system to another. It records each hop along the route, providing details about the IPs of routers and the delay at each point.

Use Cases

In OSINT investigations, traceroute can provide insights about the routing paths and geography of the network infrastructure supporting a website or service. This can help to identify network bottlenecks, potential censorship or manipulation of network traffic, and give an overall sense of the network's structure and efficiency. Additionally, the IP addresses collected during the traceroute may provide additional points of inquiry for further OSINT investigation.

  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
Carbon Footprint
Description

This task calculates the estimated carbon footprint of a website. It's based on the amount of data being transferred and processed, and the energy usage of the servers that host and deliver the website. The larger the website and the more complex its features, the higher its carbon footprint is likely to be.

Use Cases

From an OSINT perspective, understanding a website's carbon footprint doesn't directly provide insights into its internal workings or the organization behind it. However, it can still be valuable data in broader analyses, especially in contexts where environmental impact is a consideration. For example, it can be useful for activists, researchers, or ethical hackers who are interested in the sustainability of digital infrastructure, and who want to hold organizations accountable for their environmental impact.

Server Info
Description

This task retrieves various pieces of information about the server hosting the target website. This can include the server type (e.g., Apache, Nginx), the hosting provider, the Autonomous System Number (ASN), and more. The information is usually obtained through a combination of IP address lookups and analysis of HTTP response headers.

Use Cases

In an OSINT context, server information can provide valuable clues about the organization behind a website. For instance, the choice of hosting provider could suggest the geographical region in which the organization operates, while the server type could hint at the technologies used by the organization. The ASN could also be used to find other domains hosted by the same organization.

  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
Whois Lookup
Description

This task retrieves Whois records for the target domain. Whois records are a rich source of information, including the name and contact information of the domain registrant, the domain's creation and expiration dates, the domain's nameservers, and more. The information is usually obtained through a query to a Whois database server.

Use Cases

In an OSINT context, Whois records can provide valuable clues about the entity behind a website. They can show when the domain was first registered and when it's set to expire, which could provide insights into the operational timeline of the entity. The contact information, though often redacted or anonymized, can sometimes lead to additional avenues of investigation. The nameservers could also be used to link together multiple domains owned by the same entity.

  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
Domain Info
Description

This task retrieves Whois records for the target domain. Whois records are a rich source of information, including the name and contact information of the domain registrant, the domain's creation and expiration dates, the domain's nameservers, and more. The information is usually obtained through a query to a Whois database server.

Use Cases

In an OSINT context, Whois records can provide valuable clues about the entity behind a website. They can show when the domain was first registered and when it's set to expire, which could provide insights into the operational timeline of the entity. The contact information, though often redacted or anonymized, can sometimes lead to additional avenues of investigation. The nameservers could also be used to link together multiple domains owned by the same entity.

  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
DNS Security Extensions
Description

Without DNSSEC, it's possible for MITM attackers to spoof records and lead users to phishing sites. This is because the DNS system includes no built-in methods to verify that the response to the request was not forged, or that any other part of the process wasnt interrupted by an attacker. The DNS Security Extensions (DNSSEC) secures DNS lookups by signing your DNS records using public keys, so browsers can detect if the response has been tampered with. Another solution to this issue is DoH (DNS over HTTPS) and DoT (DNS over TLD).

Use Cases

DNSSEC information provides insight into an organization's level of cybersecurity maturity and potential vulnerabilities, particularly around DNS spoofing and cache poisoning. If no DNS secururity (DNSSEC, DoH, DoT, etc) is implemented, this may provide an entry point for an attacker.

  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
Site Features
Description

Checks which core features are present on a site. If a feature as marked as dead, that means it's not being actively used at load time

Use Cases

This is useful to understand what a site is capable of, and what technologies to look for

HTTP Strict Transport Security
Description

HTTP Strict Transport Security (HSTS) is a web security policy mechanism that helps protect websites against protocol downgrade attacks and cookie hijacking. A website can be included in the HSTS preload list by conforming to a set of requirements and then submitting itself to the list.

Use Cases

There are several reasons why it's important for a site to be HSTS enabled: 1. User bookmarks or manually types http://example.com and is subject to a man-in-the-middle attacker HSTS automatically redirects HTTP requests to HTTPS for the target domain 2. Web application that is intended to be purely HTTPS inadvertently contains HTTP links or serves content over HTTP HSTS automatically redirects HTTP requests to HTTPS for the target domain 3. A man-in-the-middle attacker attempts to intercept traffic from a victim user using an invalid certificate and hopes the user will accept the bad certificate HSTS does not allow a user to override the invalid certificate message

  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
  • [undefined](function link() { [native code] })
DNS Server
Description

This check determines the DNS server(s) that the requested URL / IP resolves to. Also fires off a rudimentary check to see if the DNS server supports DoH, and weather it's vulnerable to DNS cache poisoning.

Use Cases
Tech Stack
Description

Checks what technologies a site is built with. This is done by fetching and parsing the site, then comparing it against a bit list of RegEx maintained by Wappalyzer to identify the unique fingerprints that different technologies leave.

Use Cases

Identifying a website's tech stack aids in evaluating its security by exposing potential vulnerabilities, informs competitive analyses and development decisions, and can guide tailored marketing strategies. Ethical application of this knowledge is crucial to avoid harmful activities like data theft or unauthorized intrusion.

Listed Pages
Description

This job finds and parses a site's listed sitemap. This file lists public sub-pages on the site, which the author wishes to be crawled by search engines. Sitemaps help with SEO, but are also useful for seeing all a sites public content at a glance.

Use Cases

Understand the structure of a site's public-facing content, and for site-owners, check that you're site's sitemap is accessible, parsable and contains everything you wish it to.

Security.txt
Description

The security.txt file tells researchers how they can responsibly disclose any security issues found on your site. The standard was proposed in RFC 9116, and specifies that this file should include a point of contact (email address), as well as optionally other info, like a link to the security disclosure policy, PGP key, proffered language, policy expiry and more. The file should be located at the root of your domain, either at /security.txt or /.well-known/security.txt.

Use Cases

This is important, as without a defined point of contact a security researcher may be unable to report a critical security issue, or may use insecure or possibly public channels to do so. From an OSINT perspective, you may also glean info about a site including their posture on security, their CSAF provider, and meta data from the PGP public key.

Linked Pages
Description

Displays all internal and external links found on a site, identified by the href attributes attached to anchor elements.

Use Cases

For site owners, this is useful for diagnosing SEO issues, improving the site structure, understanding how content is inter-connected. External links can show partnerships, dependencies, and potential reputation risks. From a security standpoint, the outbound links can help identify any potential malicious or compromised sites the website is unknowingly linking to. Analyzing internal links can aid in understanding the site's structure and potentially uncover hidden or vulnerable pages which are not intended to be public. And for an OSINT investigator, it can aid in building a comprehensive understanding of the target, uncovering related entities, resources, or even potential hidden parts of the site.

Social Tags
Description

Websites can include certain meta tags, that tell search engines and social media platforms what info to display. This usually includes a title, description, thumbnail, keywords, author, social accounts, etc.

Use Cases

Adding this data to your site will boost SEO, and as an OSINT researcher it can be useful to understand how a given web app describes itself

Email Configuration
Description

DMARC (Domain-based Message Authentication, Reporting & Conformance): DMARC is an email authentication protocol that works with SPF and DKIM to prevent email spoofing and phishing. It allows domain owners to specify how to handle unauthenticated mail via a published policy in DNS, and provides a way for receiving mail servers to send feedback about emails' compliance to the sender. BIMI (Brand Indicators for Message Identification): BIMI is an emerging email standard that enables organizations to display a logo in their customers' email clients automatically. BIMI ties the logo to the domain's DMARC record, providing another level of visual assurance to recipients that the email is legitimate. DKIM (DomainKeys Identified Mail): DKIM is an email security standard designed to make sure that messages were not altered in transit between the sending and recipient servers. It uses digital signatures linked to the domain of the sender to verify the sender and ensure message integrity. SPF (Sender Policy Framework): SPF is an email authentication method designed to prevent email spoofing. It specifies which mail servers are authorized to send email on behalf of a domain by creating a DNS record. This helps protect against spam by providing a way for receiving mail servers to check that incoming mail from a domain comes from a host authorized by that domain's administrators.

Use Cases

This information is helpful for researchers as it helps assess a domain's email security posture, uncover potential vulnerabilities, and verify the legitimacy of emails for phishing detection. These details can also provide insight into the hosting environment, potential service providers, and the configuration patterns of a target organization, assisting in investigative efforts.

Firewall Detection
Description

A WAF or web application firewall helps protect web applications by filtering and monitoring HTTP traffic between a web application and the Internet. It typically protects web applications from attacks such as cross-site forgery, cross-site-scripting (XSS), file inclusion, and SQL injection, among others.

Use Cases

It's useful to understand if a site is using a WAF, and which firewall software / service it is using, as this provides an insight into the sites protection against several attack vectors, but also may reveal vulnerabilities in the firewall itself.

HTTP Security Features
Description

Correctly configured security HTTP headers adds a layer of protection against common attacks to your site. The main headers to be aware of are: HTTP Strict Transport Security (HSTS): Enforces the use of HTTPS, mitigating man-in-the-middle attacks and protocol downgrade attempts. Content Security Policy (CSP): Constrains web page resources to prevent cross-site scripting and data injection attacks. X-Content-Type-Options: Prevents browsers from MIME-sniffing a response away from the declared content type, curbing MIME-type confusion attacks. X-Frame-Options: Protects users from clickjacking attacks by controlling whether a browser should render the page in a <frame>, <iframe>, <embed>, or <object>.

Use Cases

Reviewing security headers is important, as it offers insights into a site's defensive posture and potential vulnerabilities, enabling proactive mitigation and ensuring compliance with security best practices.

Archive History
Description

Fetches full history of archives from the Wayback machine

Use Cases

This is useful for understanding the history of a site, and how it has changed over time. It can also be useful for finding old versions of a site, or for finding content that has been removed.

Global Ranking
Description

This check shows the global rank of the requested site. This is only accurate for websites which are in the top 100 million list. We're using data from the Tranco project (see below), which collates the top sites on the web from Umbrella, Majestic, Quantcast, the Chrome User Experience Report and Cloudflare Radar.

Use Cases

Knowing a websites overall global rank can be useful for understanding the scale of the site, and for comparing it to other sites. It can also be useful for understanding the relative popularity of a site, and for identifying potential trends.

Block Detection
Description

Checks access to the URL using 10+ of the most popular privacy, malware and parental control blocking DNS servers.

Use Cases
Malware & Phishing Detection
Description

Checks if a site appears in several common malware and phishing lists, to determine it's threat level.

Use Cases

Knowing if a site is listed as a threat by any of these services can be useful for understanding the reputation of a site, and for identifying potential trends.

TLS Cipher Suites
Description

These are combinations of cryptographic algorithms used by the server to establish a secure connection. It includes the key exchange algorithm, bulk encryption algorithm, MAC algorithm, and PRF (pseudorandom function).

Use Cases

This is important info to test for from a security perspective. Because a cipher suite is only as secure as the algorithms that it contains. If the version of encryption or authentication algorithm in a cipher suite have known vulnerabilities the cipher suite and TLS connection may then vulnerable to a downgrade or other attack

TLS Security Config
Description

This uses guidelines from Mozilla's TLS Observatory to check the security of the TLS configuration. It checks for bad configurations, which may leave the site vulnerable to attack, as well as giving advice on how to fix. It will also give suggestions around outdated and modern TLS configs

Use Cases

Understanding issues with a site's TLS configuration will help you address potential vulnerabilities, and ensure the site is using the latest and most secure TLS configuration.

TLS Handshake Simulation
Description

This simulates how different clients (browsers, operating systems) would perform a TLS handshake with the server. It helps identify compatibility issues and insecure configurations.

Use Cases
Screenshot
Description

This check takes a screenshot of webpage that the requested URL / IP resolves to, and displays it.

Use Cases

This may be useful to see what a given website looks like, free of the constraints of your browser, IP, or location.

Read more here: web-check.xyz/about


Usage

Deployment

Deploying - Option #1: Netlify

Click the button below, to deploy to Netlify 👇

Deploy to Netlify

Deploying - Option #2: Vercel

Click the button below, to deploy to Vercel 👇

Deploy with Vercel

Deploying - Option #3: Docker

Run docker run -p 3000:3000 lissy93/web-check, then open localhost:3000

Docker Options

You can get the Docker image from:

Deploying - Option #4: From Source

Install the prerequisites listed in the Developing section, then run:

git clone https://github.com/Lissy93/web-check.git  # Download the code from GitHub
cd web-check                                        # Navigate into the project dir
yarn install                                        # Install the NPM dependencies
yarn build                                          # Build the app for production
yarn serve                                          # Start the app (API and GUI)

Configuring

By default, no configuration is needed.

But there are some optional environmental variables that you can set to give you access to some additional checks, or to increase rate-limits for some checks that use external APIs.

API Keys & Credentials:

Key Value
GOOGLE_CLOUD_API_KEY A Google API key (get here). This can be used to return quality metrics for a site
REACT_APP_SHODAN_API_KEY A Shodan API key (get here). This will show associated host names for a given domain
REACT_APP_WHO_API_KEY A WhoAPI key (get here). This will show more comprehensive WhoIs records than the default job
Full / Upcoming Vals
  • GOOGLE_CLOUD_API_KEY - A Google API key (get here). This can be used to return quality metrics for a site
  • REACT_APP_SHODAN_API_KEY - A Shodan API key (get here). This will show associated host names for a given domain
  • REACT_APP_WHO_API_KEY - A WhoAPI key (get here). This will show more comprehensive WhoIs records than the default job
  • SECURITY_TRAILS_API_KEY - A Security Trails API key (get here). This will show org info associated with the IP
  • CLOUDMERSIVE_API_KEY - API key for Cloudmersive (get here). This will show known threats associated with the IP
  • TRANCO_USERNAME - A Tranco email (get here). This will show the rank of a site, based on traffic
  • TRANCO_API_KEY - A Tranco API key (get here). This will show the rank of a site, based on traffic
  • URL_SCAN_API_KEY - A URLScan API key (get here). This will fetch miscalanious info about a site
  • BUILT_WITH_API_KEY - A BuiltWith API key (get here). This will show the main features of a site
  • TORRENT_IP_API_KEY - A torrent API key (get here). This will show torrents downloaded by an IP

Configuration Settings:

Key Value
PORT Port to serve the API, when running server.js (e.g. 3000)
API_ENABLE_RATE_LIMIT Enable rate-limiting for the /api endpoints (e.g. true)
API_TIMEOUT_LIMIT The timeout limit for API requests, in milliseconds (e.g. 10000)
API_CORS_ORIGIN Enable CORS, by setting your allowed hostname(s) here (e.g. example.com)
CHROME_PATH The path the Chromium executable (e.g. /usr/bin/chromium)
DISABLE_GUI Disable the GUI, and only serve the API (e.g. false)
REACT_APP_API_ENDPOINT The endpoint for the API, either local or remote (e.g. /api)

All values are optional.

You can add these as environmental variables. Either put them directly into an .env file in the projects root, or via the Netlify / Vercel UI, or by passing to the Docker container with the --env flag, or using your own environmental variable management system

Note that keys that are prefixed with REACT_APP_ are used client-side, and as such they must be scoped correctly with minimum privileges, since may be made visible when intercepting browser <-> server network requests


Developing

  1. Clone the repo, git clone git@github.com:Lissy93/web-check.git
  2. Cd into it, cd web-check
  3. Install dependencies: yarn
  4. Start the dev server, with yarn dev

You'll need Node.js (V 18.16.1 or later) installed, plus yarn as well as git. Some checks also require chromium, traceroute and dns to be installed within your environment. These jobs will just be skipped if those packages aren't present.


Community

Contributing

Contributions of any kind are very welcome, and would be much appreciated. For Code of Conduct, see Contributor Convent.

To get started, fork the repo, make your changes, add, commit and push the code, then come back here to open a pull request. If you're new to GitHub or open source, this guide or the git docs may help you get started, but feel free to reach out if you need any support.

Submit a PR

Reporting Bugs

If you've found something that doesn't work as it should, or would like to suggest a new feature, then go ahead and raise a ticket on GitHub. For bugs, please outline the steps needed to reproduce, and include relevant info like system info and resulting logs.

Raise an Issue

Supporting

The app will remain 100% free and open source. But due to the amount of traffic that the hosted instance gets, the lambda function usage is costing about $25/month. Any help with covering the costs via GitHub Sponsorship would be much appreciated. It's thanks to the support of the community that this project is able to be freely available for everyone :)

Sponsor Lissy93 on GitHub

Contributors

Credit to the following users for contributing to Web-Check

Lissy93
Alicia Sykes
liss-bot
Alicia Bot
muni106
Mounir Samite
ChrisCarini
Chris Carini
brianteeman
Brian Teeman
GreyXor
GreyXor
jinnabaalu
Jinna Baalu
bolens
Michael Bolens
robinson
Lth
abhishekMuge
Abhishek Muge
UlisesGascon
Ulises Gascón
PhiRequiem
PhiRequiem
Myzel394
Myzel394
murrple-1
Murray Christopherson
t3chn0m4g3
Marco Ochse
treatmesubj
John Hupperts
eltociear
Ikko Eltociear Ashimine
Gertje823
Gertje823
epreston
Ed Preston
0xflotus
0xflotus

Sponsors

Huge thanks to these wonderful people, who sponsor me on GitHub, their support helps cover the costs required to keep Web-Check and my other projects free for everyone. Consider joining them, by sponsoring me on GitHub if you're able.

github
GitHub
koconder
Vincent Koc
Admonstrator
Aaron Viehl
tbjers
Torgny Bjers
emlazzarin
Eddy Lazzarin
AnandChowdhary
Anand Chowdhary
shrippen
Shrippen
bile0026
Zach Biles
UlisesGascon
Ulises Gascón
digitalarche
Digital Archeology
InDieTasten
InDieTasten
araguaci
Araguaci
bmcgonag
Brian McGonagill
vlad-timofeev
Vlad Timofeev
helixzz
HeliXZz
patvdv
Patrick Van Der Veken
plgonzalezrx8
Pedro Gonzalez
mryesiller
Göksel Yeşiller
allesauseinerhand
Allesauseinerhand
forwardemail
Forward Email - Open-source & Privacy-focused Email Service (2023)
lamtrinhdev
LamTrinh.Dev
Bastii717
Bastii717
getumbrel
Umbrel
M2TD
M2TD
frankdez93
Frankdez93
terminaltrove
Terminal Trove
NixyJuppie
Nixy
nrvo
Nrvo

License

Lissy93/Web-Check is licensed under MIT © Alicia Sykes 2023.
For information, see TLDR Legal > MIT

Expand License
The MIT License (MIT)
Copyright (c) Alicia Sykes <alicia@omg.com> 

Permission is hereby granted, free of charge, to any person obtaining a copy 
of this software and associated documentation files (the "Software"), to deal 
in the Software without restriction, including without limitation the rights 
to use, copy, modify, merge, publish, distribute, sub-license, and/or sell 
copies of the Software, and to permit persons to whom the Software is furnished 
to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included install 
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANT ABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View Dependency Licenses & SBOM on FOSSA

© Alicia Sykes 2023
Licensed under MIT

Thanks for visiting :)