extern/web-check
1
0
Fork 0
mirror of https://github.com/Lissy93/web-check.git synced 2024-11-25 09:43:13 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #48 from Lissy93/FEAT/vercel-support

Browse Source 
[FEAT] Vercel support
This commit is contained in:
Alicia Sykes 2023-09-03 17:01:06 +01:00 committed by GitHub
commit 32d5962dc3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
41 changed files with 2870 additions and 2683 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

54
.github/README.md vendored
View File

@ -19,11 +19,12 @@
- [Mirror](#mirror)
- [Features](#features)
- **[Usage](#usage)**
- [Developer Setup](#developing)
- [Deploying, Option#1: Netlify](#deploying---option-1-netlify)
- [Deploying, Option#2: Docker](#deploying---option-2-docker)
- [Deploying, Option#3: Source](#deploying---option-3-from-source)
- [Deploying, Option#2: Vercel](#deploying---option-2-vercel)
- [Deploying, Option#3: Docker](#deploying---option-3-docker)
- [Deploying, Option#4: Source](#deploying---option-4-from-source)
- [Configuration Options](#configuring)
- [Developer Setup](#developing)
- **[Community](#community)**
- [Contributing](#contributing)
- [Bugs](#reporting-bugs)
@ -56,11 +57,6 @@ A hosted version can be accessed at: **[web-check.as93.net](https://web-check.as
### Mirror
The source for this repo is mirrored to CodeBerg, available at: **[codeberg.org/alicia/web-check](https://codeberg.org/alicia/web-check)**
### Motivation
Often when you're looking into a website, there's several things you always initially check.
None of this is hard to find with a series of basic curl commands and NPMAP plus a combination of online tools. But it's just so much easier to have everything done all at once, presented clearly and visible in one place :)
### Features
<details>
@ -738,35 +734,35 @@ _Note that not all checks will work for all sites. Sometimes it's not possible t
## Usage
### Developing
1. Clone the repo, `git clone git@github.com:Lissy93/web-check.git`
2. Cd into it, `cd web-check`
3. Install dependencies: `yarn`
4. Start the dev server, with `yarn dev`
You'll need [Node.js](https://nodejs.org/en) (V 18.16.1 or later) installed, plus [yarn](https://yarnpkg.com/getting-started/install) as well as [git](https://git-scm.com/).
Some checks also require `chromium`, `traceroute` and `dns` to be installed within your environment. These jobs will just be skipped if those packages aren't present.
### Deploying - Option #1: Netlify
Click the button below, to deploy to Netlify 👇
[![Deploy to Netlify](https://img.shields.io/badge/Deploy-Netlify-%2330c8c9?style=for-the-badge&logo=netlify&labelColor=1e0e41 'Deploy Web-Check to Netlify, via 1-Click Script')](https://app.netlify.com/start/deploy?repository=https://github.com/lissy93/web-check)
### Deploying - Option #2: Docker
### Deploying - Option #2: Vercel
Run `docker run -p 8888:3000 lissy93/web-check`, then open `http://localhost:3000`
Click the button below, to deploy to Vercel 👇
[![Deploy with Vercel](https://img.shields.io/badge/Deploy-Vercel-%23ffffff?style=for-the-badge&logo=vercel&labelColor=1e0e41)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flissy93%2Fweb-check&project-name=web-check&repository-name=web-check-fork&demo-title=Web-Check%20Demo&demo-description=Check%20out%20web-check.xyz%20to%20see%20a%20live%20demo%20of%20this%20application%20running.&demo-url=https%3A%2F%2Fweb-check.xyz&demo-image=https%3A%2F%2Fraw.githubusercontent.com%2FLissy93%2Fweb-check%2Fmaster%2F.github%2Fscreenshots%2Fweb-check-screenshot10.png)
### Deploying - Option #3: Docker
Run `docker run -p 3000:3000 lissy93/web-check`, then open `http://localhost:3000`
<details>
<summary>Docker Options</summary>
You can get the Docker image from:
- DockerHub: [`lissy93/web-check`](https://hub.docker.com/r/lissy93/web-check)
- GHCR: [`ghcr.io/lissy93/web-check`](https://github.com/Lissy93/web-check/pkgs/container/web-check)
- Or build the image yourself by cloning the repo and running `docker build -t web-check .`
### Deploying - Option #3: From Source
</details>
Follow the instructions in the [Developing](#developing) section above, then run `yarn build` && `yarn serve` to build and serve the application.
The following commands will work:
### Deploying - Option #4: From Source
Install the prerequisites listed in the [Developing](#developing) section, then run:
```bash
git clone https://github.com/Lissy93/web-check.git # Grab the code
@ -776,7 +772,6 @@ yarn build # Build the app for production
yarn serve # Start the app (API and GUI)
```
### Configuring
By default, no configuration is needed.
@ -807,6 +802,17 @@ Note that keys that are prefixed with `REACT_APP_` are used client-side, and as
The above can be added into an `.env` file in the projects root, or via the Netlify UI, or by passing directly to the Docker container with the --env flag.
All variables are optional.
### Developing
1. Clone the repo, `git clone git@github.com:Lissy93/web-check.git`
2. Cd into it, `cd web-check`
3. Install dependencies: `yarn`
4. Start the dev server, with `yarn dev`
You'll need [Node.js](https://nodejs.org/en) (V 18.16.1 or later) installed, plus [yarn](https://yarnpkg.com/getting-started/install) as well as [git](https://git-scm.com/).
Some checks also require `chromium`, `traceroute` and `dns` to be installed within your environment. These jobs will just be skipped if those packages aren't present.
---
## Community

69
.gitignore vendored
View File

@ -1,27 +1,60 @@
# keys
# ------------------------
# ENVIRONMENT SETTINGS
# ------------------------
.env
# dependencies
/node_modules
/.pnp
.pnp.js
# ------------------------
# PRODUCTION
# ------------------------
/build/
# logs
# ------------------------
# DEPLOYMENT
# ------------------------
.vercel/
.netlify/
.webpack/
.serverless/
# ------------------------
# DEPENDENCIES
# ------------------------
node_modules/
.yarn/cache/
.yarn/unplugged/
.yarn/build-state.yml
.yarn/install-state.gz
.pnpm/
.pnp.*
# ------------------------
# LOGS
# ------------------------
logs/
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*
# testing
/coverage
# ------------------------
# TESTING
# ------------------------
coverage/
.nyc_output/
# production
/build
# Random AWS and Netlify crap
.netlify
.serverless
.webpack
# OS generated files
# ------------------------
# OS SPECIFIC
# ------------------------
.DS_Store
Thumbs.db
# ------------------------
# EDITORS
# ------------------------
.idea/
.vscode/
*.swp
*.swo

66
api/_common/middleware.js
View File

@ -2,37 +2,81 @@ const normalizeUrl = (url) => {
return url.startsWith('http') ? url : `https://${url}`;
};
const headers = {
'Access-Control-Allow-Origin': process.env.API_CORS_ORIGIN || '*',
'Access-Control-Allow-Credentials': true,
'Content-Type': 'application/json;charset=UTF-8',
};
const commonMiddleware = (handler) => {
return async (event, context, callback) => {
const queryParams = event.queryStringParameters || event.query || {};
// Vercel
const vercelHandler = async (request, response) => {
const queryParams = request.query || {};
const rawUrl = queryParams.url;
if (!rawUrl) {
callback(null, {
statusCode: 500,
body: JSON.stringify({ error: 'No URL specified' }),
});
return response.status(500).json({ error: 'No URL specified' });
}
const url = normalizeUrl(rawUrl);
try {
const response = await handler(url, event, context);
if (response.body && response.statusCode) {
callback(null, response);
const handlerResponse = await handler(url, request);
if (handlerResponse.body && handlerResponse.statusCode) {
response.status(handlerResponse.statusCode).json(handlerResponse.body);
} else {
response.status(200).json(
typeof handlerResponse === 'object' ? handlerResponse : JSON.parse(handlerResponse)
);
}
} catch (error) {
response.status(500).json({ error: error.message });
}
};
// Netlify
const netlifyHandler = async (event, context, callback) => {
const queryParams = event.queryStringParameters || event.query || {};
const rawUrl = queryParams.url;
if (!rawUrl) {
callback(null, {
statusCode: 500,
body: JSON.stringify({ error: 'No URL specified' }),
headers,
});
return;
}
const url = normalizeUrl(rawUrl);
try {
const handlerResponse = await handler(url, event, context);
if (handlerResponse.body && handlerResponse.statusCode) {
callback(null, handlerResponse);
} else {
callback(null, {
statusCode: 200,
body: typeof response === 'object' ? JSON.stringify(response) : response,
body: typeof handlerResponse === 'object' ? JSON.stringify(handlerResponse) : handlerResponse,
headers,
});
}
} catch (error) {
callback(null, {
statusCode: 500,
body: JSON.stringify({ error: error.message }),
headers,
});
}
};
// The format of the handlers varies between platforms
// E.g. Netlify + AWS expect Lambda functions, but Vercel or Node needs standard handler
const platformEnv = (process.env.PLATFORM || '').toUpperCase(); // Has user set platform manually?
const nativeMode = (['VERCEL', 'NODE'].includes(platformEnv) || process.env.VERCEL || process.env.WC_SERVER);
return nativeMode ? vercelHandler : netlifyHandler;
};
module.exports = commonMiddleware;

3
api/archives.js
View File

@ -80,4 +80,5 @@ const getWaybackData = async (url) => {
}
};
exports.handler = middleware(getWaybackData);
module.exports = middleware(getWaybackData);
module.exports.handler = middleware(getWaybackData);

10
api/block-lists.js
View File

@ -94,12 +94,12 @@ const checkDomainAgainstDnsServers = async (domain) => {
return results;
};
exports.handler = middleware(async (url) => {
const handler = middleware(async (url) => {
const domain = new URL(url).hostname;
const results = await checkDomainAgainstDnsServers(domain);
return {
statusCode: 200,
body: JSON.stringify({ blocklists: results })
};
return { blocklists: results };
});
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/carbon.js
View File

@ -48,4 +48,5 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/cookies.js
View File

@ -54,4 +54,5 @@ const handler = async (url) => {
return { headerCookies, clientCookies };
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/dns-server.js
View File

@ -1,7 +1,7 @@
const dns = require('dns');
const dnsPromises = dns.promises;
const axios = require('axios');
const commonMiddleware = require('./_common/middleware');
const middleware = require('./_common/middleware');
const handler = async (url) => {
try {
@ -41,4 +41,5 @@ const handler = async (url) => {
}
};
exports.handler = commonMiddleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/dns.js
View File

@ -51,4 +51,5 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

8
api/dnssec.js
View File

@ -1,7 +1,7 @@
const https = require('https');
const commonMiddleware = require('./_common/middleware'); // Make sure this path is correct
const middleware = require('./_common/middleware'); // Make sure this path is correct
const fetchDNSRecords = async (domain, event, context) => {
const handler = async (domain) => {
const dnsTypes = ['DNSKEY', 'DS', 'RRSIG'];
const records = {};
@ -49,4 +49,6 @@ const fetchDNSRecords = async (domain, event, context) => {
return records;
};
exports.handler = commonMiddleware(fetchDNSRecords);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/features.js
View File

@ -1,7 +1,7 @@
const https = require('https');
const middleware = require('./_common/middleware');
const builtWithHandler = async (url) => {
const handler = async (url) => {
const apiKey = process.env.BUILT_WITH_API_KEY;
if (!url) {
@ -45,4 +45,5 @@ const builtWithHandler = async (url) => {
}
};
exports.handler = middleware(builtWithHandler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/firewall.js
View File

@ -102,4 +102,5 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

4
api/get-ip.js
View File

@ -18,4 +18,6 @@ const handler = async (url) => {
return await lookupAsync(address);
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/headers.js
View File

@ -15,4 +15,5 @@ const handler = async (url, event, context) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

23
api/hsts.js
View File

@ -1,18 +1,15 @@
const https = require('https');
const middleware = require('./_common/middleware');
exports.handler = middleware(async (url, event, context) => {
const handler = async (url, event, context) => {
const errorResponse = (message, statusCode = 500) => {
return {
statusCode: statusCode,
body: JSON.stringify({ error: message }),
};
};
const hstsIncompatible = (message, statusCode = 200) => {
return {
statusCode: statusCode,
body: JSON.stringify({ message, compatible: false }),
};
const hstsIncompatible = (message, compatible = false, hstsHeader = null ) => {
return { message, compatible, hstsHeader };
};
@ -35,14 +32,7 @@ exports.handler = middleware(async (url, event, context) => {
} else if (!preload) {
resolve(hstsIncompatible(`HSTS header does not contain the preload directive.`));
} else {
resolve({
statusCode: 200,
body: JSON.stringify({
message: "Site is compatible with the HSTS preload list!",
compatible: true,
hstsHeader: hstsHeader,
}),
});
resolve(hstsIncompatible(`Site is compatible with the HSTS preload list!`, true, hstsHeader));
}
}
});
@ -53,5 +43,8 @@ exports.handler = middleware(async (url, event, context) => {
req.end();
});
});
};
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/http-security.js
View File

@ -22,4 +22,5 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/legacy-rank.js
View File

@ -66,5 +66,6 @@ return new Promise((resolve, reject) => {
});
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/linked-pages.js
View File

@ -1,7 +1,7 @@
const axios = require('axios');
const cheerio = require('cheerio');
const urlLib = require('url');
const commonMiddleware = require('./_common/middleware');
const middleware = require('./_common/middleware');
const handler = async (url) => {
const response = await axios.get(url);
@ -45,4 +45,5 @@ const handler = async (url) => {
return { internal: internalLinks, external: externalLinks };
};
exports.handler = commonMiddleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/mail-config.js
View File

@ -1,4 +1,4 @@
const commonMiddleware = require('./_common/middleware');
const middleware = require('./_common/middleware');
const dns = require('dns').promises;
const URL = require('url-parse');
@ -72,4 +72,5 @@ const handler = async (url, event, context) => {
}
};
module.exports.handler = commonMiddleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

13
api/ports.js
View File

@ -73,17 +73,12 @@ const handler = async (url, event, context) => {
return errorResponse('The function timed out before completing.');
}
return {
statusCode: 200,
body: JSON.stringify({ openPorts, failedPorts }),
};
return { openPorts, failedPorts };
};
const errorResponse = (message, statusCode = 444) => {
return {
statusCode: statusCode,
body: JSON.stringify({ error: message }),
};
return { error: message };
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

7
api/quality.js
View File

@ -4,10 +4,6 @@ const middleware = require('./_common/middleware');
const handler = async (url, event, context) => {
const apiKey = process.env.GOOGLE_CLOUD_API_KEY;
if (!url) {
throw new Error('URL param is required');
}
if (!apiKey) {
throw new Error('API key (GOOGLE_CLOUD_API_KEY) not set');
}
@ -19,4 +15,5 @@ const handler = async (url, event, context) => {
return response.data;
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

6
api/rank.js
View File

@ -10,7 +10,7 @@ const handler = async (url) => {
{ auth: { username: process.env.TRANCO_USERNAME, password: process.env.TRANCO_API_KEY } }
: {};
const response = await axios.get(
`https://tranco-list.eu/api/ranks/domain/${domain}`, { timeout: 2000 }, auth,
`https://tranco-list.eu/api/ranks/domain/${domain}`, { timeout: 5000 }, auth,
);
if (!response.data || !response.data.ranks || response.data.ranks.length === 0) {
return { skipped: `Skipping, as ${domain} isn't ranked in the top 100 million sites yet.`};
@ -21,4 +21,6 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

4
api/redirects.js
View File

@ -24,4 +24,6 @@ const handler = async (url) => {
};
const middleware = require('./_common/middleware');
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/robots-txt.js
View File

@ -67,4 +67,5 @@ const handler = async function(url) {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/screenshot.js
View File

@ -2,7 +2,7 @@ const puppeteer = require('puppeteer-core');
const chromium = require('chrome-aws-lambda');
const middleware = require('./_common/middleware');
const screenshotHandler = async (targetUrl) => {
const handler = async (targetUrl) => {
if (!targetUrl) {
throw new Error('URL is missing from queryStringParameters');
@ -58,4 +58,5 @@ const screenshotHandler = async (targetUrl) => {
}
};
exports.handler = middleware(screenshotHandler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

7
api/security-txt.js
View File

@ -38,7 +38,7 @@ const isPgpSigned = (result) => {
return false;
};
const securityTxtHandler = async (urlParam) => {
const handler = async (urlParam) => {
let url;
try {
@ -69,8 +69,6 @@ const securityTxtHandler = async (urlParam) => {
return { isPresent: false };
};
exports.handler = middleware(securityTxtHandler);
async function fetchSecurityTxt(baseURL, path) {
return new Promise((resolve, reject) => {
const url = new URL(path, baseURL);
@ -91,3 +89,6 @@ async function fetchSecurityTxt(baseURL, path) {
});
});
}
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

26
api/sitemap.js
View File

@ -1,4 +1,4 @@
const commonMiddleware = require('./_common/middleware');
const middleware = require('./_common/middleware');
const axios = require('axios');
const xml2js = require('xml2js');
@ -25,10 +25,7 @@ const handler = async (url) => {
}
if (!sitemapUrl) {
return {
statusCode: 404,
body: JSON.stringify({ skipped: 'No sitemap found' }),
};
return { skipped: 'No sitemap found' };
}
sitemapRes = await axios.get(sitemapUrl, { timeout: 5000 });
@ -40,25 +37,18 @@ const handler = async (url) => {
const parser = new xml2js.Parser();
const sitemap = await parser.parseStringPromise(sitemapRes.data);
return {
statusCode: 200,
body: JSON.stringify(sitemap),
};
return sitemap;
} catch (error) {
// If error occurs
console.log(error.message);
if (error.code === 'ECONNABORTED') {
return {
statusCode: 500,
body: JSON.stringify({ error: 'Request timed out' }),
};
return { error: 'Request timed out' };
} else {
return {
statusCode: 500,
body: JSON.stringify({ error: error.message }),
};
return { error: error.message };
}
}
};
exports.handler = commonMiddleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

16
api/social-tags.js
View File

@ -1,4 +1,4 @@
const commonMiddleware = require('./_common/middleware');
const middleware = require('./_common/middleware');
const axios = require('axios');
const cheerio = require('cheerio');
@ -50,16 +50,9 @@ const handler = async (url) => {
};
if (Object.keys(metadata).length === 0) {
return {
statusCode: 200,
body: JSON.stringify({ skipped: 'No metadata found' }),
};
return { skipped: 'No metadata found' };
}
return {
statusCode: 200,
body: JSON.stringify(metadata),
};
return metadata;
} catch (error) {
return {
statusCode: 500,
@ -68,4 +61,5 @@ const handler = async (url) => {
}
};
exports.handler = commonMiddleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/ssl.js
View File

@ -1,7 +1,7 @@
const tls = require('tls');
const middleware = require('./_common/middleware');
const fetchSiteCertificateHandler = async (urlString) => {
const handler = async (urlString) => {
try {
const parsedUrl = new URL(urlString);
const options = {
@ -40,4 +40,5 @@ const fetchSiteCertificateHandler = async (urlString) => {
}
};
exports.handler = middleware(fetchSiteCertificateHandler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/status.js
View File

@ -2,7 +2,7 @@ const https = require('https');
const { performance, PerformanceObserver } = require('perf_hooks');
const middleware = require('./_common/middleware');
const checkURLHandler = async (url) => {
const handler = async (url) => {
if (!url) {
throw new Error('You must provide a URL query parameter!');
}
@ -55,4 +55,5 @@ const checkURLHandler = async (url) => {
}
};
exports.handler = middleware(checkURLHandler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/tech-stack.js
View File

@ -1,7 +1,7 @@
const Wappalyzer = require('wappalyzer');
const middleware = require('./_common/middleware');
const analyzeSiteTechnologies = async (url) => {
const handler = async (url) => {
const options = {};
const wappalyzer = new Wappalyzer(options);
@ -27,4 +27,5 @@ const analyzeSiteTechnologies = async (url) => {
}
};
exports.handler = middleware(analyzeSiteTechnologies);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

12
api/threats.js
View File

@ -5,6 +5,9 @@ const middleware = require('./_common/middleware');
const getGoogleSafeBrowsingResult = async (url) => {
try {
const apiKey = process.env.GOOGLE_CLOUD_API_KEY;
if (!apiKey) {
return { error: 'GOOGLE_CLOUD_API_KEY is required for the Google Safe Browsing check' };
}
const apiEndpoint = `https://safebrowsing.googleapis.com/v4/threatMatches:find?key=${apiKey}`;
const requestBody = {
@ -63,11 +66,15 @@ const getPhishTankResult = async (url) => {
}
const getCloudmersiveResult = async (url) => {
const apiKey = process.env.CLOUDMERSIVE_API_KEY;
if (!apiKey) {
return { error: 'CLOUDMERSIVE_API_KEY is required for the Cloudmersive check' };
}
try {
const endpoint = 'https://api.cloudmersive.com/virus/scan/website';
const headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Apikey': process.env.CLOUDMERSIVE_API_KEY,
'Apikey': apiKey,
};
const data = `Url=${encodeURIComponent(url)}`;
const response = await axios.post(endpoint, data, { headers });
@ -92,4 +99,5 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/tls.js
View File

@ -25,4 +25,5 @@ const handler = async (url) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

5
api/trace-route.js
View File

@ -2,7 +2,7 @@ const traceroute = require('traceroute');
const url = require('url');
const middleware = require('./_common/middleware');
const executeTraceroute = async (urlString, context) => {
const handler = async (urlString, context) => {
// Parse the URL and get the hostname
const urlObject = url.parse(urlString);
const host = urlObject.hostname;
@ -28,4 +28,5 @@ const executeTraceroute = async (urlString, context) => {
};
};
exports.handler = middleware(executeTraceroute);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

3
api/txt-records.js
View File

@ -29,4 +29,5 @@ const handler = async (url, event, context) => {
}
};
exports.handler = middleware(handler);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

6
api/whois.js
View File

@ -83,7 +83,7 @@ const fetchFromMyAPI = async (hostname) => {
}
};
const fetchWhoisData = async (url) => {
const handler = async (url) => {
if (!url.startsWith('http://') && !url.startsWith('https://')) {
url = 'http://' + url;
}
@ -106,4 +106,6 @@ const fetchWhoisData = async (url) => {
};
};
exports.handler = middleware(fetchWhoisData);
module.exports = middleware(handler);
module.exports.handler = middleware(handler);

6
package.json
View File

@ -20,7 +20,9 @@
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
"dev:vercel": "PLATFORM='vercel' npx vercel dev",
"dev:netlify": "PLATFORM='netlify' npx netlify dev",
"dev:node": "npx concurrently --names \"frontend,backend\" \"REACT_APP_API_ENDPOINT='http://localhost:3001/api' npx react-scripts start\" \"PLATFORM='node' DISABLE_GUI='true' PORT='3001' API_CORS_ORIGIN='*' npx nodemon server\""
},
"dependencies": {
"@netlify/functions": "^1.6.0",
@ -35,12 +37,12 @@
"@types/react-router-dom": "^5.3.3",
"@types/react-simple-maps": "^3.0.0",
"@types/styled-components": "^5.1.26",
"aws-serverless-express": "^3.4.0",
"axios": "^1.4.0",
"cheerio": "^1.0.0-rc.12",
"chrome-aws-lambda": "^10.1.0",
"chromium": "^3.0.3",
"connect-history-api-fallback": "^2.0.0",
"cors": "^2.8.5",
"csv-parser": "^3.0.0",
"dotenv": "^16.3.1",
"flatted": "^3.2.7",

39
public/error.html Normal file
View File

@ -0,0 +1,39 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Web-Check</title>
</head>
<body>
<section>
<a class="t" href="/"><h1><img src="https://i.ibb.co/q1gZN2p/web-check-logo.png" width="48" />Web-Check</h1></a>
<p class="moji">😪</p>
<p>There was an error finding this route.</p>
<span>Docs and Source: <a href="https://github.com/lissy93/web-check">github.com/lissy93/web-check</a></span>
</section>
<style>
body { background: #141d2b; color: #fff; }
h1 {
color: #9fef00; text-shadow: #0f1620 2px 2px 0px;
font-size: 3rem; margin: 1rem auto; flex-wrap: wrap;
display: flex; align-items: flex-start; gap: 1rem;
}
section {
display: flex; flex-direction: column; align-items: center; margin: 1rem; gap: 0.5rem;
background: #1a2332; box-shadow: #0f1620 4px 4px 0px; border-radius: 8px; padding: 1rem;
max-width: 800px; margin: 1rem auto;
}
p { font-size: 1.2rem; }
a { color: #9fef00; font-family: monospace; }
a.t { text-decoration: none; margin: 0;}
span { opacity: 0.8; font-size: 0.85rem; }
h1 {
font-family: PTMono, Ubuntu, Fira Sans, Helvetica, sans-serif;
}
p, span, a, section, div { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }
code { color: #9fef00cc;}
.moji { font-size: 8rem; margin: 0; }
</style>
</body>
</html>

37
public/placeholder.html Normal file
View File

@ -0,0 +1,37 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Web-Check</title>
</head>
<body>
<section>
<a class="t" href="/"><h1><img src="https://i.ibb.co/q1gZN2p/web-check-logo.png" width="48" />Web-Check</h1></a>
<p><!-- CONTENT --></p>
<span>Docs and Source: <a href="https://github.com/lissy93/web-check">github.com/lissy93/web-check</a></span>
</section>
<style>
body { background: #141d2b; color: #fff; }
h1 {
color: #9fef00; text-shadow: #0f1620 2px 2px 0px;
font-size: 3rem; margin: 1rem auto; flex-wrap: wrap;
display: flex; align-items: flex-start; gap: 1rem;
}
section {
display: flex; flex-direction: column; align-items: center; margin: 1rem; gap: 0.5rem;
background: #1a2332; box-shadow: #0f1620 4px 4px 0px; border-radius: 8px; padding: 1rem;
max-width: 800px; margin: 1rem auto;
}
p { font-size: 1.2rem; }
a { color: #9fef00; font-family: monospace; }
a.t { text-decoration: none; margin: 0;}
span { opacity: 0.8; font-size: 0.85rem; }
h1 {
font-family: PTMono, Ubuntu, Fira Sans, Helvetica, sans-serif;
}
p, span, a, section, div { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }
code { color: #9fef00cc;}
</style>
</body>
</html>

188
server.js
View File

@ -1,116 +1,148 @@
const express = require('express');
const awsServerlessExpress = require('aws-serverless-express');
const fs = require('fs');
const path = require('path');
const cors = require('cors');
const historyApiFallback = require('connect-history-api-fallback');
require('dotenv').config();
const app = express();
const port = process.env.PORT || 3000; // The port to run the server on
const API_DIR = '/api'; // Name of the dir containing the lambda functions
const dirPath = path.join(__dirname, API_DIR); // Path to the lambda functions dir
const guiPath = path.join(__dirname, 'build');
const placeholderFilePath = path.join(__dirname, 'public', 'placeholder.html');
const handlers = {}; // Will store list of API endpoints
process.env.WC_SERVER = 'true'; // Tells middleware to return in non-lambda mode
// Execute the lambda function
const executeHandler = async (handler, req) => {
return new Promise((resolve, reject) => {
const callback = (err, response) => err ? reject(err) : resolve(response);
const promise = handler(req, {}, callback);
// Enable CORS
app.use(cors({
origin: process.env.API_CORS_ORIGIN || '*',
}));
if (promise && typeof promise.then === 'function') {
promise.then(resolve).catch(reject);
}
});
};
// Array of all the lambda function file names
const fileNames = fs.readdirSync(dirPath, { withFileTypes: true })
// Read and register each API function as an Express routes
fs.readdirSync(dirPath, { withFileTypes: true })
.filter(dirent => dirent.isFile() && dirent.name.endsWith('.js'))
.map(dirent => dirent.name);
.forEach(dirent => {
const routeName = dirent.name.split('.')[0];
const route = `${API_DIR}/${routeName}`;
const handler = require(path.join(dirPath, dirent.name));
handlers[route] = handler;
const handlers = {};
app.get(route, async (req, res) => {
try {
await handler(req, res);
} catch (err) {
res.status(500).json({ error: err.message });
}
});
});
fileNames.forEach(file => {
const routeName = file.split('.')[0];
const route = `${API_DIR}/${routeName}`;
const handler = require(path.join(dirPath, file)).handler;
handlers[route] = handler;
// Create a single API endpoint to execute all lambda functions
app.get('/api', async (req, res) => {
const results = {};
const { url } = req.query;
const maxExecutionTime = process.env.API_TIMEOUT_LIMIT || 20000;
app.get(route, async (req, res) => {
try {
const { statusCode = 200, body = '' } = await executeHandler(handler, req);
res.status(statusCode).json(JSON.parse(body));
} catch (err) {
res.status(500).json({ error: err.message });
}
const executeHandler = async (handler, req, res) => {
return new Promise(async (resolve, reject) => {
try {
const mockRes = {
status: (statusCode) => mockRes,
json: (body) => resolve({ body }),
};
await handler({ ...req, query: { url } }, mockRes);
} catch (err) {
reject(err);
}
});
};
const timeout = (ms, jobName = null) => {
return new Promise((_, reject) => {
setTimeout(() => {
reject(new Error(
`Timed out after ${ms/1000} seconds${jobName ? `, when executing ${jobName}` : ''}`
));
}, ms);
});
};
const handlerPromises = Object.entries(handlers).map(async ([route, handler]) => {
const routeName = route.replace(`${API_DIR}/`, '');
try {
const result = await Promise.race([
executeHandler(handler, req, res),
timeout(maxExecutionTime, routeName)
]);
results[routeName] = result.body;
} catch (err) {
results[routeName] = { error: err.message };
}
});
await Promise.all(handlerPromises);
res.json(results);
});
});
const timeout = (ms, jobName = null) => {
return new Promise((_, reject) => {
setTimeout(() => {
reject(new Error(`Timed out after the ${ms/1000} second limit${jobName ? `, when executing the ${jobName} task` : ''}`));
}, ms);
});
}
app.get('/api', async (req, res) => {
const results = {};
const { url } = req.query;
const maxExecutionTime = process.env.API_TIMEOUT_LIMIT || 15000;
const handlerPromises = Object.entries(handlers).map(async ([route, handler]) => {
const routeName = route.replace(`${API_DIR}/`, '');
try {
const result = await Promise.race([
executeHandler(handler, { query: { url } }),
timeout(maxExecutionTime, routeName)
]);
results[routeName] = JSON.parse((result || {}).body);
} catch (err) {
results[routeName] = { error: err.message };
}
});
await Promise.all(handlerPromises);
res.json(results);
});
// Handle SPA routing
app.use(historyApiFallback({
rewrites: [
{ from: /^\/api\/.*$/, to: function(context) { return context.parsedUrl.path; } },
{ from: /^\/api\/.*$/, to: (context) => context.parsedUrl.path },
]
}));
// Serve up the GUI - if build dir exists, and GUI feature enabled
if (process.env.DISABLE_GUI && process.env.DISABLE_GUI !== 'false') {
app.get('*', (req, res) => {
res.status(500).send(
'Welcome to Web-Check!<br />Access the API endpoints at '
+'<a href="/api"><code>/api</code></a>'
);
app.get('*', async (req, res) => {
const placeholderContent = await fs.promises.readFile(placeholderFilePath, 'utf-8');
const htmlContent = placeholderContent.replace(
'<!-- CONTENT -->',
'Web-Check API is up and running!<br />Access the endpoints at '
+'<a href="/api"><code>/api</code></a>'
);
res.status(500).send(htmlContent);
});
} else if (!fs.existsSync(guiPath)) {
app.get('*', (req, res) => {
res.status(500).send(
'Welcome to Web-Check!<br />Looks like the GUI app has not yet been compiled, '
+'run <code>yarn build</code> to continue, then restart the server.'
app.get('*', async (req, res) => {
const placeholderContent = await fs.promises.readFile(placeholderFilePath, 'utf-8');
const htmlContent = placeholderContent.replace(
'<!-- CONTENT -->',
'Looks like the GUI app has not yet been compiled.<br /> ' +
'Run <code>yarn build</code> to continue, then restart the server.'
);
res.status(500).send(htmlContent);
});
} else { // GUI enabled, and build files present, let's go!!
app.use(express.static(guiPath));
}
// Create serverless express server
const port = process.env.PORT || 3000;
const server = awsServerlessExpress.createServer(app).listen(port, () => {
console.log(`Server is running on port ${port}`);
app.use((req, res, next) => {
res.status(404).sendFile(path.join(__dirname, 'public', 'error.html'));
});
exports.handler = (event, context) => {
awsServerlessExpress.proxy(server, event, context);
// Print nice welcome message to user
const printMessage = () => {
console.log(
`\x1b[36m\n` +
' __ __ _ ___ _ _ \n' +
' \\ \\ / /__| |__ ___ / __| |_ ___ __| |__\n' +
' \\ \\/\\/ / -_) \'_ \\___| (__| \' \\/ -_) _| / /\n' +
' \\_/\\_/\\___|_.__/ \\___|_||_\\___\\__|_\\_\\\n' +
`\x1b[0m\n`,
`\x1b[1m\x1b[32m🚀 Web-Check is up and running at http://localhost:${port} \x1b[0m\n\n`,
`\x1b[2m\x1b[36m🛟 For documentation and support, visit the GitHub repo: ` +
`https://github.com/lissy93/web-check \n`,
`💖 Found Web-Check useful? Consider sponsoring us on GitHub ` +
`to help fund maintenance & development.\x1b[0m`
);
};
// Create server
app.listen(port, () => {
printMessage();
});

4874
yarn.lock
View File

File diff suppressed because it is too large Load Diff