feat: Implement push-based external endpoints (#724)

* refactor: Move SSH outside of endpoint.go
* refactor: Use pointers for Alert receivers
* feat: Implement push-based external endpoints
* Fix failing tests
* Validate external endpoints on start
* Add tests for external endpoints
* refactor some error equality checks
* Improve docs and refactor some code
* Fix UI-related issues with external endpoints
This commit is contained in:
TwiN 2024-04-08 21:00:40 -04:00 committed by GitHub
parent cacfbc0185
commit f54c45e20e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
28 changed files with 808 additions and 189 deletions

176
README.md
View File

@ -9,7 +9,6 @@
[![Follow TwiN](https://img.shields.io/github/followers/TwiN?label=Follow&style=social)](https://github.com/TwiN)
Gatus is a developer-oriented health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS
queries as well as evaluate the result of said queries by using a list of conditions on values like the status code,
the response time, the certificate expiration, the body and many others. The icing on top is that each of these health
@ -46,6 +45,8 @@ Have any feedback or questions? [Create a discussion](https://github.com/TwiN/ga
- [Features](#features)
- [Usage](#usage)
- [Configuration](#configuration)
- [Endpoints](#endpoints)
- [External Endpoints](#external-endpoints)
- [Conditions](#conditions)
- [Placeholders](#placeholders)
- [Functions](#functions)
@ -137,6 +138,7 @@ if no traffic makes it to your applications. This puts you in a situation where
that will notify you about the degradation of your services rather than you reassuring them that you're working on
fixing the issue before they even know about it.
## Features
The main features of Gatus are:
@ -151,6 +153,7 @@ The main features of Gatus are:
![Gatus dashboard conditions](.github/assets/dashboard-conditions.png)
## Usage
<details>
@ -209,40 +212,13 @@ If you want to test it locally, see [Docker](#docker).
## Configuration
| Parameter | Description | Default |
|:------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------|
|:-----------------------------|:-------------------------------------------------------------------------------------------------------------------------------------|:---------------------------|
| `debug` | Whether to enable debug logs. | `false` |
| `metrics` | Whether to expose metrics at /metrics. | `false` |
| `storage` | [Storage configuration](#storage) | `{}` |
| `endpoints` | List of endpoints to monitor. | Required `[]` |
| `endpoints[].enabled` | Whether to monitor the endpoint. | `true` |
| `endpoints[].name` | Name of the endpoint. Can be anything. | Required `""` |
| `endpoints[].group` | Group name. Used to group multiple endpoints together on the dashboard. <br />See [Endpoint groups](#endpoint-groups). | `""` |
| `endpoints[].url` | URL to send the request to. | Required `""` |
| `endpoints[].method` | Request method. | `GET` |
| `endpoints[].conditions` | Conditions used to determine the health of the endpoint. <br />See [Conditions](#conditions). | `[]` |
| `endpoints[].interval` | Duration to wait between every status check. | `60s` |
| `endpoints[].graphql` | Whether to wrap the body in a query param (`{"query":"$body"}`). | `false` |
| `endpoints[].body` | Request body. | `""` |
| `endpoints[].headers` | Request headers. | `{}` |
| `endpoints[].dns` | Configuration for an endpoint of type DNS. <br />See [Monitoring an endpoint using DNS queries](#monitoring-an-endpoint-using-dns-queries). | `""` |
| `endpoints[].dns.query-type` | Query type (e.g. MX) | `""` |
| `endpoints[].dns.query-name` | Query name (e.g. example.com) | `""` |
| `endpoints[].ssh` | Configuration for an endpoint of type SSH. <br />See [Monitoring an endpoint using SSH](#monitoring-an-endpoint-using-ssh). | `""` |
| `endpoints[].ssh.username` | SSH username (e.g. example) | Required `""` |
| `endpoints[].ssh.password` | SSH password (e.g. password) | Required `""` |
| `endpoints[].alerts[].type` | Type of alert. <br />See [Alerting](#alerting) for all valid types. | Required `""` |
| `endpoints[].alerts[].enabled` | Whether to enable the alert. | `true` |
| `endpoints[].alerts[].failure-threshold` | Number of failures in a row needed before triggering the alert. | `3` |
| `endpoints[].alerts[].success-threshold` | Number of successes in a row before an ongoing incident is marked as resolved. | `2` |
| `endpoints[].alerts[].send-on-resolved` | Whether to send a notification once a triggered alert is marked as resolved. | `false` |
| `endpoints[].alerts[].description` | Description of the alert. Will be included in the alert sent. | `""` |
| `endpoints[].client` | [Client configuration](#client-configuration). | `{}` |
| `endpoints[].ui` | UI configuration at the endpoint level. | `{}` |
| `endpoints[].ui.hide-hostname` | Whether to hide the hostname in the result. | `false` |
| `endpoints[].ui.hide-url` | Whether to ensure the URL is not displayed in the results. Useful if the URL contains a token. | `false` |
| `endpoints[].ui.dont-resolve-failed-conditions` | Whether to resolve failed conditions for the UI. | `false` |
| `endpoints[].ui.badge.reponse-time` | List of response time thresholds. Each time a threshold is reached, the badge has a different color. | `[50, 200, 300, 500, 750]` |
| `metrics` | Whether to expose metrics at `/metrics`. | `false` |
| `storage` | [Storage configuration](#storage). | `{}` |
| `alerting` | [Alerting configuration](#alerting). | `{}` |
| `endpoints` | [Endpoints configuration](#endpoints). | Required `[]` |
| `external-endpoints` | [External Endpoints configuration](#external-endpoints). | `[]` |
| `security` | [Security configuration](#security). | `{}` |
| `disable-monitoring-lock` | Whether to [disable the monitoring lock](#disable-monitoring-lock). | `false` |
| `skip-invalid-config-update` | Whether to ignore invalid configuration update. <br />See [Reloading configuration on the fly](#reloading-configuration-on-the-fly). | `false` |
@ -264,6 +240,79 @@ If you want to test it locally, see [Docker](#docker).
| `maintenance` | [Maintenance configuration](#maintenance). | `{}` |
### Endpoints
Endpoints are URLs, applications, or services that you want to monitor. Each endpoint has a list of conditions that are
evaluated on an interval that you define. If any condition fails, the endpoint is considered as unhealthy.
You can then configure alerts to be triggered when an endpoint is unhealthy once a certain threshold is reached.
| Parameter | Description | Default |
|:------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------|
| `endpoints` | List of endpoints to monitor. | Required `[]` |
| `endpoints[].enabled` | Whether to monitor the endpoint. | `true` |
| `endpoints[].name` | Name of the endpoint. Can be anything. | Required `""` |
| `endpoints[].group` | Group name. Used to group multiple endpoints together on the dashboard. <br />See [Endpoint groups](#endpoint-groups). | `""` |
| `endpoints[].url` | URL to send the request to. | Required `""` |
| `endpoints[].method` | Request method. | `GET` |
| `endpoints[].conditions` | Conditions used to determine the health of the endpoint. <br />See [Conditions](#conditions). | `[]` |
| `endpoints[].interval` | Duration to wait between every status check. | `60s` |
| `endpoints[].graphql` | Whether to wrap the body in a query param (`{"query":"$body"}`). | `false` |
| `endpoints[].body` | Request body. | `""` |
| `endpoints[].headers` | Request headers. | `{}` |
| `endpoints[].dns` | Configuration for an endpoint of type DNS. <br />See [Monitoring an endpoint using DNS queries](#monitoring-an-endpoint-using-dns-queries). | `""` |
| `endpoints[].dns.query-type` | Query type (e.g. MX). | `""` |
| `endpoints[].dns.query-name` | Query name (e.g. example.com). | `""` |
| `endpoints[].ssh` | Configuration for an endpoint of type SSH. <br />See [Monitoring an endpoint using SSH](#monitoring-an-endpoint-using-ssh). | `""` |
| `endpoints[].ssh.username` | SSH username (e.g. example). | Required `""` |
| `endpoints[].ssh.password` | SSH password (e.g. password). | Required `""` |
| `endpoints[].alerts` | List of all alerts for a given endpoint. <br />See [Alerting](#alerting). | `[]` |
| `endpoints[].client` | [Client configuration](#client-configuration). | `{}` |
| `endpoints[].ui` | UI configuration at the endpoint level. | `{}` |
| `endpoints[].ui.hide-hostname` | Whether to hide the hostname in the result. | `false` |
| `endpoints[].ui.hide-url` | Whether to ensure the URL is not displayed in the results. Useful if the URL contains a token. | `false` |
| `endpoints[].ui.dont-resolve-failed-conditions` | Whether to resolve failed conditions for the UI. | `false` |
| `endpoints[].ui.badge.reponse-time` | List of response time thresholds. Each time a threshold is reached, the badge has a different color. | `[50, 200, 300, 500, 750]` |
### External Endpoints
Unlike regular endpoints, external endpoints are not monitored by Gatus, but they are instead pushed programmatically.
This allows you to monitor anything you want, even when what you want to check lives in an environment that would not normally be accessible by Gatus.
For instance:
- You can create your own agent that lives in a private network and pushes the status of your services to a publicly-exposed Gatus instance
- You can monitor services that are not supported by Gatus
- You can implement your own monitoring system while using Gatus as the dashboard
| Parameter | Description | Default |
|:-------------------------------|:-----------------------------------------------------------------------------------------------------------------------|:--------------|
| `external-endpoints` | List of endpoints to monitor. | `[]` |
| `external-endpoints[].enabled` | Whether to monitor the endpoint. | `true` |
| `external-endpoints[].name` | Name of the endpoint. Can be anything. | Required `""` |
| `external-endpoints[].group` | Group name. Used to group multiple endpoints together on the dashboard. <br />See [Endpoint groups](#endpoint-groups). | `""` |
| `external-endpoints[].token` | Bearer token required to push status to. | Required `""` |
| `external-endpoints[].alerts` | List of all alerts for a given endpoint. <br />See [Alerting](#alerting). | `[]` |
Example:
```yaml
external-endpoints:
- name: ext-ep-test
group: core
token: "potato"
alerts:
- type: discord
description: "healthcheck failed"
send-on-resolved: true
```
To push the status of an external endpoint, the request would have to look like this:
```
POST /api/v1/endpoints/{key}/external?success={success}
```
Where:
- `{key}` has the pattern `<GROUP_NAME>_<ENDPOINT_NAME>` in which both variables have ` `, `/`, `_`, `,` and `.` replaced by `-`.
- Using the example configuration above, the key would be `core_ext-ep-test`.
- `{success}` is a boolean (`true` or `false`) value indicating whether the health check was successful or not.
### Conditions
Here are some examples of conditions you can use:
@ -355,7 +404,7 @@ In order to support a wide range of environments, each monitored endpoint has a
the client used to send the request.
| Parameter | Description | Default |
| :------------------------------------- | :-------------------------------------------------------------------------- | :-------------- |
|:---------------------------------------|:----------------------------------------------------------------------------|:----------------|
| `client.insecure` | Whether to skip verifying the server's certificate chain and host name. | `false` |
| `client.ignore-redirect` | Whether to ignore redirects (true) or follow them (false, default). | `false` |
| `client.timeout` | Duration before timing out. | `10s` |
@ -371,7 +420,7 @@ the client used to send the request.
| `client.network` | The network to use for ICMP endpoint client (`ip`, `ip4` or `ip6`). | `"ip"` |
> 📝 Some of these parameters are ignored based on the type of endpoint. For instance, there's no certificate involved
in ICMP requests (ping), therefore, setting `client.insecure` to `true` for an endpoint of that type will not do anything.
> in ICMP requests (ping), therefore, setting `client.insecure` to `true` for an endpoint of that type will not do anything.
This default configuration is as follows:
@ -441,12 +490,38 @@ endpoints:
> 📝 Note that Gatus will use the [gcloud default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) within its environment to generate the token.
### Alerting
Gatus supports multiple alerting providers, such as Slack and PagerDuty, and supports different alerts for each
individual endpoints with configurable descriptions and thresholds.
Alerts are configured at the endpoint level like so:
| Parameter | Description | Default |
|:-----------------------------|:-------------------------------------------------------------------------------|:--------------|
| `alerts` | List of all alerts for a given endpoint. | `[]` |
| `alerts[].type` | Type of alert. <br />See table below for all valid types. | Required `""` |
| `alerts[].enabled` | Whether to enable the alert. | `true` |
| `alerts[].failure-threshold` | Number of failures in a row needed before triggering the alert. | `3` |
| `alerts[].success-threshold` | Number of successes in a row before an ongoing incident is marked as resolved. | `2` |
| `alerts[].send-on-resolved` | Whether to send a notification once a triggered alert is marked as resolved. | `false` |
| `alerts[].description` | Description of the alert. Will be included in the alert sent. | `""` |
Here's an example of what an alert configuration might look like at the endpoint level:
```yaml
endpoints:
- name: example
url: "https://example.org"
conditions:
- "[STATUS] == 200"
alerts:
- type: slack
description: "healthcheck failed"
send-on-resolved: true
```
> 📝 If an alerting provider is not properly configured, all alerts configured with the provider's type will be
ignored.
> ignored.
| Parameter | Description | Default |
|:--------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------|:--------|
@ -473,7 +548,7 @@ ignored.
#### Configuring Discord alerts
| Parameter | Description | Default |
|:-------------------------------------------|:-------------------------------------------------------------------------------------------|:--------------|
|:-------------------------------------------|:-------------------------------------------------------------------------------------------|:------------------------------------|
| `alerting.discord` | Configuration for alerts of type `discord` | `{}` |
| `alerting.discord.webhook-url` | Discord Webhook URL | Required `""` |
| `alerting.discord.title` | Title of the notification | `":helmet_with_white_cross: Gatus"` |
@ -600,6 +675,7 @@ endpoints:
![GitHub alert](.github/assets/github-alerts.png)
#### Configuring GitLab alerts
| Parameter | Description | Default |
|:------------------------------------|:----------------------------------------------------------------------------------------------------------------|:--------------|
@ -995,6 +1071,7 @@ endpoints:
description: "healthcheck failed"
```
#### Configuring Slack alerts
| Parameter | Description | Default |
|:------------------------------------------|:-------------------------------------------------------------------------------------------|:--------------|
@ -1405,7 +1482,7 @@ security:
```
> ⚠ Make sure to carefully select to cost of the bcrypt hash. The higher the cost, the longer it takes to compute the hash,
and basic auth verifies the password against the hash on every request. As of 2023-01-06, I suggest a cost of 9.
> and basic auth verifies the password against the hash on every request. As of 2023-01-06, I suggest a cost of 9.
#### OIDC
@ -1433,6 +1510,7 @@ security:
Confused? Read [Securing Gatus with OIDC using Auth0](https://twin.sh/articles/56/securing-gatus-with-oidc-using-auth0).
### TLS Encryption
Gatus supports basic encryption with TLS. To enable this, certificate files in PEM format have to be provided.
@ -1445,6 +1523,7 @@ web:
private-key-file: "private.key"
```
### Metrics
To enable metrics, you must set `metrics` to `true`. Doing so will expose Prometheus-friendly metrics at the `/metrics`
endpoint on the same port your application is configured to run on (`web.port`).
@ -1653,8 +1732,9 @@ Placeholders `[STATUS]` and `[BODY]` as well as the fields `endpoints[].body`, `
This works for applications such as databases (Postgres, MySQL, etc.) and caches (Redis, Memcached, etc.).
> 📝 `[CONNECTED] == true` does not guarantee that the endpoint itself is healthy - it only guarantees that there's
something at the given address listening to the given port, and that a connection to that address was successfully
established.
> something at the given address listening to the given port, and that a connection to that address was successfully
> established.
### Monitoring a UDP endpoint
By prefixing `endpoints[].url` with `udp:\\`, you can monitor UDP endpoints at a very basic level:
@ -1672,6 +1752,7 @@ Placeholders `[STATUS]` and `[BODY]` as well as the fields `endpoints[].body`, `
This works for UDP based application.
### Monitoring a SCTP endpoint
By prefixing `endpoints[].url` with `sctp:\\`, you can monitor Stream Control Transmission Protocol (SCTP) endpoints at a very basic level:
@ -1688,6 +1769,7 @@ Placeholders `[STATUS]` and `[BODY]` as well as the fields `endpoints[].body`, `
This works for SCTP based application.
### Monitoring a WebSocket endpoint
By prefixing `endpoints[].url` with `ws://` or `wss://`, you can monitor WebSocket endpoints at a very basic level:
@ -1704,6 +1786,7 @@ endpoints:
The `[BODY]` placeholder contains the output of the query, and `[CONNECTED]`
shows whether the connection was successfully established.
### Monitoring an endpoint using ICMP
By prefixing `endpoints[].url` with `icmp:\\`, you can monitor endpoints at a very basic level using ICMP, or more
commonly known as "ping" or "echo":
@ -1722,6 +1805,7 @@ You can specify a domain prefixed by `icmp://`, or an IP address prefixed by `ic
If you run Gatus on Linux, please read the Linux section on https://github.com/prometheus-community/pro-bing#linux
if you encounter any problems.
### Monitoring an endpoint using DNS queries
Defining a `dns` configuration in an endpoint will automatically mark said endpoint as an endpoint of type DNS:
```yaml
@ -1741,6 +1825,7 @@ There are two placeholders that can be used in the conditions for endpoints of t
- The placeholder `[DNS_RCODE]` resolves to the name associated to the response code returned by the query, such as
`NOERROR`, `FORMERR`, `SERVFAIL`, `NXDOMAIN`, etc.
### Monitoring an endpoint using SSH
You can monitor endpoints using SSH by prefixing `endpoints[].url` with `ssh:\\`:
```yaml
@ -1764,6 +1849,7 @@ The following placeholders are supported for endpoints of type SSH:
- `[CONNECTED]` resolves to `true` if the SSH connection was successful, `false` otherwise
- `[STATUS]` resolves the exit code of the command executed on the remote server (e.g. `0` for success)
### Monitoring an endpoint using STARTTLS
If you have an email server that you want to ensure there are no problems with, monitoring it through STARTTLS
will serve as a good initial indicator:
@ -1779,6 +1865,7 @@ endpoints:
- "[CERTIFICATE_EXPIRATION] > 48h"
```
### Monitoring an endpoint using TLS
Monitoring endpoints using SSL/TLS encryption, such as LDAP over TLS, can help detect certificate expiration:
```yaml
@ -1808,9 +1895,9 @@ endpoints:
```
> ⚠ The usage of the `[DOMAIN_EXPIRATION]` placeholder requires Gatus to send a request to the official IANA WHOIS service [through a library](https://github.com/TwiN/whois)
and in some cases, a secondary request to a TLD-specific WHOIS server (e.g. `whois.nic.sh`).
To prevent the WHOIS service from throttling your IP address if you send too many requests, Gatus will prevent you from
using the `[DOMAIN_EXPIRATION]` placeholder on an endpoint with an interval of less than `5m`.
> and in some cases, a secondary request to a TLD-specific WHOIS server (e.g. `whois.nic.sh`).
> To prevent the WHOIS service from throttling your IP address if you send too many requests, Gatus will prevent you from
> using the `[DOMAIN_EXPIRATION]` placeholder on an endpoint with an interval of less than `5m`.
### disable-monitoring-lock
@ -1957,6 +2044,7 @@ endpoints:
```
</details>
### Proxy client configuration
You can configure a proxy for the client to use by setting the `proxy-url` parameter in the client configuration.
@ -1971,6 +2059,7 @@ endpoints:
- "[STATUS] == 200"
```
### How to fix 431 Request Header Fields Too Large error
Depending on where your environment is deployed and what kind of middleware or reverse proxy sits in front of Gatus,
you may run into this issue. This could be because the request headers are too large, e.g. big cookies.
@ -2113,5 +2202,6 @@ You can download Gatus as a binary using the following command:
go install github.com/TwiN/gatus/v5@latest
```
### High level design overview
![Gatus diagram](.github/assets/gatus-diagram.jpg)

View File

@ -71,7 +71,7 @@ func (alert *Alert) ValidateAndSetDefaults() error {
}
// GetDescription retrieves the description of the alert
func (alert Alert) GetDescription() string {
func (alert *Alert) GetDescription() string {
if alert.Description == nil {
return ""
}
@ -80,7 +80,7 @@ func (alert Alert) GetDescription() string {
// IsEnabled returns whether an alert is enabled or not
// Returns true if not set
func (alert Alert) IsEnabled() bool {
func (alert *Alert) IsEnabled() bool {
if alert.Enabled == nil {
return true
}
@ -88,7 +88,7 @@ func (alert Alert) IsEnabled() bool {
}
// IsSendingOnResolved returns whether an alert is sending on resolve or not
func (alert Alert) IsSendingOnResolved() bool {
func (alert *Alert) IsSendingOnResolved() bool {
if alert.SendOnResolved == nil {
return false
}

View File

@ -1,6 +1,7 @@
package alert
import (
"errors"
"testing"
)
@ -38,7 +39,7 @@ func TestAlert_ValidateAndSetDefaults(t *testing.T) {
}
for _, scenario := range scenarios {
t.Run(scenario.name, func(t *testing.T) {
if err := scenario.alert.ValidateAndSetDefaults(); err != scenario.expectedError {
if err := scenario.alert.ValidateAndSetDefaults(); !errors.Is(err, scenario.expectedError) {
t.Errorf("expected error %v, got %v", scenario.expectedError, err)
}
if scenario.alert.SuccessThreshold != scenario.expectedSuccessThreshold {
@ -52,34 +53,34 @@ func TestAlert_ValidateAndSetDefaults(t *testing.T) {
}
func TestAlert_IsEnabled(t *testing.T) {
if !(Alert{Enabled: nil}).IsEnabled() {
if !(&Alert{Enabled: nil}).IsEnabled() {
t.Error("alert.IsEnabled() should've returned true, because Enabled was set to nil")
}
if value := false; (Alert{Enabled: &value}).IsEnabled() {
if value := false; (&Alert{Enabled: &value}).IsEnabled() {
t.Error("alert.IsEnabled() should've returned false, because Enabled was set to false")
}
if value := true; !(Alert{Enabled: &value}).IsEnabled() {
if value := true; !(&Alert{Enabled: &value}).IsEnabled() {
t.Error("alert.IsEnabled() should've returned true, because Enabled was set to true")
}
}
func TestAlert_GetDescription(t *testing.T) {
if (Alert{Description: nil}).GetDescription() != "" {
if (&Alert{Description: nil}).GetDescription() != "" {
t.Error("alert.GetDescription() should've returned an empty string, because Description was set to nil")
}
if value := "description"; (Alert{Description: &value}).GetDescription() != value {
if value := "description"; (&Alert{Description: &value}).GetDescription() != value {
t.Error("alert.GetDescription() should've returned false, because Description was set to 'description'")
}
}
func TestAlert_IsSendingOnResolved(t *testing.T) {
if (Alert{SendOnResolved: nil}).IsSendingOnResolved() {
if (&Alert{SendOnResolved: nil}).IsSendingOnResolved() {
t.Error("alert.IsSendingOnResolved() should've returned false, because SendOnResolved was set to nil")
}
if value := false; (Alert{SendOnResolved: &value}).IsSendingOnResolved() {
if value := false; (&Alert{SendOnResolved: &value}).IsSendingOnResolved() {
t.Error("alert.IsSendingOnResolved() should've returned false, because SendOnResolved was set to false")
}
if value := true; !(Alert{SendOnResolved: &value}).IsSendingOnResolved() {
if value := true; !(&Alert{SendOnResolved: &value}).IsSendingOnResolved() {
t.Error("alert.IsSendingOnResolved() should've returned true, because SendOnResolved was set to true")
}
}

View File

@ -76,6 +76,8 @@ func (a *API) createRouter(cfg *config.Config) *fiber.App {
unprotectedAPIRouter.Get("/v1/endpoints/:key/uptimes/:duration/badge.svg", UptimeBadge)
unprotectedAPIRouter.Get("/v1/endpoints/:key/response-times/:duration/badge.svg", ResponseTimeBadge(cfg))
unprotectedAPIRouter.Get("/v1/endpoints/:key/response-times/:duration/chart.svg", ResponseTimeChart)
// This endpoint requires authz with bearer token, so technically it is protected
unprotectedAPIRouter.Post("/v1/endpoints/:key/external", CreateExternalEndpointResult(cfg))
// SPA
app.Get("/", SinglePageApplication(cfg.UI))
app.Get("/endpoints/:name", SinglePageApplication(cfg.UI))

View File

@ -2,12 +2,14 @@ package api
import (
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"time"
"github.com/TwiN/gatus/v5/config"
"github.com/TwiN/gatus/v5/core/ui"
"github.com/TwiN/gatus/v5/storage/store"
"github.com/TwiN/gatus/v5/storage/store/common"
"github.com/TwiN/gatus/v5/storage/store/common/paging"
@ -52,9 +54,9 @@ func UptimeBadge(c *fiber.Ctx) error {
key := c.Params("key")
uptime, err := store.Get().GetUptimeByKey(key, from, time.Now())
if err != nil {
if err == common.ErrEndpointNotFound {
if errors.Is(err, common.ErrEndpointNotFound) {
return c.Status(404).SendString(err.Error())
} else if err == common.ErrInvalidTimeRange {
} else if errors.Is(err, common.ErrInvalidTimeRange) {
return c.Status(400).SendString(err.Error())
}
return c.Status(500).SendString(err.Error())
@ -68,7 +70,7 @@ func UptimeBadge(c *fiber.Ctx) error {
// ResponseTimeBadge handles the automatic generation of badge based on the group name and endpoint name passed.
//
// Valid values for :duration -> 7d, 24h, 1h
func ResponseTimeBadge(config *config.Config) fiber.Handler {
func ResponseTimeBadge(cfg *config.Config) fiber.Handler {
return func(c *fiber.Ctx) error {
duration := c.Params("duration")
var from time.Time
@ -85,9 +87,9 @@ func ResponseTimeBadge(config *config.Config) fiber.Handler {
key := c.Params("key")
averageResponseTime, err := store.Get().GetAverageResponseTimeByKey(key, from, time.Now())
if err != nil {
if err == common.ErrEndpointNotFound {
if errors.Is(err, common.ErrEndpointNotFound) {
return c.Status(404).SendString(err.Error())
} else if err == common.ErrInvalidTimeRange {
} else if errors.Is(err, common.ErrInvalidTimeRange) {
return c.Status(400).SendString(err.Error())
}
return c.Status(500).SendString(err.Error())
@ -95,7 +97,7 @@ func ResponseTimeBadge(config *config.Config) fiber.Handler {
c.Set("Content-Type", "image/svg+xml")
c.Set("Cache-Control", "no-cache, no-store, must-revalidate")
c.Set("Expires", "0")
return c.Status(200).Send(generateResponseTimeBadgeSVG(duration, averageResponseTime, key, config))
return c.Status(200).Send(generateResponseTimeBadgeSVG(duration, averageResponseTime, key, cfg))
}
}
@ -105,9 +107,9 @@ func HealthBadge(c *fiber.Ctx) error {
pagingConfig := paging.NewEndpointStatusParams()
status, err := store.Get().GetEndpointStatusByKey(key, pagingConfig.WithResults(1, 1))
if err != nil {
if err == common.ErrEndpointNotFound {
if errors.Is(err, common.ErrEndpointNotFound) {
return c.Status(404).SendString(err.Error())
} else if err == common.ErrInvalidTimeRange {
} else if errors.Is(err, common.ErrInvalidTimeRange) {
return c.Status(400).SendString(err.Error())
}
return c.Status(500).SendString(err.Error())
@ -131,9 +133,9 @@ func HealthBadgeShields(c *fiber.Ctx) error {
pagingConfig := paging.NewEndpointStatusParams()
status, err := store.Get().GetEndpointStatusByKey(key, pagingConfig.WithResults(1, 1))
if err != nil {
if err == common.ErrEndpointNotFound {
if errors.Is(err, common.ErrEndpointNotFound) {
return c.Status(404).SendString(err.Error())
} else if err == common.ErrInvalidTimeRange {
} else if errors.Is(err, common.ErrInvalidTimeRange) {
return c.Status(400).SendString(err.Error())
}
return c.Status(500).SendString(err.Error())
@ -271,10 +273,13 @@ func generateResponseTimeBadgeSVG(duration string, averageResponseTime int, key
}
func getBadgeColorFromResponseTime(responseTime int, key string, cfg *config.Config) string {
endpoint := cfg.GetEndpointByKey(key)
thresholds := ui.GetDefaultConfig().Badge.ResponseTime.Thresholds
if endpoint := cfg.GetEndpointByKey(key); endpoint != nil {
thresholds = endpoint.UIConfig.Badge.ResponseTime.Thresholds
}
// the threshold config requires 5 values, so we can be sure it's set here
for i := 0; i < 5; i++ {
if responseTime <= endpoint.UIConfig.Badge.ResponseTime.Thresholds[i] {
if responseTime <= thresholds[i] {
return badgeColors[i]
}
}

View File

@ -1,6 +1,7 @@
package api
import (
"errors"
"log"
"math"
"net/http"
@ -42,9 +43,9 @@ func ResponseTimeChart(c *fiber.Ctx) error {
}
hourlyAverageResponseTime, err := store.Get().GetHourlyAverageResponseTimeByKey(c.Params("key"), from, time.Now())
if err != nil {
if err == common.ErrEndpointNotFound {
if errors.Is(err, common.ErrEndpointNotFound) {
return c.Status(404).SendString(err.Error())
} else if err == common.ErrInvalidTimeRange {
} else if errors.Is(err, common.ErrInvalidTimeRange) {
return c.Status(400).SendString(err.Error())
}
return c.Status(500).SendString(err.Error())

View File

@ -65,13 +65,13 @@ func getEndpointStatusesFromRemoteInstances(remoteConfig *remote.Config) ([]*cor
body, err := io.ReadAll(response.Body)
if err != nil {
_ = response.Body.Close()
log.Printf("[handler.getEndpointStatusesFromRemoteInstances] Silently failed to retrieve endpoint statuses from %s: %s", instance.URL, err.Error())
log.Printf("[api.getEndpointStatusesFromRemoteInstances] Silently failed to retrieve endpoint statuses from %s: %s", instance.URL, err.Error())
continue
}
var endpointStatuses []*core.EndpointStatus
if err = json.Unmarshal(body, &endpointStatuses); err != nil {
_ = response.Body.Close()
log.Printf("[handler.getEndpointStatusesFromRemoteInstances] Silently failed to retrieve endpoint statuses from %s: %s", instance.URL, err.Error())
log.Printf("[api.getEndpointStatusesFromRemoteInstances] Silently failed to retrieve endpoint statuses from %s: %s", instance.URL, err.Error())
continue
}
_ = response.Body.Close()

67
api/external_endpoint.go Normal file
View File

@ -0,0 +1,67 @@
package api
import (
"errors"
"log"
"strings"
"time"
"github.com/TwiN/gatus/v5/config"
"github.com/TwiN/gatus/v5/core"
"github.com/TwiN/gatus/v5/storage/store"
"github.com/TwiN/gatus/v5/storage/store/common"
"github.com/TwiN/gatus/v5/watchdog"
"github.com/gofiber/fiber/v2"
)
func CreateExternalEndpointResult(cfg *config.Config) fiber.Handler {
return func(c *fiber.Ctx) error {
// Check if the success query parameter is present
success, exists := c.Queries()["success"]
if !exists || (success != "true" && success != "false") {
return c.Status(400).SendString("missing or invalid success query parameter")
}
// Check if the authorization bearer token header is correct
authorizationHeader := string(c.Request().Header.Peek("Authorization"))
if !strings.HasPrefix(authorizationHeader, "Bearer ") {
return c.Status(401).SendString("invalid Authorization header")
}
token := strings.TrimSpace(strings.TrimPrefix(authorizationHeader, "Bearer "))
if len(token) == 0 {
return c.Status(401).SendString("bearer token must not be empty")
}
key := c.Params("key")
externalEndpoint := cfg.GetExternalEndpointByKey(key)
if externalEndpoint == nil {
log.Printf("[api.CreateExternalEndpointResult] External endpoint with key=%s not found", key)
return c.Status(404).SendString("not found")
}
if externalEndpoint.Token != token {
log.Printf("[api.CreateExternalEndpointResult] Invalid token for external endpoint with key=%s", key)
return c.Status(401).SendString("invalid token")
}
// Persist the result in the storage
result := &core.Result{
Timestamp: time.Now(),
Success: c.QueryBool("success"),
Errors: []string{},
}
convertedEndpoint := externalEndpoint.ToEndpoint()
if err := store.Get().Insert(convertedEndpoint, result); err != nil {
if errors.Is(err, common.ErrEndpointNotFound) {
return c.Status(404).SendString(err.Error())
}
log.Printf("[api.CreateExternalEndpointResult] Failed to insert result in storage: %s", err.Error())
return c.Status(500).SendString(err.Error())
}
log.Printf("[api.CreateExternalEndpointResult] Successfully inserted result for external endpoint with key=%s and success=%s", c.Params("key"), success)
// Check if an alert should be triggered or resolved
if !cfg.Maintenance.IsUnderMaintenance() {
watchdog.HandleAlerting(convertedEndpoint, result, cfg.Alerting, cfg.Debug)
externalEndpoint.NumberOfSuccessesInARow = convertedEndpoint.NumberOfSuccessesInARow
externalEndpoint.NumberOfFailuresInARow = convertedEndpoint.NumberOfFailuresInARow
}
// Return the result
return c.Status(200).SendString("")
}
}

View File

@ -0,0 +1,131 @@
package api
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/TwiN/gatus/v5/alerting"
"github.com/TwiN/gatus/v5/alerting/alert"
"github.com/TwiN/gatus/v5/alerting/provider/discord"
"github.com/TwiN/gatus/v5/config"
"github.com/TwiN/gatus/v5/config/maintenance"
"github.com/TwiN/gatus/v5/core"
"github.com/TwiN/gatus/v5/storage/store"
"github.com/TwiN/gatus/v5/storage/store/common/paging"
)
func TestCreateExternalEndpointResult(t *testing.T) {
defer store.Get().Clear()
defer cache.Clear()
cfg := &config.Config{
Alerting: &alerting.Config{
Discord: &discord.AlertProvider{},
},
ExternalEndpoints: []*core.ExternalEndpoint{
{
Name: "n",
Group: "g",
Token: "token",
Alerts: []*alert.Alert{
{
Type: alert.TypeDiscord,
FailureThreshold: 2,
SuccessThreshold: 2,
},
},
},
},
Maintenance: &maintenance.Config{},
}
api := New(cfg)
router := api.Router()
scenarios := []struct {
Name string
Path string
AuthorizationHeaderBearerToken string
ExpectedCode int
}{
{
Name: "no-token",
Path: "/api/v1/endpoints/g_n/external?success=true",
AuthorizationHeaderBearerToken: "",
ExpectedCode: 401,
},
{
Name: "bad-token",
Path: "/api/v1/endpoints/g_n/external?success=true",
AuthorizationHeaderBearerToken: "Bearer bad-token",
ExpectedCode: 401,
},
{
Name: "bad-key",
Path: "/api/v1/endpoints/bad_key/external?success=true",
AuthorizationHeaderBearerToken: "Bearer token",
ExpectedCode: 404,
},
{
Name: "good-token-success-true",
Path: "/api/v1/endpoints/g_n/external?success=true",
AuthorizationHeaderBearerToken: "Bearer token",
ExpectedCode: 200,
},
{
Name: "good-token-success-false",
Path: "/api/v1/endpoints/g_n/external?success=false",
AuthorizationHeaderBearerToken: "Bearer token",
ExpectedCode: 200,
},
{
Name: "good-token-success-false-again",
Path: "/api/v1/endpoints/g_n/external?success=false",
AuthorizationHeaderBearerToken: "Bearer token",
ExpectedCode: 200,
},
}
for _, scenario := range scenarios {
t.Run(scenario.Name, func(t *testing.T) {
request := httptest.NewRequest("POST", scenario.Path, http.NoBody)
if len(scenario.AuthorizationHeaderBearerToken) > 0 {
request.Header.Set("Authorization", scenario.AuthorizationHeaderBearerToken)
}
response, err := router.Test(request)
if err != nil {
return
}
defer response.Body.Close()
if response.StatusCode != scenario.ExpectedCode {
t.Errorf("%s %s should have returned %d, but returned %d instead", request.Method, request.URL, scenario.ExpectedCode, response.StatusCode)
}
})
}
t.Run("verify-end-results", func(t *testing.T) {
endpointStatus, err := store.Get().GetEndpointStatus("g", "n", paging.NewEndpointStatusParams().WithResults(1, 10))
if err != nil {
t.Errorf("failed to get endpoint status: %s", err.Error())
return
}
if endpointStatus.Key != "g_n" {
t.Errorf("expected key to be g_n but got %s", endpointStatus.Key)
}
if len(endpointStatus.Results) != 3 {
t.Errorf("expected 3 results but got %d", len(endpointStatus.Results))
}
if !endpointStatus.Results[0].Success {
t.Errorf("expected first result to be successful")
}
if endpointStatus.Results[1].Success {
t.Errorf("expected second result to be unsuccessful")
}
if endpointStatus.Results[2].Success {
t.Errorf("expected third result to be unsuccessful")
}
externalEndpointFromConfig := cfg.GetExternalEndpointByKey("g_n")
if externalEndpointFromConfig.NumberOfFailuresInARow != 2 {
t.Errorf("expected 2 failures in a row but got %d", externalEndpointFromConfig.NumberOfFailuresInARow)
}
if externalEndpointFromConfig.NumberOfSuccessesInARow != 0 {
t.Errorf("expected 0 successes in a row but got %d", externalEndpointFromConfig.NumberOfSuccessesInARow)
}
})
}

View File

@ -67,15 +67,18 @@ type Config struct {
// Disabling this may lead to inaccurate response times
DisableMonitoringLock bool `yaml:"disable-monitoring-lock,omitempty"`
// Security Configuration for securing access to Gatus
// Security is the configuration for securing access to Gatus
Security *security.Config `yaml:"security,omitempty"`
// Alerting Configuration for alerting
// Alerting is the configuration for alerting providers
Alerting *alerting.Config `yaml:"alerting,omitempty"`
// Endpoints List of endpoints to monitor
// Endpoints is the list of endpoints to monitor
Endpoints []*core.Endpoint `yaml:"endpoints,omitempty"`
// ExternalEndpoints is the list of all external endpoints
ExternalEndpoints []*core.ExternalEndpoint `yaml:"external-endpoints,omitempty"`
// Storage is the configuration for how the data is stored
Storage *storage.Config `yaml:"storage,omitempty"`
@ -100,7 +103,6 @@ type Config struct {
}
func (config *Config) GetEndpointByKey(key string) *core.Endpoint {
// TODO: Should probably add a mutex here to prevent concurrent access
for i := 0; i < len(config.Endpoints); i++ {
ep := config.Endpoints[i]
if util.ConvertGroupAndEndpointNameToKey(ep.Group, ep.Name) == key {
@ -110,9 +112,19 @@ func (config *Config) GetEndpointByKey(key string) *core.Endpoint {
return nil
}
func (config *Config) GetExternalEndpointByKey(key string) *core.ExternalEndpoint {
for i := 0; i < len(config.ExternalEndpoints); i++ {
ee := config.ExternalEndpoints[i]
if util.ConvertGroupAndEndpointNameToKey(ee.Group, ee.Name) == key {
return ee
}
}
return nil
}
// HasLoadedConfigurationBeenModified returns whether one of the file that the
// configuration has been loaded from has been modified since it was last read
func (config Config) HasLoadedConfigurationBeenModified() bool {
func (config *Config) HasLoadedConfigurationBeenModified() bool {
lastMod := config.lastFileModTime.Unix()
fileInfo, err := os.Stat(config.configPath)
if err != nil {
@ -125,7 +137,7 @@ func (config Config) HasLoadedConfigurationBeenModified() bool {
}
return nil
})
return err == errEarlyReturn
return errors.Is(err, errEarlyReturn)
}
return !fileInfo.ModTime().IsZero() && config.lastFileModTime.Unix() < fileInfo.ModTime().Unix()
}
@ -135,7 +147,7 @@ func (config *Config) UpdateLastFileModTime() {
config.lastFileModTime = time.Now()
}
// LoadConfiguration loads the full configuration composed from the main configuration file
// LoadConfiguration loads the full configuration composed of the main configuration file
// and all composed configuration files
func LoadConfiguration(configPath string) (*Config, error) {
var configBytes []byte
@ -241,6 +253,9 @@ func parseAndValidateConfigBytes(yamlBytes []byte) (config *Config, err error) {
if err := validateEndpointsConfig(config); err != nil {
return nil, err
}
if err := validateExternalEndpointsConfig(config); err != nil {
return nil, err
}
if err := validateWebConfig(config); err != nil {
return nil, err
}
@ -336,6 +351,19 @@ func validateEndpointsConfig(config *Config) error {
return nil
}
func validateExternalEndpointsConfig(config *Config) error {
for _, externalEndpoint := range config.ExternalEndpoints {
if config.Debug {
log.Printf("[config.validateExternalEndpointsConfig] Validating external endpoint '%s'", externalEndpoint.Name)
}
if err := externalEndpoint.ValidateAndSetDefaults(); err != nil {
return fmt.Errorf("invalid external endpoint %s: %w", externalEndpoint.DisplayName(), err)
}
}
log.Printf("[config.validateExternalEndpointsConfig] Validated %d external endpoints", len(config.ExternalEndpoints))
return nil
}
func validateSecurityConfig(config *Config) error {
if config.Security != nil {
if config.Security.IsValid() {

View File

@ -304,11 +304,13 @@ func TestParseAndValidateConfigBytes(t *testing.T) {
storage:
type: sqlite
path: %s
maintenance:
enabled: true
start: 00:00
duration: 4h
every: [Monday, Thursday]
ui:
title: T
header: H
@ -318,6 +320,16 @@ ui:
link: "https://example.org"
- name: "Status page"
link: "https://status.example.org"
external-endpoints:
- name: ext-ep-test
group: core
token: "potato"
alerts:
- type: discord
description: "healthcheck failed"
send-on-resolved: true
endpoints:
- name: website
url: https://twin.sh/health
@ -358,10 +370,33 @@ endpoints:
if mc := config.Maintenance; mc == nil || mc.Start != "00:00" || !mc.IsEnabled() || mc.Duration != 4*time.Hour || len(mc.Every) != 2 {
t.Error("Expected Config.Maintenance to be configured properly")
}
if len(config.ExternalEndpoints) != 1 {
t.Error("Should have returned one external endpoint")
}
if config.ExternalEndpoints[0].Name != "ext-ep-test" {
t.Errorf("Name should have been %s", "ext-ep-test")
}
if config.ExternalEndpoints[0].Group != "core" {
t.Errorf("Group should have been %s", "core")
}
if config.ExternalEndpoints[0].Token != "potato" {
t.Errorf("Token should have been %s", "potato")
}
if len(config.ExternalEndpoints[0].Alerts) != 1 {
t.Error("Should have returned one alert")
}
if config.ExternalEndpoints[0].Alerts[0].Type != alert.TypeDiscord {
t.Errorf("Type should have been %s", alert.TypeDiscord)
}
if config.ExternalEndpoints[0].Alerts[0].FailureThreshold != 3 {
t.Errorf("FailureThreshold should have been %d, got %d", 3, config.ExternalEndpoints[0].Alerts[0].FailureThreshold)
}
if config.ExternalEndpoints[0].Alerts[0].SuccessThreshold != 2 {
t.Errorf("SuccessThreshold should have been %d, got %d", 2, config.ExternalEndpoints[0].Alerts[0].SuccessThreshold)
}
if len(config.Endpoints) != 3 {
t.Error("Should have returned two endpoints")
}
if config.Endpoints[0].URL != "https://twin.sh/health" {
t.Errorf("URL should have been %s", "https://twin.sh/health")
}
@ -383,7 +418,6 @@ endpoints:
if len(config.Endpoints[0].Conditions) != 1 {
t.Errorf("There should have been %d conditions", 1)
}
if config.Endpoints[1].URL != "https://api.github.com/healthz" {
t.Errorf("URL should have been %s", "https://api.github.com/healthz")
}

View File

@ -103,26 +103,24 @@ func TestIntegrationQuery(t *testing.T) {
}
}
func TestEndpoint_ValidateAndSetDefaultsWithNoDNSQueryName(t *testing.T) {
defer func() { recover() }()
func TestDNS_validateAndSetDefault(t *testing.T) {
dns := &DNS{
QueryType: "A",
QueryName: "",
}
err := dns.validateAndSetDefault()
if err == nil {
t.Fatal("Should've returned an error because endpoint's dns didn't have a query name, which is a mandatory field for dns")
t.Error("Should've returned an error because endpoint's dns didn't have a query name, which is a mandatory field for dns")
}
}
func TestEndpoint_ValidateAndSetDefaultsWithInvalidDNSQueryType(t *testing.T) {
defer func() { recover() }()
dns := &DNS{
QueryType: "B",
QueryName: "example.com",
}
err := dns.validateAndSetDefault()
if err == nil {
t.Fatal("Should've returned an error because endpoint's dns query type is invalid, it needs to be a valid query name like A, AAAA, CNAME...")
t.Error("Should've returned an error because endpoint's dns query type is invalid, it needs to be a valid query name like A, AAAA, CNAME...")
}
}

View File

@ -55,12 +55,6 @@ var (
// ErrEndpointWithNoURL is the error with which Gatus will panic if an endpoint is configured with no url
ErrEndpointWithNoURL = errors.New("you must specify an url for each endpoint")
// ErrEndpointWithNoName is the error with which Gatus will panic if an endpoint is configured with no name
ErrEndpointWithNoName = errors.New("you must specify a name for each endpoint")
// ErrEndpointWithInvalidNameOrGroup is the error with which Gatus will panic if an endpoint has an invalid character where it shouldn't
ErrEndpointWithInvalidNameOrGroup = errors.New("endpoint name and group must not have \" or \\")
// ErrUnknownEndpointType is the error with which Gatus will panic if an endpoint has an unknown type
ErrUnknownEndpointType = errors.New("unknown endpoint type")
@ -72,13 +66,9 @@ var (
// This is because the free whois service we are using should not be abused, especially considering the fact that
// the data takes a while to be updated.
ErrInvalidEndpointIntervalForDomainExpirationPlaceholder = errors.New("the minimum interval for an endpoint with a condition using the " + DomainExpirationPlaceholder + " placeholder is 300s (5m)")
// ErrEndpointWithoutSSHUsername is the error with which Gatus will panic if an endpoint with SSH monitoring is configured without a user.
ErrEndpointWithoutSSHUsername = errors.New("you must specify a username for each endpoint with SSH")
// ErrEndpointWithoutSSHPassword is the error with which Gatus will panic if an endpoint with SSH monitoring is configured without a password.
ErrEndpointWithoutSSHPassword = errors.New("you must specify a password for each endpoint with SSH")
)
// Endpoint is the configuration of a monitored
// Endpoint is the configuration of a service to be monitored
type Endpoint struct {
// Enabled defines whether to enable the monitoring of the endpoint
Enabled *bool `yaml:"enabled,omitempty"`
@ -95,6 +85,9 @@ type Endpoint struct {
// DNS is the configuration of DNS monitoring
DNS *DNS `yaml:"dns,omitempty"`
// SSH is the configuration of SSH monitoring.
SSH *SSH `yaml:"ssh,omitempty"`
// Method of the request made to the url of the endpoint
Method string `yaml:"method,omitempty"`
@ -127,27 +120,6 @@ type Endpoint struct {
// NumberOfSuccessesInARow is the number of successful evaluations in a row
NumberOfSuccessesInARow int `yaml:"-"`
// SSH is the configuration of SSH monitoring.
SSH *SSH `yaml:"ssh,omitempty"`
}
type SSH struct {
// Username is the username to use when connecting to the SSH server.
Username string `yaml:"username,omitempty"`
// Password is the password to use when connecting to the SSH server.
Password string `yaml:"password,omitempty"`
}
// ValidateAndSetDefaults validates the endpoint
func (s *SSH) ValidateAndSetDefaults() error {
if s.Username == "" {
return ErrEndpointWithoutSSHUsername
}
if s.Password == "" {
return ErrEndpointWithoutSSHPassword
}
return nil
}
// IsEnabled returns whether the endpoint is enabled or not
@ -188,7 +160,12 @@ func (endpoint *Endpoint) Type() EndpointType {
// ValidateAndSetDefaults validates the endpoint's configuration and sets the default value of args that have one
func (endpoint *Endpoint) ValidateAndSetDefaults() error {
// Set default values
if err := validateEndpointNameGroupAndAlerts(endpoint.Name, endpoint.Group, endpoint.Alerts); err != nil {
return err
}
if len(endpoint.URL) == 0 {
return ErrEndpointWithNoURL
}
if endpoint.ClientConfig == nil {
endpoint.ClientConfig = client.GetDefaultConfig()
} else {
@ -221,20 +198,6 @@ func (endpoint *Endpoint) ValidateAndSetDefaults() error {
if _, contentTypeHeaderExists := endpoint.Headers[ContentTypeHeader]; !contentTypeHeaderExists && endpoint.GraphQL {
endpoint.Headers[ContentTypeHeader] = "application/json"
}
for _, endpointAlert := range endpoint.Alerts {
if err := endpointAlert.ValidateAndSetDefaults(); err != nil {
return err
}
}
if len(endpoint.Name) == 0 {
return ErrEndpointWithNoName
}
if strings.ContainsAny(endpoint.Name, "\"\\") || strings.ContainsAny(endpoint.Group, "\"\\") {
return ErrEndpointWithInvalidNameOrGroup
}
if len(endpoint.URL) == 0 {
return ErrEndpointWithNoURL
}
if len(endpoint.Conditions) == 0 {
return ErrEndpointWithNoCondition
}
@ -249,6 +212,9 @@ func (endpoint *Endpoint) ValidateAndSetDefaults() error {
if endpoint.DNS != nil {
return endpoint.DNS.validateAndSetDefault()
}
if endpoint.SSH != nil {
return endpoint.SSH.validate()
}
if endpoint.Type() == EndpointTypeUNKNOWN {
return ErrUnknownEndpointType
}
@ -257,9 +223,6 @@ func (endpoint *Endpoint) ValidateAndSetDefaults() error {
if err != nil {
return err
}
if endpoint.SSH != nil {
return endpoint.SSH.ValidateAndSetDefaults()
}
return nil
}
@ -276,6 +239,15 @@ func (endpoint *Endpoint) Key() string {
return util.ConvertGroupAndEndpointNameToKey(endpoint.Group, endpoint.Name)
}
// Close HTTP connections between watchdog and endpoints to avoid dangling socket file descriptors
// on configuration reload.
// More context on https://github.com/TwiN/gatus/issues/536
func (endpoint *Endpoint) Close() {
if endpoint.Type() == EndpointTypeHTTP {
client.GetHTTPClient(endpoint.ClientConfig).CloseIdleConnections()
}
}
// EvaluateHealth sends a request to the endpoint's URL and evaluates the conditions of the endpoint.
func (endpoint *Endpoint) EvaluateHealth() *Result {
result := &Result{Success: true, Errors: []string{}}
@ -419,15 +391,6 @@ func (endpoint *Endpoint) call(result *Result) {
}
}
// Close HTTP connections between watchdog and endpoints to avoid dangling socket file descriptors
// on configuration reload.
// More context on https://github.com/TwiN/gatus/issues/536
func (endpoint *Endpoint) Close() {
if endpoint.Type() == EndpointTypeHTTP {
client.GetHTTPClient(endpoint.ClientConfig).CloseIdleConnections()
}
}
func (endpoint *Endpoint) buildHTTPRequest() *http.Request {
var bodyBuffer *bytes.Buffer
if endpoint.GraphQL {

32
core/endpoint_common.go Normal file
View File

@ -0,0 +1,32 @@
package core
import (
"errors"
"strings"
"github.com/TwiN/gatus/v5/alerting/alert"
)
var (
// ErrEndpointWithNoName is the error with which Gatus will panic if an endpoint is configured with no name
ErrEndpointWithNoName = errors.New("you must specify a name for each endpoint")
// ErrEndpointWithInvalidNameOrGroup is the error with which Gatus will panic if an endpoint has an invalid character where it shouldn't
ErrEndpointWithInvalidNameOrGroup = errors.New("endpoint name and group must not have \" or \\")
)
// validateEndpointNameGroupAndAlerts validates the name, group and alerts of an endpoint
func validateEndpointNameGroupAndAlerts(name, group string, alerts []*alert.Alert) error {
if len(name) == 0 {
return ErrEndpointWithNoName
}
if strings.ContainsAny(name, "\"\\") || strings.ContainsAny(group, "\"\\") {
return ErrEndpointWithInvalidNameOrGroup
}
for _, endpointAlert := range alerts {
if err := endpointAlert.ValidateAndSetDefaults(); err != nil {
return err
}
}
return nil
}

View File

@ -0,0 +1,51 @@
package core
import (
"errors"
"testing"
"github.com/TwiN/gatus/v5/alerting/alert"
)
func TestValidateEndpointNameGroupAndAlerts(t *testing.T) {
scenarios := []struct {
name string
group string
alerts []*alert.Alert
expectedErr error
}{
{
name: "n",
group: "g",
alerts: []*alert.Alert{{Type: "slack"}},
},
{
name: "n",
alerts: []*alert.Alert{{Type: "slack"}},
},
{
group: "g",
alerts: []*alert.Alert{{Type: "slack"}},
expectedErr: ErrEndpointWithNoName,
},
{
name: "\"",
alerts: []*alert.Alert{{Type: "slack"}},
expectedErr: ErrEndpointWithInvalidNameOrGroup,
},
{
name: "n",
group: "\\",
alerts: []*alert.Alert{{Type: "slack"}},
expectedErr: ErrEndpointWithInvalidNameOrGroup,
},
}
for _, scenario := range scenarios {
t.Run(scenario.name, func(t *testing.T) {
err := validateEndpointNameGroupAndAlerts(scenario.name, scenario.group, scenario.alerts)
if !errors.Is(err, scenario.expectedErr) {
t.Errorf("expected error to be %v but got %v", scenario.expectedErr, err)
}
})
}
}

View File

@ -4,6 +4,7 @@ import (
"bytes"
"crypto/tls"
"crypto/x509"
"errors"
"io"
"net/http"
"strings"
@ -123,6 +124,7 @@ func TestEndpoint(t *testing.T) {
Name: "website-health",
URL: "https://twin.sh/health",
Conditions: []Condition{"[DOMAIN_EXPIRATION] > 100h"},
Interval: 5 * time.Minute,
},
ExpectedResult: &Result{
Success: true,
@ -195,7 +197,10 @@ func TestEndpoint(t *testing.T) {
} else {
client.InjectHTTPClient(nil)
}
scenario.Endpoint.ValidateAndSetDefaults()
err := scenario.Endpoint.ValidateAndSetDefaults()
if err != nil {
t.Error("did not expect an error, got", err)
}
result := scenario.Endpoint.EvaluateHealth()
if result.Success != scenario.ExpectedResult.Success {
t.Errorf("Expected success to be %v, got %v", scenario.ExpectedResult.Success, result.Success)
@ -430,7 +435,10 @@ func TestEndpoint_ValidateAndSetDefaultsWithClientConfig(t *testing.T) {
Timeout: 0,
},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
if endpoint.ClientConfig == nil {
t.Error("client configuration should've been set to the default configuration")
} else {
@ -466,7 +474,7 @@ func TestEndpoint_ValidateAndSetDefaultsWithDNS(t *testing.T) {
}
func TestEndpoint_ValidateAndSetDefaultsWithSSH(t *testing.T) {
tests := []struct {
scenarios := []struct {
name string
username string
password string
@ -492,20 +500,20 @@ func TestEndpoint_ValidateAndSetDefaultsWithSSH(t *testing.T) {
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
for _, scenario := range scenarios {
t.Run(scenario.name, func(t *testing.T) {
endpoint := &Endpoint{
Name: "ssh-test",
URL: "https://example.com",
SSH: &SSH{
Username: test.username,
Password: test.password,
Username: scenario.username,
Password: scenario.password,
},
Conditions: []Condition{Condition("[STATUS] == 0")},
}
err := endpoint.ValidateAndSetDefaults()
if err != test.expectedErr {
t.Errorf("expected error %v, got %v", test.expectedErr, err)
if !errors.Is(err, scenario.expectedErr) {
t.Errorf("expected error %v, got %v", scenario.expectedErr, err)
}
})
}
@ -575,7 +583,10 @@ func TestEndpoint_buildHTTPRequest(t *testing.T) {
URL: "https://twin.sh/health",
Conditions: []Condition{condition},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
request := endpoint.buildHTTPRequest()
if request.Method != "GET" {
t.Error("request.Method should've been GET, but was", request.Method)
@ -598,7 +609,10 @@ func TestEndpoint_buildHTTPRequestWithCustomUserAgent(t *testing.T) {
"User-Agent": "Test/2.0",
},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
request := endpoint.buildHTTPRequest()
if request.Method != "GET" {
t.Error("request.Method should've been GET, but was", request.Method)
@ -622,7 +636,10 @@ func TestEndpoint_buildHTTPRequestWithHostHeader(t *testing.T) {
"Host": "example.com",
},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
request := endpoint.buildHTTPRequest()
if request.Method != "POST" {
t.Error("request.Method should've been POST, but was", request.Method)
@ -649,7 +666,10 @@ func TestEndpoint_buildHTTPRequestWithGraphQLEnabled(t *testing.T) {
}
}`,
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
request := endpoint.buildHTTPRequest()
if request.Method != "POST" {
t.Error("request.Method should've been POST, but was", request.Method)
@ -671,7 +691,10 @@ func TestIntegrationEvaluateHealth(t *testing.T) {
URL: "https://twin.sh/health",
Conditions: []Condition{condition, bodyCondition},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
result := endpoint.EvaluateHealth()
if !result.ConditionResults[0].Success {
t.Errorf("Condition '%s' should have been a success", condition)
@ -699,7 +722,10 @@ func TestIntegrationEvaluateHealthWithErrorAndHideURL(t *testing.T) {
HideURL: true,
},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
result := endpoint.EvaluateHealth()
if result.Success {
t.Error("Because one of the conditions was invalid, result.Success should have been false")
@ -724,7 +750,10 @@ func TestIntegrationEvaluateHealthForDNS(t *testing.T) {
},
Conditions: []Condition{conditionSuccess, conditionBody},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
result := endpoint.EvaluateHealth()
if !result.ConditionResults[0].Success {
t.Errorf("Conditions '%s' and '%s' should have been a success", conditionSuccess, conditionBody)
@ -792,7 +821,10 @@ func TestIntegrationEvaluateHealthForICMP(t *testing.T) {
URL: "icmp://127.0.0.1",
Conditions: []Condition{"[CONNECTED] == true"},
}
endpoint.ValidateAndSetDefaults()
err := endpoint.ValidateAndSetDefaults()
if err != nil {
t.Fatal("did not expect an error, got", err)
}
result := endpoint.EvaluateHealth()
if !result.ConditionResults[0].Success {
t.Errorf("Conditions '%s' should have been a success", endpoint.Conditions[0])

89
core/external_endpoint.go Normal file
View File

@ -0,0 +1,89 @@
package core
import (
"errors"
"github.com/TwiN/gatus/v5/alerting/alert"
"github.com/TwiN/gatus/v5/util"
)
var (
// ErrExternalEndpointWithNoToken is the error with which Gatus will panic if an external endpoint is configured without a token.
ErrExternalEndpointWithNoToken = errors.New("you must specify a token for each external endpoint")
)
// ExternalEndpoint is an endpoint whose result is pushed from outside Gatus, which means that
// said endpoints are not monitored by Gatus itself; Gatus only displays their results and takes
// care of alerting
type ExternalEndpoint struct {
// Enabled defines whether to enable the monitoring of the endpoint
Enabled *bool `yaml:"enabled,omitempty"`
// Name of the endpoint. Can be anything.
Name string `yaml:"name"`
// Group the endpoint is a part of. Used for grouping multiple endpoints together on the front end.
Group string `yaml:"group,omitempty"`
// Token is the bearer token that must be provided through the Authorization header to push results to the endpoint
Token string `yaml:"token,omitempty"`
// Alerts is the alerting configuration for the endpoint in case of failure
Alerts []*alert.Alert `yaml:"alerts,omitempty"`
// NumberOfFailuresInARow is the number of unsuccessful evaluations in a row
NumberOfFailuresInARow int `yaml:"-"`
// NumberOfSuccessesInARow is the number of successful evaluations in a row
NumberOfSuccessesInARow int `yaml:"-"`
}
// ValidateAndSetDefaults validates the ExternalEndpoint and sets the default values
func (externalEndpoint *ExternalEndpoint) ValidateAndSetDefaults() error {
if err := validateEndpointNameGroupAndAlerts(externalEndpoint.Name, externalEndpoint.Group, externalEndpoint.Alerts); err != nil {
return err
}
if len(externalEndpoint.Token) == 0 {
return ErrExternalEndpointWithNoToken
}
for _, externalEndpointAlert := range externalEndpoint.Alerts {
if err := externalEndpointAlert.ValidateAndSetDefaults(); err != nil {
return err
}
}
return nil
}
// IsEnabled returns whether the endpoint is enabled or not
func (externalEndpoint *ExternalEndpoint) IsEnabled() bool {
if externalEndpoint.Enabled == nil {
return true
}
return *externalEndpoint.Enabled
}
// DisplayName returns an identifier made up of the Name and, if not empty, the Group.
func (externalEndpoint *ExternalEndpoint) DisplayName() string {
if len(externalEndpoint.Group) > 0 {
return externalEndpoint.Group + "/" + externalEndpoint.Name
}
return externalEndpoint.Name
}
// Key returns the unique key for the Endpoint
func (externalEndpoint *ExternalEndpoint) Key() string {
return util.ConvertGroupAndEndpointNameToKey(externalEndpoint.Group, externalEndpoint.Name)
}
// ToEndpoint converts the ExternalEndpoint to an Endpoint
func (externalEndpoint *ExternalEndpoint) ToEndpoint() *Endpoint {
endpoint := &Endpoint{
Enabled: externalEndpoint.Enabled,
Name: externalEndpoint.Name,
Group: externalEndpoint.Group,
Alerts: externalEndpoint.Alerts,
NumberOfFailuresInARow: externalEndpoint.NumberOfFailuresInARow,
NumberOfSuccessesInARow: externalEndpoint.NumberOfSuccessesInARow,
}
return endpoint
}

View File

@ -0,0 +1,25 @@
package core
import (
"testing"
)
func TestExternalEndpoint_ToEndpoint(t *testing.T) {
externalEndpoint := &ExternalEndpoint{
Name: "name",
Group: "group",
}
convertedEndpoint := externalEndpoint.ToEndpoint()
if externalEndpoint.Name != convertedEndpoint.Name {
t.Errorf("expected %s, got %s", externalEndpoint.Name, convertedEndpoint.Name)
}
if externalEndpoint.Group != convertedEndpoint.Group {
t.Errorf("expected %s, got %s", externalEndpoint.Group, convertedEndpoint.Group)
}
if externalEndpoint.Key() != convertedEndpoint.Key() {
t.Errorf("expected %s, got %s", externalEndpoint.Key(), convertedEndpoint.Key())
}
if externalEndpoint.DisplayName() != convertedEndpoint.DisplayName() {
t.Errorf("expected %s, got %s", externalEndpoint.DisplayName(), convertedEndpoint.DisplayName())
}
}

View File

@ -7,7 +7,7 @@ import (
// Result of the evaluation of a Endpoint
type Result struct {
// HTTPStatus is the HTTP response status code
HTTPStatus int `json:"status"`
HTTPStatus int `json:"status,omitempty"`
// DNSRCode is the response code of a DNS query in a human-readable format
//
@ -30,7 +30,7 @@ type Result struct {
Errors []string `json:"errors,omitempty"`
// ConditionResults results of the Endpoint's conditions
ConditionResults []*ConditionResult `json:"conditionResults"`
ConditionResults []*ConditionResult `json:"conditionResults,omitempty"`
// Success whether the result signifies a success or not
Success bool `json:"success"`

29
core/ssh.go Normal file
View File

@ -0,0 +1,29 @@
package core
import (
"errors"
)
var (
// ErrEndpointWithoutSSHUsername is the error with which Gatus will panic if an endpoint with SSH monitoring is configured without a user.
ErrEndpointWithoutSSHUsername = errors.New("you must specify a username for each SSH endpoint")
// ErrEndpointWithoutSSHPassword is the error with which Gatus will panic if an endpoint with SSH monitoring is configured without a password.
ErrEndpointWithoutSSHPassword = errors.New("you must specify a password for each SSH endpoint")
)
type SSH struct {
Username string `yaml:"username,omitempty"`
Password string `yaml:"password,omitempty"`
}
// validate validates the endpoint
func (s *SSH) validate() error {
if len(s.Username) == 0 {
return ErrEndpointWithoutSSHUsername
}
if len(s.Password) == 0 {
return ErrEndpointWithoutSSHPassword
}
return nil
}

25
core/ssh_test.go Normal file
View File

@ -0,0 +1,25 @@
package core
import (
"errors"
"testing"
)
func TestSSH_validate(t *testing.T) {
ssh := &SSH{}
if err := ssh.validate(); err == nil {
t.Error("expected an error")
} else if !errors.Is(err, ErrEndpointWithoutSSHUsername) {
t.Errorf("expected error to be '%v', got '%v'", ErrEndpointWithoutSSHUsername, err)
}
ssh.Username = "username"
if err := ssh.validate(); err == nil {
t.Error("expected an error")
} else if !errors.Is(err, ErrEndpointWithoutSSHPassword) {
t.Errorf("expected error to be '%v', got '%v'", ErrEndpointWithoutSSHPassword, err)
}
ssh.Password = "password"
if err := ssh.validate(); err != nil {
t.Errorf("expected no error, got '%v'", err)
}
}

View File

@ -83,6 +83,9 @@ func initializeStorage(cfg *config.Config) {
for _, endpoint := range cfg.Endpoints {
keys = append(keys, endpoint.Key())
}
for _, externalEndpoint := range cfg.ExternalEndpoints {
keys = append(keys, externalEndpoint.Key())
}
numberOfEndpointStatusesDeleted := store.Get().DeleteAllEndpointStatusesNotInKeys(keys)
if numberOfEndpointStatusesDeleted > 0 {
log.Printf("[main.initializeStorage] Deleted %d endpoint statuses because their matching endpoints no longer existed", numberOfEndpointStatusesDeleted)

View File

@ -556,7 +556,7 @@ func (s *Store) getEndpointIDGroupAndNameByKey(tx *sql.Tx, key string) (id int64
key,
).Scan(&id, &group, &name)
if err != nil {
if err == sql.ErrNoRows {
if errors.Is(err, sql.ErrNoRows) {
return 0, "", "", common.ErrEndpointNotFound
}
return 0, "", "", err
@ -738,7 +738,7 @@ func (s *Store) getEndpointID(tx *sql.Tx, endpoint *core.Endpoint) (int64, error
var id int64
err := tx.QueryRow("SELECT endpoint_id FROM endpoints WHERE endpoint_key = $1", endpoint.Key()).Scan(&id)
if err != nil {
if err == sql.ErrNoRows {
if errors.Is(err, sql.ErrNoRows) {
return 0, common.ErrEndpointNotFound
}
return 0, err
@ -788,7 +788,7 @@ func (s *Store) getLastEndpointResultSuccessValue(tx *sql.Tx, endpointID int64)
var success bool
err := tx.QueryRow("SELECT success FROM endpoint_results WHERE endpoint_id = $1 ORDER BY endpoint_result_id DESC LIMIT 1", endpointID).Scan(&success)
if err != nil {
if err == sql.ErrNoRows {
if errors.Is(err, sql.ErrNoRows) {
return false, errNoRowsReturned
}
return false, err

View File

@ -50,6 +50,9 @@ func monitor(endpoint *core.Endpoint, alertingConfig *alerting.Config, maintenan
execute(endpoint, alertingConfig, maintenanceConfig, connectivityConfig, disableMonitoringLock, enabledMetrics, debug)
}
}
// Just in case somebody wandered all the way to here and wonders, "what about ExternalEndpoints?"
// Alerting is checked every time an external endpoint is pushed to Gatus, so they're not monitored
// periodically like they are for normal endpoints.
}
func execute(endpoint *core.Endpoint, alertingConfig *alerting.Config, maintenanceConfig *maintenance.Config, connectivityConfig *connectivity.Config, disableMonitoringLock, enabledMetrics, debug bool) {
@ -91,7 +94,7 @@ func execute(endpoint *core.Endpoint, alertingConfig *alerting.Config, maintenan
// UpdateEndpointStatuses updates the slice of endpoint statuses
func UpdateEndpointStatuses(endpoint *core.Endpoint, result *core.Result) {
if err := store.Get().Insert(endpoint, result); err != nil {
log.Println("[watchdog.UpdateEndpointStatuses] Failed to insert data in storage:", err.Error())
log.Println("[watchdog.UpdateEndpointStatuses] Failed to insert result in storage:", err.Error())
}
}

View File

@ -5,12 +5,14 @@
<code id="tooltip-timestamp">{{ prettifyTimestamp(result.timestamp) }}</code>
<div class="tooltip-title">Response time:</div>
<code id="tooltip-response-time">{{ (result.duration / 1000000).toFixed(0) }}ms</code>
<slot v-if="result.conditionResults && result.conditionResults.length">
<div class="tooltip-title">Conditions:</div>
<code id="tooltip-conditions">
<slot v-for="conditionResult in result.conditionResults" :key="conditionResult">
{{ conditionResult.success ? "&#10003;" : "X" }} ~ {{ conditionResult.condition }}<br/>
</slot>
</code>
</slot>
<div id="tooltip-errors-container" v-if="result.errors && result.errors.length">
<div class="tooltip-title">Errors:</div>
<code id="tooltip-errors">

View File

@ -1,6 +1,6 @@
<template>
<router-link to="../"
class="absolute top-2 left-2 inline-block px-2 pb-0.5 text-lg text-black bg-gray-100 rounded hover:bg-gray-200 focus:outline-none border border-gray-200 dark:bg-gray-700 dark:text-gray-200 dark:border-gray-500 dark:hover:bg-gray-600">
class="absolute top-2 left-5 inline-block px-2 pb-0.5 text-sm text-black bg-gray-100 rounded hover:bg-gray-200 focus:outline-none border border-gray-200 dark:bg-gray-700 dark:text-gray-200 dark:border-gray-500 dark:hover:bg-gray-600">
&larr;
</router-link>
<div>
@ -34,7 +34,7 @@
</div>
</div>
</div>
<div v-if="endpointStatus && endpointStatus.key" class="mt-12">
<div v-if="endpointStatus && endpointStatus.key && showResponseTimeChartAndBadges" class="mt-12">
<h1 class="text-xl xl:text-3xl font-mono text-gray-400">RESPONSE TIME</h1>
<hr/>
<img :src="generateResponseTimeChartImageURL()" alt="response time chart" class="mt-6"/>
@ -118,7 +118,6 @@ export default {
response.json().then(data => {
if (JSON.stringify(this.endpointStatus) !== JSON.stringify(data)) {
this.endpointStatus = data;
this.uptime = data.uptime;
let events = [];
for (let i = data.events.length - 1; i >= 0; i--) {
let event = data.events[i];
@ -148,6 +147,15 @@ export default {
events.push(event);
}
this.events = events;
// Check if there's any non-0 response time data
// If there isn't, it's likely an external endpoint, which means we should
// hide the response time chart and badges
for (let i = 0; i < data.results.length; i++) {
if (data.results[i].duration > 0) {
this.showResponseTimeChartAndBadges = true;
break;
}
}
}
});
} else {
@ -183,13 +191,13 @@ export default {
data() {
return {
endpointStatus: {},
uptime: {},
events: [],
hourlyAverageResponseTime: {},
// Since this page isn't at the root, we need to modify the server URL a bit
serverUrl: SERVER_URL === '.' ? '..' : SERVER_URL,
currentPage: 1,
showAverageResponseTime: true,
showResponseTimeChartAndBadges: false,
chartLabels: [],
chartValues: [],
}

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long