netbird/management
Bethuel Mmbaga 7bda385e1b
[management] Optimize network map updates (#2718)
* Skip peer update on unchanged network map (#2236)

* Enhance network updates by skipping unchanged messages

Optimizes the network update process
by skipping updates where no changes in the peer update message received.

* Add unit tests

* add locks

* Improve concurrency and update peer message handling

* Refactor account manager network update tests

* fix test

* Fix inverted network map update condition

* Add default group and policy to test data

* Run peer updates in a separate goroutine

* Refactor

* Refactor lock

* Fix peers update by including NetworkMap and posture Checks

* go mod tidy

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* [management] Skip account peers update if no changes affect peers (#2310)

* Remove incrementing network serial and updating peers after group deletion

* Update account peer if posture check is linked to policy

* Remove account peers update on saving setup key

* Refactor group link checking into re-usable functions

* Add HasPeers function to group

* Refactor group management

* Optimize group change effects on account peers

* Update account peers if ns group has peers

* Refactor group changes

* Optimize account peers update in DNS settings

* Optimize update of account peers on jwt groups sync

* Refactor peer account updates for efficiency

* Optimize peer update on user deletion and changes

* Remove condition check for network serial update

* Optimize account peers updates on route changes

* Remove UpdatePeerSSHKey method

* Remove unused isPolicyRuleGroupsEmpty

* Add tests for peer update behavior on posture check changes

* Add tests for peer update behavior on policy changes

* Add tests for peer update behavior on group changes

* Add tests for peer update behavior on dns settings changes

* Refactor

* Add tests for peer update behavior on name server changes

* Add tests for peer update behavior on user changes

* Add tests for peer update behavior on route changes

* fix tests

* Add tests for peer update behavior on setup key changes

* Add tests for peer update behavior on peers changes

* fix merge

* Fix tests

* go mod tidy

* Add NameServer and Route comparators

* Update network map diff logic with custom comparators

* Add tests

* Refactor duplicate diff handling logic

* fix linter

* fix tests

* Refactor policy group handling and update logic.

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Update route check by checking if group has peers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor posture check policy linking logic

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Simplify peer update condition in DNS management

Refactor the condition for updating account peers to remove redundant checks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add policy tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add posture checks tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix user and setup key tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix account and route tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix typo

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix nameserver tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix routes tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix group tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* upgrade diff package

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix nameserver tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* use generic differ for netip.Addr and netip.Prefix

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* go mod tidy

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add peer tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix management suite tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix postgres tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* enable diff nil structs comparison

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* skip the update only last sent the serial is larger

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor peer and user

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* skip spell check for groupD

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor group, ns group, policy and posture checks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* skip spell check for GroupD

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update account policy check before verifying policy status

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Update management/server/route_test.go

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>

* Update management/server/route_test.go

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>

* Update management/server/route_test.go

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>

* Update management/server/route_test.go

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>

* Update management/server/route_test.go

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>

* add tests missing tests for dns setting groups

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add tests for posture checks changes

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add ns group and policy tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add route and group tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* increase Linux test timeout to 10 minutes

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Run diff for client posture checks only

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add panic recovery and detailed logging in peer update comparison

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>
2024-10-23 13:05:02 +03:00
..
client [management] Move testdata to sql files (#2693) 2024-10-10 12:35:03 +02:00
cmd Add support to envsub go management configurations (#2708) 2024-10-09 20:46:23 +02:00
domain Release 0.28.0 (#2092) 2024-06-13 13:24:24 +02:00
proto [management, client] Add access control support to network routes (#2100) 2024-10-02 13:41:00 +02:00
server [management] Optimize network map updates (#2718) 2024-10-23 13:05:02 +03:00
Dockerfile Update management base docker image (#2687) 2024-10-02 19:29:51 +03:00
Dockerfile.debug Update management base docker image (#2687) 2024-10-02 19:29:51 +03:00
main.go Rename module to netbirdio/netbird (#288) 2022-03-26 12:08:54 +01:00
README.md Rebrand client cli (#320) 2022-05-22 18:53:47 +02:00

netbird Management Server

netbird management server will control and synchronize peers configuration within your Netbird account and network.

Command Options

The CLI accepts the command management with the following options:

start Netbird Management Server

Usage:
  netbird-mgmt management [flags]

Flags:
      --cert-file string            Location of your SSL certificate. Can be used when you have an existing certificate and don't want a new certificate be generated automatically. If letsencrypt-domain is specified this property has no effect
      --cert-key string             Location of your SSL certificate private key. Can be used when you have an existing certificate and don't want a new certificate be generated automatically. If letsencrypt-domain is specified this property has no effect
      --datadir string              server data directory location
  -h, --help                        help for management
      --letsencrypt-domain string   a domain to issue Let's Encrypt certificate for. Enables TLS using Let's Encrypt. Will fetch and renew certificate, and run the server with TLS
      --port int                    server port to listen on (default 33073)

Global Flags:
      --config string      Netbird config file location to write new config to (default "/etc/netbird")
      --log-file string    sets Netbird log path. If console is specified the the log will be output to stdout (default "/var/log/netbird/management.log")
      --log-level string    (default "info")

Run Management service (Docker)

You can run service in 2 modes - with TLS or without (not recommended).

Run with TLS (Let's Encrypt).

By specifying the --letsencrypt-domain the daemon will handle SSL certificate request and configuration.

In the following example 33073 is the management service default port, and 443 will be used as port for Let's Encrypt challenge and HTTP API.

The server where you are running a container has to have a public IP (for Let's Encrypt certificate challenge).

Replace with your server's public domain (e.g. mydomain.com or subdomain sub.mydomain.com).

# create a volume
docker volume create netbird-mgmt
# run the docker container
docker run -d --name netbird-management \
-p 33073:33073  \
-p 443:443  \
-v netbird-mgmt:/var/lib/netbird  \
-v ./config.json:/etc/netbird/config.json  \
netbirdio/management:latest \
--letsencrypt-domain <YOUR-DOMAIN>

An example of config.json can be found here management.json

Trigger Let's encrypt certificate generation:

curl https://<YOUR-DOMAIN>

The certificate will be persisted in the datadir/letsencrypt/ folder (e.g. /var/lib/netbird/letsencrypt/) inside the container.

Make sure that the datadir is mapped to some folder on a host machine. In case you used the volume command, you can run the following to retrieve the Mountpoint:

docker volume inspect netbird-mgmt
[
    {
        "CreatedAt": "2021-07-25T20:45:28Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/mgmt/_data",
        "Name": "netbird-mgmt",
        "Options": {},
        "Scope": "local"
    }
]

Consequent restarts of the container will pick up previously generated certificate so there is no need to trigger certificate generation with the curl command on every restart.

Run without TLS.

# create a volume
docker volume create netbird-mgmt
# run the docker container
docker run -d --name netbird-management \
-p 33073:33073  \
-v netbird-mgmt:/var/lib/netbird  \
-v ./config.json:/etc/netbird/config.json  \
netbirdio/management:latest

Debug tag

We also publish a docker image with the debug tag which has the log-level set to default, plus it uses the gcr.io/distroless/base:debug image that can be used with docker exec in order to run some commands in the Management container.

shell $ docker run -d --name netbird-management-debug \
-p 33073:33073  \
-v netbird-mgmt:/var/lib/netbird  \
-v ./config.json:/etc/netbird/config.json  \
netbirdio/management:debug-latest

shell $ docker exec -ti netbird-management-debug /bin/sh
container-shell $ 

For development purposes:

Install golang gRpc tools:

#!/bin/bash
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1

Generate gRpc code:

#!/bin/bash
protoc -I proto/ proto/management.proto --go_out=. --go-grpc_out=.