The Management gRPC API has too much business logic
happening while it has to be in the Account manager.
This also needs to make more requests to the store
through the account manager.
Bug 1: When calculating the network map, peers added by a setup key
were falling under expiration logic while they shouldn't.
Bug 2: Peers HTTP API didn't return expired peers for non-admin users
because of the expired peer check in the ACL logic.
The fix applies peer expiration checks outside of the ACL logic.
When we delete a peer from an account, we save the account in the file store.
The file store maintains peerID -> accountID and peerKey -> accountID indices.
Those can't be updated when we delete a peer because the store saves the whole account
without a peer already and has no access to the removed peer.
In this PR, we dynamically check if there are stale indices when GetAccountByPeerPubKey
and GetAccountByPeerID.
Goals:
Enable peer login expiration when adding new peer
Expire peer's login when the time comes
The account manager triggers peer expiration routine in future if the
following conditions are true:
peer expiration is enabled for the account
there is at least one peer that has expiration enabled and is connected
The time of the next expiration check is based on the nearest peer expiration.
Account manager finds a peer with the oldest last login (auth) timestamp and
calculates the time when it has to run the routine as a sum of the configured
peer login expiration duration and the peer's last login time.
When triggered, the expiration routine checks whether there are expired peers.
The management server closes the update channel of these peers and updates
network map of other peers to exclude expired peers so that the expired peers
are not able to connect anywhere.
The account manager can reschedule or cancel peer expiration in the following cases:
when admin changes account setting (peer expiration enable/disable)
when admin updates the expiration duration of the account
when admin updates peer expiration (enable/disable)
when peer connects (Sync)
P.S. The network map calculation was updated to exclude peers that have login expired.
Extend HTTP API with Account endpoints to configure global peer login expiration.
GET /api/accounts
PUT /api/account/{id}/
The GET endpoint returns an array of accounts with
always one account in the list. No exceptions.
The PUT endpoint updates account settings:
PeerLoginExpiration and PeerLoginExpirationEnabled.
PeerLoginExpiration is a duration in seconds after which peers' logins will expire.
This PR adds a peer login expiration logic that requires
peers created by a user to re-authenticate (re-login) after
a certain threshold of time (24h by default).
The Account object now has a PeerLoginExpiration
property that indicates the duration after which a peer's
login will expire and a login will be required. Defaults to 24h.
There are two new properties added to the Peer object:
LastLogin that indicates the last time peer successfully used
the Login gRPC endpoint and LoginExpirationEnabled that
enables/disables peer login expiration.
The login expiration logic applies only to peers that were created
by a user and not those that were added with a setup key.
This feature allows using the custom claim in the JWT token as a user ID.
Refactor claims extractor with options support
Add is_current to the user API response
Replace Peer.Key as internal identifier with a randomly generated Peer.ID
in the Management service.
Every group now references peers by ID instead of a public key.
Every route now references peers by ID instead of a public key.
FileStore does store.json file migration on startup by generating Peer.ID and replacing
all Peer.Key identifier references .
Adding --external-ip-map and --dns-resolver-address to up command and shorthand option to global flags.
Refactor get and read config functions with new ConfigInput type.
updated cobra package to latest release.
This PR adds system activity tracking.
The management service records events like
add/remove peer, group, rule, route, etc.
The activity events are stored in the SQLite event store
and can be queried by the HTTP API.
Updated tests, API, and account manager methods
Sync routes to peers in the distribution groups
Added store upgrade by adding the All group to routes that don't have them
Add a usage_limit parameter to the API.
This limits the number of times a setup key
can be used.
usage_limit == 0 indicates the the usage is inlimited.
Use stdout and stderr log path only if on Linux and attempt to create the path
Update status system with FQDN fields and
status command to display the domain names of remote and local peers
Set some DNS logs to tracing
update readme file
Due to peer reconnects when restarting the Management service,
there are lots of SaveStore operations to update peer status.
Store.SavePeerStatus stores peer status separately and the
FileStore implementation stores it in memory.
Added DNS update protocol message
Added sync to clients
Update nameserver API with new fields
Added default NS groups
Added new dns-name flag for the management service append to peer DNS label
This PR simplifies Store and FileStore
by keeping just the Get and Save account methods.
The AccountManager operates mostly around
a single account, so it makes sense to fetch
the whole account object from the store.
This PR brings open-telemetry metrics to the
Management service.
The Management service exposes new HTTP endpoint
/metrics on 8081 port by default.
The port can be changed by specifying
--metrics-port PORT flag when starting the service.
This will help us understand usage on self-hosted deployments
The collection may be disabled by using the flag --disable-anonymous-metrics or
NETBIRD_DISABLE_ANONYMOUS_METRICS in setup.env
This PR brings user invites logic to the Management service
via HTTP API.
The POST /users/ API endpoint creates a new user in the Idp
and then in the local storage.
Once the invited user signs ups, the account invitation is redeemed.
There are a few limitations.
This works only with an enabled IdP manager.
Users that already have a registered account can't be invited.
Add DNS package and Nameserver group objects
Add CRUD operations for Nameserver Groups to account manager
Add Routes and Nameservers to Account Copy method
Run docker tests with timeout and serial flags
Support Generic OAuth 2.0 Device Authorization Grant
as per RFC specification https://www.rfc-editor.org/rfc/rfc8628.
The previous version supported only Auth0 as an IDP backend.
This implementation enables the Interactive SSO Login feature
for any IDP compatible with the specification, e.g., Keycloak.
When creating TLSConfig from provided certificate file, the HTTP/2 support is not enabled.
It works with Certmanager because it adds h2 support.
We enable it the same way when creating TLSConfig from files.
This PR is a part of an effort to use standard ports (443 or 80) that are usually allowed by default in most of the environments.
Right now Management Service runs the Let'sEncrypt manager on port 443, HTTP API server on port 33071,
and a gRPC server on port 33073. There are three separate listeners.
This PR combines these listeners into one.
With this change, the HTTP and gRPC server runs on either 443 with TLS or 80 without TLS
by default (no --port specified).
Let's Encrypt manager always runs on port 443 if enabled.
The backward compatibility server runs on port 33073 (with TLS or without).
HTTP port 33071 is obsolete and not used anymore.
Newly installed agents will connect to port 443 by default instead of port 33073 if not specified otherwise.
Right now Signal Service runs the Let'sEncrypt manager on port 80
and a gRPC server on port 10000. There are two separate listeners.
This PR combines these listeners into one with a cmux lib.
The gRPC server runs on either 443 with TLS or 80 without TLS.
Let's Encrypt manager always runs on port 80.
The Management client will try reconnecting in case.
of network issues or non-permanent errors.
If the device was off-boarded, then the client will stop retrying.
* Send netmask from account network
Added the GetPeerNetwork method to account manager
Pass a copy of the network to the toPeerConfig function
to retrieve the netmask from the network instead of constant
updated methods and added test
* check if the network is the same for 2 peers
* Use expect with BeEquivalentTo
This PR adds support for SSH access through the NetBird network
without managing SSH skeys.
NetBird client app has an embedded SSH server (Linux/Mac only)
and a netbird ssh command.
Introduced an OpenAPI specification.
Updated API handlers to use the specification types.
Added patch operation for rules and groups
and methods to the account manager.
HTTP PUT operations require id, fail if not provided.
Use snake_case for HTTP request and response body
There are a few places where an account is created.
When we create a new account, there should be
some defaults set. E.g. created by and group ALL.
It makes sense to add it in one place to avoid inconsistencies.
* fix(acl): update each peer's network when rule,group or peer changed
* fix(ACL): update network test
* fix(acl): cleanup indexes before update them
* fix(acl): clean up rules indexes only for account
Before this change, NetBird Agent wasn't handling
peer interface configuration changes dynamically.
Also, remote peer configuration changes have
not been applied (e.g. AllowedIPs changed).
Not a very common cause, but still it should be handled.
Now, Agent reacts to PeerConfig changes sent from the
management service and restarts remote connections
if AllowedIps have been changed.
The peer IP allocation logic was allocating sequential peer IP from the 100.64.0.0/10
address block.
Each account is created with a random subnet from 100.64.0.0/10.
The total amount of potential subnets is 64.
The new logic allocates random peer IP
from the account subnet.
This gives us flexibility to add support for
multi subnet accounts without overlapping IPs.
Send Desktop UI client version as user-agent to daemon
This is sent on every login request to the management
Parse the GRPC context on the system package and
retrieves the user-agent
Management receives the new UIVersion field and
store in the Peer's system meta
* rename wiretrustee-signal to netbird-signal
* Rename Signal repositories and source bin
* Adjust docker-compose with signal volume [skip ci]
Co-authored-by: mlsmaycon <mlsmaycon@gmail.com>
Rename documentation and goreleaser build names
Added a migration function for when the old path exists and the new one doesn't
updated the configure.sh to generate the docker-compose with a new path only
if no pre-existing volume with old name exists
UI and CLI Clients are now able to use SSO login by default
we will check if the management has configured or supports SSO providers
daemon will handle fetching and waiting for an access token
Oauth package was moved to internal to avoid one extra package at this stage
Secrets were removed from OAuth
CLI clients have less and better output
2 new status were introduced, NeedsLogin and FailedLogin for better messaging
With NeedsLogin we no longer have endless login attempts
The management will validate the JWT as it does in the API
and will register the Peer to the user's account.
New fields were added to grpc messages in management
and client daemon and its clients were updated
Peer has one new field, UserID,
that will hold the id of the user that registered it
JWT middleware CheckJWT got a splitter
and renamed to support validation for non HTTP requests
Added test for adding new Peer with UserID
Lots of tests update because of a new field
Exposes endpoint under "/users/" that returns information on users.
Calls IDP manager to get information not stored locally (email, name),
which in the case of the managed version is auth0.
* feat(management): add groups
* squash
* feat(management): add handlers for groups
* feat(management): add handlers for groups
* chore(management): add tests for the get group of the management
* chore(management): add tests for save group
* Replace Wiretrustee links and naming
* Upper case for Netbrid in README
* Replace logo
* Dashboard URL to app.netbird.io
Co-authored-by: Misha Bragin <bangvalo@gmail.com>
* Fix IDP Manager config structs with correct tags
When loading the configuration from file
we will use the Auth0ClientConfig and when
sending the post to retrieve a token
we use the auth0JWTRequest with proper tags
Also, removed the idle timeout as it was closing
all idle connections
When account id supplied via claim, we should
handle change of the domain classification.
If category of domain change to private, we
should re-evaluate the private account
* Added Domain Category field and fix store tests
* Add GetAccountByDomain method
* Add Domain Category to authorization claims
* Initial GetAccountWithAuthorizationClaims test cases
* Renamed Private Domain map and index it on saving account
* New Go build tags
* Added NewRegularUser function
* Updated restore to account for primary domain account
Also, added another test case
* Added grouping user of private domains
Also added auxiliary methods for update metadata and domain attributes
* Update http handles get account method and tests
* Fix lint and document another case
* Removed unnecessary log
* Move use cases to method and add flow comments
* Split the new user and existing logic from GetAccountWithAuthorizationClaims
* Review: minor corrections
Co-authored-by: braginini <bangvalo@gmail.com>
* extract claim information from JWT
* get account function
* Store domain
* tests missing domain
* update existing account with domain
* add store domain tests
* test: WIP mocking the grpc server for testing the sending of the client information
* WIP: Test_SystemMetaDataFromClient with mocks, todo:
* fix: failing meta data test
* test: add system meta expectation in management client test
* fix: removing deprecated register function, replacing with new one
* fix: removing deprecated register function from mockclient interface impl
* fix: fixing interface declaration
* chore: remove unused commented code
Co-authored-by: braginini <bangvalo@gmail.com>
* moved wiretrustee version from main to system.info
* added wiretrustee version for all supported platforms
* typo corrected
* refactor: use single WiretrusteeVersion() func to get version of the client
Co-authored-by: braginini <bangvalo@gmail.com>
* update interface tests and configuration messages
* little debug
* little debug on both errors
* print all devs
* list of devices
* debug func
* handle interface close
* debug socks
* debug socks
* if ports match
* use random assigned ports
* remove unused const
* close management client connection when stopping engine
* GracefulStop when management clients are closed
* enable workflows on PRs too
* remove iface_test debug code
* get account id from access token claim
* use GetOrCreateAccountByUser and add test
* correct account id claim
* remove unused account
* Idp manager interface
* auth0 idp manager
* use if instead of switch case
* remove unnecessary lock
* NewAuth0Manager
* move idpmanager to its own package
* update metadata when accountId is not supplied
* update tests with idpmanager field
* format
* new idp manager and config support
* validate if we fetch the interface before converting to string
* split getJWTToken
* improve tests
* proper json fields and handle defer body close
* fix ci lint notes
* documentation and proper defer position
* UpdateUserAppMetadata tests
* update documentation
* ManagerCredentials interface
* Marshal and Unmarshal functions
* fix tests
* ManagerHelper and ManagerHTTPClient
* further tests with mocking
* rename package and custom http client
* sync local packages
* remove idp suffix
* feature: support new management service protocol
* chore: add more logging to track networkmap serial
* refactor: organize peer update code in engine
* chore: fix lint issues
* refactor: extract Signal client interface
* test: add signal client mock
* refactor: introduce Management Service client interface
* chore: place management and signal clients mocks to respective packages
* test: add Serial test to the engine
* fix: lint issues
* test: unit tests for a networkMapUpdate
* test: unit tests Sync update
* feature: introduce NetworkMap to the management protocol with a Serial ID
* test: add Management Sync method protocol test
* test: add Management Sync method NetworkMap field check [FAILING]
* test: add Management Sync method NetworkMap field check [FAILING]
* feature: fill NetworkMap property to when Deleting peer
* feature: fill NetworkMap in the Sync protocol
* test: code review mentions - GeneratePrivateKey() in the test
* fix: wiretrustee client use wireguard GeneratePrivateKey() instead of GenerateKey()
* test: add NetworkMap test
* fix: management_proto test remove store.json on test finish
* chore: [management] - add account serial ID
* Fix concurrency on the client (#183)
* reworked peer connection establishment logic eliminating race conditions and deadlocks while running many peers
* chore: move serial to Network from Account
* feature: increment Network serial ID when adding/removing peers
* chore: extract network struct init to network.go
* chore: add serial test when adding peer to the account
* test: add ModificationID test on AddPeer and DeletePeer
* feature: add User entity to Account
* test: new file store creation test
* test: add FileStore persist-restore tests
* test: add GetOrCreateAccountByUser Accountmanager test
* refactor: rename account manager users file
* refactor: use userId instead of accountId when handling Management HTTP API
* fix: new account creation for every request
* fix: golint
* chore: add account creator to Account Entity to identify who created the account.
* chore: use xid ID generator for account IDs
* fix: test failures
* test: check that CreatedBy is stored when account is stored
* chore: add account copy method
* test: remove test for non existent GetOrCreateAccount func
* chore: add accounts conversion function
* fix: golint
* refactor: simplify admin user creation
* refactor: move migration script to a separate package
* refactor: move goroutine that runs Signal Client Receive to the engine for better control
* chore: fix comments typo
* test: fix golint
* chore: comments update
* chore: consider connection state=READY in signal and management clients
* chore: fix typos
* test: fix signal ping-pong test
* chore: add wait condition to signal client
* refactor: add stream status to the Signal client
* refactor: defer mutex unlock
* feature: add logging to a file
* refactor: move InitLog to util lib
* docs: update signal and management docs
* chore: update docker compose
* set --log-file to console
* chore: comment out log volume in docker compose
Co-authored-by: mlsmaycon <mlsmaycon@gmail.com>
* feature: update STUNs and TURNs in engine
* fix: setup TURN credentials request only when refresh enabled
* feature: update TURNs and STUNs in teh client app on Management update
* chore: disable peer reflexive candidates in ICE
* chore: relocate management.json
* chore: make TURN secret and pwd plain text in config
* abstract peer channel
* remove wip code
* refactor NewServer with Peer updates channel
* feature: add TURN credentials manager
* hmac logic
* example test function
* test: add TimeBasedAuthSecretsManager_GenerateCredentials test
* test: make tests for now with hardcoded secret
* test: add TimeBasedAuthSecretsManager_SetupRefresh test
* test: add TimeBasedAuthSecretsManager_SetupRefresh test
* test: add TimeBasedAuthSecretsManager_CancelRefresh test
* feature: extract TURNConfig to the management config
* feature: return hash based TURN credentials only on initial sync
* feature: make TURN time based secret credentials optional
Co-authored-by: mlsmaycon <mlsmaycon@gmail.com>
* feature: add peer GET and DELETE API methods
* refactor: extract peer business logic to a separate file
* refactor: extract peer business logic to a separate file
* feature: add peer update HTTP endpoint
* chore: fill peer new fields
* merge with main
* refactor: HTTP methods according to standards
* feature: add peer system metadata
* feature: add peer status
* test: fix removal
* feature: add peer GET and DELETE API methods
* refactor: extract peer business logic to a separate file
* refactor: extract peer business logic to a separate file
* feature: add peer update HTTP endpoint
* chore: fill peer new fields
* merge with main
* refactor: HTTP methods according to standards
* chore: setup keys POST endpoint without ID
* feature: replace RegisterPeer with Login method that does both - registration and login
* test: add management login test
* feature: add WiretrusteeConfig to the Login response to configure peer global config
* feature: add client peer login support
* fix: missing parts
* chore: update go deps
* feature: support Management Service gRPC endpoints [CLIENT]
* feature: finalize client sync with management
* fix: management store peer key lower case restore
* fix: management returns peer ip without a mask
* refactor: remove cmd pkg
* fix: invalid tun interface name on mac
* fix: timeout when calling management client
* fix: tests and lint errors
* fix: golang-test workflow
* fix: client service tests
* fix: iface build
* feature: detect management scheme on startup
* chore: better logs for management
* fix: goreleaser
* fix: lint errors
* fix: signal TLS
* fix: direct Wireguard connection
* chore: verbose logging on direct connection
* Add client's interaction with management service
* Getting updates
* Fixed key and nil ptr
* Added setupKey param
* Added managment address parameter
* Fixed test
* feature: use RemotePeers from the management server instead of deprecated Peers
* merge: merge changes from main
* debug image and use wiretrustee/management repository
* Update documentation and docker-compose to include management
* improve documentation and add debug image build
* update docker-compose section with management service notes.
* fix broken doc link
* feature: add config properties to the SyncResponse of the management gRpc service
* fix: lint errors
* chore: modify management protocol according to the review notes
* fix: management proto fields sequence
* feature: add proper peer configuration to be synced
* chore: minor changes
* feature: finalize peer config management
* fix: lint errors
* feature: add management server config file
* refactor: extract hosts-config to a separate file
* refactor: review notes applied to correct file_store usage
* refactor: extract management service configuration to a file
* refactor: simplify management config
* feature: add peer sync and a server public key endpoints
* test: add Management.Sync() gRpc endpoint test
* feat: implement peer sync
* docs: added some comments to the Management server
* chore: use for loop over channel when monitoring peer updates
* fix: exit infinite loop when sending updates to peers
* test: add multiple concurrent peers test for management service
* chore: remove unused test
* fix: reduce the amount peers for a concurrent peer update test
Co-authored-by: braginini <m.bragin@wiretrustee.com>
