Restructure data handling for improved performance and flexibility.
Introduce 'G'-prefixed fields to represent Gorm relations, simplifying resource management.
Eliminate complexity in lookup tables for enhanced query and write speed.
Enable independent operations on data structures, requiring adjustments in the Store interface and Account Manager.
Fix group delete panic
In case if in the db the DNSSettings is null then can cause panic in delete group function
because this field is pointer and it was not checked. Because of in the future implementation
this variable will be filled in any case then make no sense to keep the pointer type.
Fix DNSSettings copy function
With this change, we don't need to update all peers on startup. We will
check the existence of an update channel when returning a list or single peer on API.
Then after restarting of server consumers of API will see peer not
connected status till the creation of an updated channel which indicates
peer successful connection.
* Retrieve all workspace users via pagination, excluding custom user attributes
* Retrieve all authentik users via pagination
* Retrieve all Azure AD users via pagination
* Simplify user data appending operation
Reduced unnecessary iteration and used an efficient way to append all users to 'indexedUsers'
* Fix ineffectual assignment to reqURL
* Retrieve all Okta users via pagination
* Add missing GetAccount metrics
* Refactor
* minimize memory allocation
Refactored the memory allocation for the 'users' slice in the Okta IDP code. Previously, the slice was only initialized but not given a size. Now the size of userList is utilized to optimize memory allocation, reducing potential slice resizing and memory re-allocation costs while appending users.
* Add logging for entries received from IdP management
Added informative and debug logging statements in account.go file. Logging has been added to identify the number of entries received from Identity Provider (IdP) management. This will aid in tracking and debugging any potential data ingestion issues.
This PR fixes an issue were only one route containing routing groups was being synced to peers.
It also prevents sending routes for peers that aren't connect via ACL.
Moved all checks to Account.getEnabledAndDisabledRoutesByPeer.
Co-authored-by: Yury Gargay <yury.gargay@gmail.com>
Co-authored-by: braginini <bangvalo@gmail.com>
This pull request modifies the IdP and cache manager(s) to prevent the sending of app metadata
to the upstream IDP on self-hosted instances.
As a result, the IdP will now load all users from the IdP without filtering based on accountID.
We disable user invites as the administrator's own IDP system manages them.
If there is a difference between local and cached data, we trigger a cache refresh;
as we remove users from the local store and potentially from the remote IDP,
we need to switch the source of truth to the local store to prevent unwanted endless
cache for cases where the removal from the IDP fails or for cases
where the userDeleteFromIDPEnabled got enabled after the first user deletion.
This commit enhances the functionality of the network routes endpoint by introducing a new parameter called `peers_group`. This addition allows users to associate network routes with specific peer groups, simplifying the management and distribution of routes within a network.
Implement user deletion across all IDP-ss. Expires all user peers
when the user is deleted. Users are permanently removed from a local
store, but in IDP, we remove Netbird attributes for the user
untilUserDeleteFromIDPEnabled setting is not enabled.
To test, an admin user should remove any additional users.
Until the UI incorporates this feature, use a curl DELETE request
targeting the /users/<USER_ID> management endpoint. Note that this
request only removes user attributes and doesn't trigger a delete
from the IDP.
To enable user removal from the IdP, set UserDeleteFromIDPEnabled
to true in account settings. Until we have a UI for this, make this
change directly in the store file.
Store the deleted email addresses in encrypted in activity store.
This PR showcases the implementation of additional linter rules. I've updated the golangci-lint GitHub Actions to the latest available version. This update makes sure that the tool works the same way locally - assuming being updated regularly - and with the GitHub Actions.
I've also taken care of keeping all the GitHub Actions up to date, which helps our code stay current. But there's one part, goreleaser that's a bit tricky to test on our computers. So, it's important to take a close look at that.
To make it easier to understand what I've done, I've made separate changes for each thing that the new linters found. This should help the people reviewing the changes see what's going on more clearly. Some of the changes might not be obvious at first glance.
Things to consider for the future
CI runs on Ubuntu so the static analysis only happens for Linux. Consider running it for the rest: Darwin, Windows
The ephemeral manager keep the inactive ephemeral peers in a linked list. The manager schedule a cleanup procedure to the head of the linked list (to the most deprecated peer). At the end of cleanup schedule the next cleanup to the new head.
If a device connect back to the server the manager will remote it from the peers list.
For better auditing this PR adds a dashboard login event to the management service.
For that the user object was extended with a field for last login that is not actively saved to the database but kept in memory until next write. The information about the last login can be extracted from the JWT claims nb_last_login. This timestamp will be stored and compared on each API request. If the value changes we generate an event to inform about a login.
For peer propagation this commit triggers
network map update in two cases:
1) peer login
2) user AutoGroups update
Also it issues new activity message about new user group
for peer login process.
Previous implementation only adds JWT groups to user. This fix also
removes JWT groups from user auto assign groups.
Pelase note, it also happen when user works with dashboard.
Enhancements to Peer Group Assignment:
1. Auto-assigned groups are now applied to all peers every time a user logs into the network.
2. Feature activation is available in the account settings.
3. API modifications included to support these changes for account settings updates.
4. If propagation is enabled, updates to a user's auto-assigned groups are immediately reflected across all user peers.
5. With the JWT group sync feature active, auto-assigned groups are forcefully updated whenever a peer logs in using user credentials.
* Check links of groups before delete it
* Add delete group handler test
* Rename dns error msg
* Add delete group test
* Remove rule check
The policy cover this scenario
* Fix test
* Check disabled management grps
* Change error message
* Add new activity for group delete event
The new functionality allows blocking a user in the Management service.
Blocked users lose access to the Dashboard, aren't able to modify the network map,
and all of their connected devices disconnect and are set to the "login expired" state.
Technically all above was achieved with the updated PUT /api/users endpoint,
that was extended with the is_blocked field.
When peer login expires, all remote peers are updated to exclude the peer from connecting.
Once a peer re-authenticates, the remote peers are not updated.
This peer fixes the behavior.
The peer login expiration ACL check introduced in #714
filters out peers that are expired and agents receive a network map
without that expired peers.
However, the agents should see those peers in status "Disconnected".
This PR extends the Agent <-> Management protocol
by introducing a new field OfflinePeers
that contain expired peers. Agents keep track of those and display
then just in the Status response.
The Management gRPC API has too much business logic
happening while it has to be in the Account manager.
This also needs to make more requests to the store
through the account manager.
Goals:
Enable peer login expiration when adding new peer
Expire peer's login when the time comes
The account manager triggers peer expiration routine in future if the
following conditions are true:
peer expiration is enabled for the account
there is at least one peer that has expiration enabled and is connected
The time of the next expiration check is based on the nearest peer expiration.
Account manager finds a peer with the oldest last login (auth) timestamp and
calculates the time when it has to run the routine as a sum of the configured
peer login expiration duration and the peer's last login time.
When triggered, the expiration routine checks whether there are expired peers.
The management server closes the update channel of these peers and updates
network map of other peers to exclude expired peers so that the expired peers
are not able to connect anywhere.
The account manager can reschedule or cancel peer expiration in the following cases:
when admin changes account setting (peer expiration enable/disable)
when admin updates the expiration duration of the account
when admin updates peer expiration (enable/disable)
when peer connects (Sync)
P.S. The network map calculation was updated to exclude peers that have login expired.
Extend HTTP API with Account endpoints to configure global peer login expiration.
GET /api/accounts
PUT /api/account/{id}/
The GET endpoint returns an array of accounts with
always one account in the list. No exceptions.
The PUT endpoint updates account settings:
PeerLoginExpiration and PeerLoginExpirationEnabled.
PeerLoginExpiration is a duration in seconds after which peers' logins will expire.
This PR adds a peer login expiration logic that requires
peers created by a user to re-authenticate (re-login) after
a certain threshold of time (24h by default).
The Account object now has a PeerLoginExpiration
property that indicates the duration after which a peer's
login will expire and a login will be required. Defaults to 24h.
There are two new properties added to the Peer object:
LastLogin that indicates the last time peer successfully used
the Login gRPC endpoint and LoginExpirationEnabled that
enables/disables peer login expiration.
The login expiration logic applies only to peers that were created
by a user and not those that were added with a setup key.
This feature allows using the custom claim in the JWT token as a user ID.
Refactor claims extractor with options support
Add is_current to the user API response
Replace Peer.Key as internal identifier with a randomly generated Peer.ID
in the Management service.
Every group now references peers by ID instead of a public key.
Every route now references peers by ID instead of a public key.
FileStore does store.json file migration on startup by generating Peer.ID and replacing
all Peer.Key identifier references .
This PR adds system activity tracking.
The management service records events like
add/remove peer, group, rule, route, etc.
The activity events are stored in the SQLite event store
and can be queried by the HTTP API.
Updated tests, API, and account manager methods
Sync routes to peers in the distribution groups
Added store upgrade by adding the All group to routes that don't have them
Add a usage_limit parameter to the API.
This limits the number of times a setup key
can be used.
usage_limit == 0 indicates the the usage is inlimited.
Added DNS update protocol message
Added sync to clients
Update nameserver API with new fields
Added default NS groups
Added new dns-name flag for the management service append to peer DNS label
This PR simplifies Store and FileStore
by keeping just the Get and Save account methods.
The AccountManager operates mostly around
a single account, so it makes sense to fetch
the whole account object from the store.
This PR brings user invites logic to the Management service
via HTTP API.
The POST /users/ API endpoint creates a new user in the Idp
and then in the local storage.
Once the invited user signs ups, the account invitation is redeemed.
There are a few limitations.
This works only with an enabled IdP manager.
Users that already have a registered account can't be invited.
Add DNS package and Nameserver group objects
Add CRUD operations for Nameserver Groups to account manager
Add Routes and Nameservers to Account Copy method
Run docker tests with timeout and serial flags
* Send netmask from account network
Added the GetPeerNetwork method to account manager
Pass a copy of the network to the toPeerConfig function
to retrieve the netmask from the network instead of constant
updated methods and added test
* check if the network is the same for 2 peers
* Use expect with BeEquivalentTo
This PR adds support for SSH access through the NetBird network
without managing SSH skeys.
NetBird client app has an embedded SSH server (Linux/Mac only)
and a netbird ssh command.
Introduced an OpenAPI specification.
Updated API handlers to use the specification types.
Added patch operation for rules and groups
and methods to the account manager.
HTTP PUT operations require id, fail if not provided.
Use snake_case for HTTP request and response body
There are a few places where an account is created.
When we create a new account, there should be
some defaults set. E.g. created by and group ALL.
It makes sense to add it in one place to avoid inconsistencies.
The management will validate the JWT as it does in the API
and will register the Peer to the user's account.
New fields were added to grpc messages in management
and client daemon and its clients were updated
Peer has one new field, UserID,
that will hold the id of the user that registered it
JWT middleware CheckJWT got a splitter
and renamed to support validation for non HTTP requests
Added test for adding new Peer with UserID
Lots of tests update because of a new field
Exposes endpoint under "/users/" that returns information on users.
Calls IDP manager to get information not stored locally (email, name),
which in the case of the managed version is auth0.
* feat(management): add groups
* squash
* feat(management): add handlers for groups
* feat(management): add handlers for groups
* chore(management): add tests for the get group of the management
* chore(management): add tests for save group
When account id supplied via claim, we should
handle change of the domain classification.
If category of domain change to private, we
should re-evaluate the private account
* Added Domain Category field and fix store tests
* Add GetAccountByDomain method
* Add Domain Category to authorization claims
* Initial GetAccountWithAuthorizationClaims test cases
* Renamed Private Domain map and index it on saving account
* New Go build tags
* Added NewRegularUser function
* Updated restore to account for primary domain account
Also, added another test case
* Added grouping user of private domains
Also added auxiliary methods for update metadata and domain attributes
* Update http handles get account method and tests
* Fix lint and document another case
* Removed unnecessary log
* Move use cases to method and add flow comments
* Split the new user and existing logic from GetAccountWithAuthorizationClaims
* Review: minor corrections
Co-authored-by: braginini <bangvalo@gmail.com>
* extract claim information from JWT
* get account function
* Store domain
* tests missing domain
* update existing account with domain
* add store domain tests
* get account id from access token claim
* use GetOrCreateAccountByUser and add test
* correct account id claim
* remove unused account
* Idp manager interface
* auth0 idp manager
* use if instead of switch case
* remove unnecessary lock
* NewAuth0Manager
* move idpmanager to its own package
* update metadata when accountId is not supplied
* update tests with idpmanager field
* format
* new idp manager and config support
* validate if we fetch the interface before converting to string
* split getJWTToken
* improve tests
* proper json fields and handle defer body close
* fix ci lint notes
* documentation and proper defer position
* UpdateUserAppMetadata tests
* update documentation
* ManagerCredentials interface
* Marshal and Unmarshal functions
* fix tests
* ManagerHelper and ManagerHTTPClient
* further tests with mocking
* rename package and custom http client
* sync local packages
* remove idp suffix
* chore: [management] - add account serial ID
* Fix concurrency on the client (#183)
* reworked peer connection establishment logic eliminating race conditions and deadlocks while running many peers
* chore: move serial to Network from Account
* feature: increment Network serial ID when adding/removing peers
* chore: extract network struct init to network.go
* chore: add serial test when adding peer to the account
* test: add ModificationID test on AddPeer and DeletePeer
* feature: add User entity to Account
* test: new file store creation test
* test: add FileStore persist-restore tests
* test: add GetOrCreateAccountByUser Accountmanager test
* refactor: rename account manager users file
* refactor: use userId instead of accountId when handling Management HTTP API
* fix: new account creation for every request
* fix: golint
* chore: add account creator to Account Entity to identify who created the account.
* chore: use xid ID generator for account IDs
* fix: test failures
* test: check that CreatedBy is stored when account is stored
* chore: add account copy method
* test: remove test for non existent GetOrCreateAccount func
* chore: add accounts conversion function
* fix: golint
* refactor: simplify admin user creation
* refactor: move migration script to a separate package
* feature: add peer GET and DELETE API methods
* refactor: extract peer business logic to a separate file
* refactor: extract peer business logic to a separate file
* feature: add peer update HTTP endpoint
* chore: fill peer new fields
* merge with main
* refactor: HTTP methods according to standards
* chore: setup keys POST endpoint without ID
* feature: replace RegisterPeer with Login method that does both - registration and login
* test: add management login test
* feature: add WiretrusteeConfig to the Login response to configure peer global config
* feature: add client peer login support
* fix: missing parts
* chore: update go deps
* feature: support Management Service gRPC endpoints [CLIENT]
* feature: finalize client sync with management
* fix: management store peer key lower case restore
* fix: management returns peer ip without a mask
* refactor: remove cmd pkg
* fix: invalid tun interface name on mac
* fix: timeout when calling management client
* fix: tests and lint errors
* fix: golang-test workflow
* fix: client service tests
* fix: iface build
* feature: detect management scheme on startup
* chore: better logs for management
* fix: goreleaser
* fix: lint errors
* fix: signal TLS
* fix: direct Wireguard connection
* chore: verbose logging on direct connection
* Add client's interaction with management service
* Getting updates
* Fixed key and nil ptr
* Added setupKey param
* Added managment address parameter
* Fixed test
* feature: use RemotePeers from the management server instead of deprecated Peers
* merge: merge changes from main
* feature: add config properties to the SyncResponse of the management gRpc service
* fix: lint errors
* chore: modify management protocol according to the review notes
* fix: management proto fields sequence
* feature: add proper peer configuration to be synced
* chore: minor changes
* feature: finalize peer config management
* fix: lint errors
* feature: add management server config file
* refactor: extract hosts-config to a separate file
* refactor: review notes applied to correct file_store usage
* refactor: extract management service configuration to a file
* refactor: simplify management config
