Commit Graph

52 Commits

Author SHA1 Message Date
Bethuel Mmbaga
9683da54b0
[management] Refactor nameserver groups to use store methods (#2888) 2024-11-26 17:39:04 +01:00
Bethuel Mmbaga
0e48a772ff
[management] Refactor DNS settings to use store methods (#2883) 2024-11-26 13:43:05 +01:00
Bethuel Mmbaga
f118d81d32
[management] Refactor policy to use store methods (#2878) 2024-11-26 10:46:05 +01:00
Bethuel Mmbaga
ca12bc6953
[management] Refactor posture check to use store methods (#2874) 2024-11-25 16:26:24 +01:00
Bethuel Mmbaga
12f442439a
[management] Refactor group to use store methods (#2867)
* Refactor setup key handling to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add lock to get account groups

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add check for regular user

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* get only required groups for auto-group validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add account lock and return auto groups map on validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor account peers update

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor groups to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor GetGroupByID and add NewGroupNotFoundError

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add AddPeer and RemovePeer methods to Group struct

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Preserve store engine in SqlStore transactions

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Run groups ops in transaction

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix missing group removed from setup key activity

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix sonar

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Change setup key log level to debug for missing group

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Retrieve modified peers once for group events

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add account locking and merge group deletion methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
2024-11-15 20:09:32 +03:00
Bethuel Mmbaga
6cb697eed6
[management] Refactor setup key to use store methods (#2861)
* Refactor setup key handling to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add lock to get account groups

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add check for regular user

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* get only required groups for auto-group validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add account lock and return auto groups map on validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix missing group removed from setup key activity

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Remove context from DB queries

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add user permission check and add setup events into events to store slice

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Retrieve all groups once during setup key auto-group validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix lint

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix sonar

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
2024-11-11 19:46:10 +03:00
pascal-fischer
10480eb52f
[management] Setup key improvements (#2775) 2024-10-28 17:52:23 +01:00
Maycon Santos
da3a053e2b
[management] Refactor getAccountIDWithAuthorizationClaims (#2715)
This change restructures the getAccountIDWithAuthorizationClaims method to improve readability, maintainability, and performance.

- have dedicated methods to handle possible cases
- introduced Store.UpdateAccountDomainAttributes and Store.GetAccountUsers methods
- Remove GetAccount and SaveAccount dependency
- added tests
2024-10-12 08:35:51 +02:00
pascal-fischer
8284ae959c
[management] Move testdata to sql files (#2693) 2024-10-10 12:35:03 +02:00
Bethuel Mmbaga
7f09b39769
[management] Refactor User JWT group sync (#2690)
* Refactor GetAccountIDByUserOrAccountID

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* sync user jwt group changes

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* propagate jwt group changes to peers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix no jwt groups synced

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests and lint

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Move the account peer update outside the transaction

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* move updateUserPeersInGroups to account manager

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* move event store outside of transaction

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* get user with update lock

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Run jwt sync in transaction

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
2024-10-04 17:17:01 +03:00
pascal-fischer
158936fb15
[management] Remove file store (#2689) 2024-10-03 15:50:35 +02:00
Bethuel Mmbaga
acb73bd64a
[management] Remove redundant get account calls in GetAccountFromToken (#2615)
* refactor access control middleware and user access by JWT groups

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor jwt groups extractor

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor handlers to get account when necessary

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor getAccountFromToken

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor getAccountWithAuthorizationClaims

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* revert handles change

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* remove GetUserByID from account manager

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor getAccountWithAuthorizationClaims to return account id

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor handlers to use GetAccountIDFromToken

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* remove locks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add GetGroupByName from store

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add GetGroupByID from store and refactor

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor retrieval of policy and posture checks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor user permissions and retrieves PAT

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor route, setupkey, nameserver and dns to get record(s) from store

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor store

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix lint

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix add missing policy source posture checks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add store lock

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add get account

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
2024-09-27 17:10:50 +03:00
pascal-fischer
6c50b0c84b
[management] Add transaction to addPeer (#2469)
This PR removes the GetAccount and SaveAccount operations from the AddPeer and instead makes use of gorm.Transaction to add the new peer.
2024-09-16 15:47:03 +02:00
pascal-fischer
049b5fb7ed
Split DB calls in peer login (#2439) 2024-08-19 12:50:11 +02:00
Maycon Santos
165988429c
Add write lock for peer when saving its connection status (#2359) 2024-07-31 14:53:32 +02:00
Maycon Santos
1f48fdf6ca
Add SavePeer method to prevent a possible account inconsistency (#2296)
SyncPeer was storing the account with a simple read lock

This change introduces the SavePeer method to the store to be used in these cases
2024-07-26 07:49:05 +02:00
Bethuel Mmbaga
1537b0f5e7
Add batch save/update for groups and users (#2245)
* Add functionality to update multiple users

* Remove SaveUsers from DefaultAccountManager

* Add SaveGroups method to AccountManager interface

* Refactoring

* Add SaveUsers and SaveGroups methods to store interface

* Refactor method SaveAccount to SaveUsers and SaveGroups

The method SaveAccount in user.go and group.go files was split into two separate methods. Now, user-specific data is handled by SaveUsers and group-specific data is handled by SaveGroups method. This provides a cleaner and more efficient way to save user and group data.

* Add account ID to user and group in SqlStore

* Refactor SaveUsers and SaveGroups in store

* Remove unnecessary ID assignment in SaveUsers and SaveGroups
2024-07-15 17:04:06 +03:00
pascal-fischer
765aba2c1c
Add context to throughout the project and update logging (#2209)
propagate context from all the API calls and log request ID, account ID and peer ID

---------

Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>
2024-07-03 11:33:02 +02:00
Bethuel Mmbaga
fc15ee6351
auto migrate older management to sqlite (#2170) 2024-06-20 19:45:57 +02:00
Bethuel Mmbaga
95299be52d
Deprecate FileStore engine support (#2119)
* Removejsonfile' from test matrix in workflows

* Remove sqlite to json migration command

* Refactor store engine implementation to remove JSON file store support

The codebase has been refactored to remove support for JSON file store storage engine, with SQLite serving as the default store engine. New functions have been added to handle unsupported store engines and to migrate data from file store to SQLite.

* Remove 'downCmd' from migration commands

* Refactoring

* Add sqlite cleanup

* Remove comment
2024-06-13 13:39:19 +03:00
pascal-fischer
521f7dd39f
Improve login performance (#2061) 2024-05-31 16:41:12 +02:00
pascal-fischer
012235ff12
Add FindExistingPostureCheck (#2075) 2024-05-30 15:22:42 +02:00
Bethuel Mmbaga
d35a79d3b5
Upgrade gRPC and OpenTelemetry packages for compatibility (#2003)
Upgrades `go.opentelemetry.io/otel` from version` v1.11.1` to `v1.26.0`. The upgrade addresses compatibility issues caused by the removal of several sub-packages in the latest OpenTelemetry release, which were causing broken dependencies.

**Key Changes:**
- Upgraded `go.opentelemetry.io/otel` from `v1.11.1` to `v1.26.0`.

- Fixed broken dependencies by replacing the deprecated sub-packages:
  - `go.opentelemetry.io/otel/metric/instrument`
  - `go.opentelemetry.io/otel/metric/instrument/asyncint64`
  - `go.opentelemetry.io/otel/metric/instrument/syncint64`
  
- Upgraded `google.golang.org/grpc` from `v1.56.3`  to `v1.64.0` which deprecate `Dial` and `DialContext` to `NewClient`.
2024-05-27 08:39:18 +02:00
Bethuel Mmbaga
a5811a2d7d
Implement experimental PostgreSQL store (#1939)
* migrate sqlite store to
 generic sql store

* fix conflicts

* init postgres store

* Add postgres store tests

* Refactor postgres store engine name

* fix tests

* Run postgres store tests on linux only

* fix tests

* Refactor

* cascade policy rules on policy deletion

* fix tests

* run postgres cases in new db

* close store connection after tests

* refactor

* using testcontainers

* sync go sum

* remove postgres service

* remove store cleanup

* go mod tidy

* remove env

* use postgres as engine and initialize test store with testcontainer

---------

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>
2024-05-16 19:28:37 +03:00
pascal-fischer
2e0047daea
Improve Sync performance (#1901) 2024-05-07 14:30:03 +02:00
Maycon Santos
fc7c1e397f
Disable force jsonfile variable (#1611)
This enables windows management tests

Added another DNS server to the dns server tests
2024-03-15 10:50:02 +01:00
Viktor Liu
199bf73103
Remove usage stats (#1665) 2024-03-05 09:45:32 +01:00
Yury Gargay
bbea4c3cc3
Use SQLite store as default when running tests when env is not set (#1612) 2024-02-22 16:51:56 +01:00
Viktor Liu
b7a6cbfaa5
Add account usage logic (#1567)
---------

Co-authored-by: Yury Gargay <yury.gargay@gmail.com>
2024-02-22 12:27:08 +01:00
Yury Gargay
0fbf72434e
Make SQLite default for new installations (#1529)
* Make SQLite default for new installations

* if var is not set, return empty string

this allows getStoreEngineFromDatadir to detect json store files

---------

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>
2024-02-20 15:06:32 +01:00
Yury Gargay
9bc7b9e897
Add initial support of device posture checks (#1540)
This PR implements the following posture checks:

* Agent minimum version allowed
* OS minimum version allowed
* Geo-location based on connection IP

For the geo-based location, we rely on GeoLite2 databases which are free IP geolocation databases. MaxMind was tested and we provide a script that easily allows to download of all necessary files, see infrastructure_files/download-geolite2.sh.

The OpenAPI spec should extensively cover the life cycle of current version posture checks.
2024-02-20 09:59:56 +01:00
pascal-fischer
141065f14e
Merge branch 'main' into feature/peer-approval 2023-11-29 16:27:01 +01:00
Maycon Santos
c2eaf8a1c0
Add account deletion endpoint (#1331)
Adding support to account owners to delete an account

This will remove all users from local, and if --user-delete-from-idp is set it will remove from the remote IDP
2023-11-28 14:23:38 +01:00
Pascal Fischer
a729c83b06 extract peer into seperate package 2023-11-28 13:45:26 +01:00
Maycon Santos
06318a15e1
Log store engine type (#1234) 2023-10-19 21:14:05 +02:00
Yury Gargay
46f5f148da
Move StoreKind under own StoreConfig configuration and rename to Engine (#1219)
* Move StoreKind under own StoreConfig configuration parameter

* Rename StoreKind option to Engine

* Rename StoreKind internal methods and types to Engine

* Add template engine value test

---------

Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>
2023-10-16 11:19:39 +02:00
Yury Gargay
32880c56a4
Implement SQLite Store using gorm and relational approach (#1065)
Restructure data handling for improved performance and flexibility. 
Introduce 'G'-prefixed fields to represent Gorm relations, simplifying resource management. 
Eliminate complexity in lookup tables for enhanced query and write speed. 
Enable independent operations on data structures, requiring adjustments in the Store interface and Account Manager.
2023-10-12 15:42:36 +02:00
pascal-fischer
da75a76d41
Adding dashboard login activity (#1092)
For better auditing this PR adds a dashboard login event to the management service.

For that the user object was extended with a field for last login that is not actively saved to the database but kept in memory until next write. The information about the last login can be extracted from the JWT claims nb_last_login. This timestamp will be stored and compared on each API request. If the value changes we generate an event to inform about a login.
2023-08-18 19:23:11 +02:00
Pascal Fischer
e30def175b switch PATs to map and add deletion 2023-03-20 16:14:55 +01:00
Pascal Fischer
b8cab2882b storing and retrieving PATs 2023-03-16 15:57:44 +01:00
Misha Bragin
9adadfade4
Use Peer.ID instead of Peer.Key as peer identifier (#664)
Replace Peer.Key as internal identifier with a randomly generated Peer.ID 
in the Management service.
Every group now references peers by ID instead of a public key.
Every route now references peers by ID instead of a public key.
FileStore does store.json file migration on startup by generating Peer.ID and replacing
all Peer.Key identifier references .
2023-02-03 10:33:28 +01:00
Misha Bragin
f37b43a542
Save Peer Status separately in the FileStore (#554)
Due to peer reconnects when restarting the Management service,
there are lots of SaveStore operations to update peer status.

Store.SavePeerStatus stores peer status separately and the
FileStore implementation stores it in memory.
2022-11-08 10:46:12 +01:00
Misha Bragin
ed7ac81027
Introduce locking on the account level (#548) 2022-11-07 17:52:23 +01:00
Misha Bragin
d0c6d88971
Simplified Store Interface (#545)
This PR simplifies Store and FileStore
by keeping just the Get and Save account methods.

The AccountManager operates mostly around
a single account, so it makes sense to fetch
the whole account object from the store.
2022-11-07 12:10:56 +01:00
Maycon Santos
04e4407ea7
Add anonymous usage metrics collection (#508)
This will help us understand usage on self-hosted deployments

The collection may be disabled by using the flag --disable-anonymous-metrics or 
NETBIRD_DISABLE_ANONYMOUS_METRICS in setup.env
2022-10-16 13:33:46 +02:00
Maycon Santos
4b34a6d6df
Add routing support to management service (#424)
Management will receive and store routes that are associated with a peer ID.
The routes are distributed to peers according to their ACLs.
2022-08-18 18:22:15 +02:00
Givi Khojanashvili
3ce3ccc39a
Add rules for ACL (#306)
Add rules HTTP endpoint for frontend - CRUD operations.
Add Default rule - allow all.
Send network map to peers based on rules.
2022-05-21 15:21:39 +02:00
Maycon Santos
0b8387bd2c
Group users of same private domain (#243)
* Added Domain Category field and fix store tests

* Add GetAccountByDomain method

* Add Domain Category to authorization claims

* Initial GetAccountWithAuthorizationClaims test cases

* Renamed Private Domain map and index it on saving account

* New Go build tags

* Added NewRegularUser function

* Updated restore to account for primary domain account

Also, added another test case

* Added grouping user of private domains

Also added auxiliary methods for update metadata and domain attributes

* Update http handles get account method and tests

* Fix lint and document another case

* Removed unnecessary log

* Move use cases to method and add flow comments

* Split the new user and existing logic from GetAccountWithAuthorizationClaims

* Review: minor corrections

Co-authored-by: braginini <bangvalo@gmail.com>
2022-03-01 15:22:18 +01:00
Mikhail Bragin
6ae27c9a9b
Refactor: support multiple users under the same account (#170)
* feature: add User entity to Account

* test: new file store creation test

* test: add FileStore persist-restore tests

* test: add GetOrCreateAccountByUser Accountmanager test

* refactor: rename account manager users file

* refactor: use userId instead of accountId when handling Management HTTP API

* fix: new account creation for every request

* fix: golint

* chore: add account creator to Account Entity to identify who created the account.

* chore: use xid ID generator for account IDs

* fix: test failures

* test: check that CreatedBy is stored when account is stored

* chore: add account copy method

* test: remove test for non existent GetOrCreateAccount func

* chore: add accounts conversion function

* fix: golint

* refactor: simplify admin user creation

* refactor: move migration script to a separate package
2021-12-27 13:17:15 +01:00
Mikhail Bragin
ec759bc461
Delete peer (#114)
* feature: add peer deletion

* feature: add peer deletion [CLIENT]

* fix: lint error

* test: fix sync block

* test: fix management test

* feature: add client stop after was deleted

* chore: remove permission denied cancellation

* chore: add larger signal backoff

* feature: notify deleted peer of removal

* fix: lint issue

* chore: add 2nd default key - one off

* test: fix account key check
2021-09-07 18:36:46 +02:00