Updated self-hosted scripts and documentation (#249)

* Updated self-hosted scripts and documentation

Added more variables to setup.env and
Updated the documentation.

We are now configuring turn server
with template as well.

* Updated self-hosted scripts and documentation

Added more variables to setup.env and
Updated the documentation.

We are now configuring turn server
with template as well.

* Updated self-hosted scripts and documentation

Added more variables to setup.env and
Updated the documentation.

We are now configuring turn server
with template as well.

* Updated self-hosted scripts and documentation

Added more variables to setup.env and
Updated the documentation.

We are now configuring turn server
with template as well.

docs/self-hosting.md

@@ -6,7 +6,7 @@ a 3rd party open-source STUN/TURN service [Coturn](https://github.com/coturn/cot
 
 All the components can be self-hosted except for the Auth0 service.
 We chose Auth0 to "outsource" the user management part of the platform because we believe that implementing a proper user auth requires significant amount of time to make it right. 
-We focused on connectivity instead.
+We focused on connectivity instead. It also offers an always free plan that should be ok for most users as its limits are high enough for most teams.
 
 If you would like to learn more about the architecture please refer to the [Wiretrustee Architecture section](architecture.md).
 
@@ -17,10 +17,11 @@ If you would like to learn more about the architecture please refer to the [Wire
 ### Requirements
 
 - Virtual machine offered by any cloud provider (e.g., AWS, DigitalOcean, Hetzner, Google Cloud, Azure ...). 
-- Any Linux OS.
+- Any Unix OS.
 - Docker Compose installed (see [Install Docker Compose](https://docs.docker.com/compose/install/)).
 - Domain name pointing to the public IP address of your server.
-- Open ports ```443, 33071, 33073, 10000, 3478``` (Dashboard, Management HTTP API, Management gRpc API, Signal gRpc, Coturn STUN/TURN respectively) on your server.
+- Wiretrustee Open ports ```443, 33071, 33073, 10000``` (Dashboard, Management HTTP API, Management gRpc API, Signal gRpc) on your server. 
+- Coturn is used for relay using the STUN/TURN protocols. It requires a listening port, ```UDP 3478```,  and range of ports,```UDP 49152-65535```, for dynamic relay connections.
 - Maybe a cup of coffee or tea :)
 
 ### Step-by-step guide
@@ -41,7 +42,7 @@ For this tutorial we will be using domain ```test.wiretrustee.com``` which point
    ```
 3. Prepare configuration files.
    
-   To simplify the setup we have prepared a script to substitute required properties in the [docker-compose.yml.tmpl](../infrastructure_files/docker-compose.yml.tmpl) and [management.json.tmpl](../infrastructure_files/management.json.tmpl) files.
+   To simplify the setup we have prepared a script to substitute required properties in the [turnserver.conf.tmpl](../infrastructure_files/turnserver.conf.tmpl),[docker-compose.yml.tmpl](../infrastructure_files/docker-compose.yml.tmpl) and [management.json.tmpl](../infrastructure_files/management.json.tmpl) files.
    
    The [setup.env](../infrastructure_files/setup.env) file contains the following properties that have to be filled:
    
@@ -57,8 +58,9 @@ For this tutorial we will be using domain ```test.wiretrustee.com``` which point
    # e.g. hello@mydomain.com
    WIRETRUSTEE_LETSENCRYPT_EMAIL=""
    ```
+   > Other options are available, but they are automatically updated.
    
-   Please follow the steps to get the values.
+   Please follow the steps to get the values. 
 
 4. Configure ```WIRETRUSTEE_AUTH0_DOMAIN``` ```WIRETRUSTEE_AUTH0_CLIENT_ID``` ```WIRETRUSTEE_AUTH0_AUDIENCE``` properties.          
    
@@ -94,3 +96,9 @@ For this tutorial we will be using domain ```test.wiretrustee.com``` which point
     docker-compose logs management
     docker-compose logs coturn
     docker-compose logs dashboard
+
+10. Once the server is running, you can access the dashboard by https://$WIRETRUSTEE_DOMAIN
+11. Adding a peer will require you to enter the management URL by following the steps in the page https://$WIRETRUSTEE_DOMAIN/add-peer and in the 3rd step:
+```shell
+sudo wiretrustee up --setup-key <PASTE-SETUP-KEY> --management-url https://$WIRETRUSTEE_DOMAIN:33073
+```

infrastructure_files/configure.sh

@@ -1,7 +1,27 @@
 #!/bin/bash
 
-unset $(grep -v '^#' ./setup.env | sed -E 's/(.*)=.*/\1/' | xargs)
+source setup.env
-export $(grep -v '^#' ./setup.env | xargs)
+
+if [[ "x-$WIRETRUSTEE_DOMAIN" == "x-" ]]
+then
+  echo WIRETRUSTEE_DOMAIN is not set, please update your setup.env file
+  exit 1
+fi
+
+# local development or tests
+if [[ $WIRETRUSTEE_DOMAIN == "localhost" || $WIRETRUSTEE_DOMAIN == "127.0.0.1" ]]
+then
+  export WIRETRUSTEE_MGMT_API_ENDPOINT=http://$WIRETRUSTEE_DOMAIN:$WIRETRUSTEE_MGMT_API_PORT
+  unset WIRETRUSTEE_MGMT_API_CERT_FILE
+  unset WIRETRUSTEE_MGMT_API_CERT_KEY_FILE
+fi
+
+# if not provided, we generate a turn password
+if [[ "x-$TURN_PASSWORD" == "x-" ]]
+then
+  export TURN_PASSWORD=$(openssl rand -base64 32|sed 's/=//g')
+fi
 
 envsubst < docker-compose.yml.tmpl > docker-compose.yml
 envsubst < management.json.tmpl > management.json
+envsubst < turnserver.conf.tmpl > turnserver.conf

infrastructure_files/docker-compose.yml.tmpl

@@ -11,19 +11,18 @@ services:
       - AUTH0_DOMAIN=$WIRETRUSTEE_AUTH0_DOMAIN
       - AUTH0_CLIENT_ID=$WIRETRUSTEE_AUTH0_CLIENT_ID
       - AUTH0_AUDIENCE=$WIRETRUSTEE_AUTH0_AUDIENCE
-      - WIRETRUSTEE_MGMT_API_ENDPOINT=https://$WIRETRUSTEE_DOMAIN:33071
+      - WIRETRUSTEE_MGMT_API_ENDPOINT=$WIRETRUSTEE_MGMT_API_ENDPOINT
       - NGINX_SSL_PORT=443
       - LETSENCRYPT_DOMAIN=$WIRETRUSTEE_DOMAIN
       - LETSENCRYPT_EMAIL=$WIRETRUSTEE_LETSENCRYPT_EMAIL
     volumes:
-      - /var/lib/wiretrustee/dashboard/letsencrypt:/etc/letsencrypt/
+      - wiretrustee-letsencrypt:/etc/letsencrypt/
   # Signal
   signal:
     image: wiretrustee/signal:latest
     restart: unless-stopped
     volumes:
       - wiretrustee-signal:/var/lib/wiretrustee
-    #      - /var/log/wiretrustee/signal.log:/var/log/wiretrustee/signal.log
     ports:
       - 10000:10000
   #     # port and command for Let's Encrypt validation
@@ -37,12 +36,11 @@ services:
       - dashboard
     volumes:
       - wiretrustee-mgmt:/var/lib/wiretrustee
-      - /var/lib/wiretrustee/dashboard/letsencrypt:/etc/letsencrypt:ro
+      - wiretrustee-letsencrypt:/etc/letsencrypt:ro
       - ./management.json:/etc/wiretrustee/management.json
-    #      - /var/log/wiretrustee/management.log:/var/log/wiretrustee/management.log
     ports:
       - 33073:33073 #gRPC port
-      - 33071:33071 #HTTP port
+      - $WIRETRUSTEE_MGMT_API_PORT:33071 #API port
   #     # port and command for Let's Encrypt validation
   #      - 443:443
   #    command: ["--letsencrypt-domain", "$WIRETRUSTEE_DOMAIN", "--log-file", "console"]
@@ -50,7 +48,7 @@ services:
   coturn:
     image: coturn/coturn
     restart: unless-stopped
-    domainname: <YOUR DOMAIN>
+    domainname: $WIRETRUSTEE_DOMAIN
     volumes:
       - ./turnserver.conf:/etc/turnserver.conf:ro
     #      - ./privkey.pem:/etc/coturn/private/privkey.pem:ro
@@ -58,4 +56,5 @@ services:
     network_mode: host
 volumes:
   wiretrustee-mgmt:
-  wiretrustee-signal:
+  wiretrustee-signal:
+  wiretrustee-letsencrypt:

infrastructure_files/management.json.tmpl

@@ -12,8 +12,8 @@
             {
                 "Proto": "udp",
                 "URI": "turn:$WIRETRUSTEE_DOMAIN:3478",
-                "Username": "",
+                "Username": "$TURN_USER",
-                "Password": null
+                "Password": "$TURN_PASSWORD"
             }
         ],
         "CredentialsTTL": "12h",
@@ -28,19 +28,14 @@
     },
     "Datadir": "",
     "HttpConfig": {
-        "Address": "0.0.0.0:33071",
+        "Address": "0.0.0.0:$WIRETRUSTEE_MGMT_API_PORT",
         "AuthIssuer": "https://$WIRETRUSTEE_AUTH0_DOMAIN/",
         "AuthAudience": "$WIRETRUSTEE_AUTH0_AUDIENCE",
-        "AuthKeysLocation": "https://$WIRETRUSTEE_AUTH0_DOMAIN/.well-known/jwks.json"
+        "AuthKeysLocation": "https://$WIRETRUSTEE_AUTH0_DOMAIN/.well-known/jwks.json",
+        "CertFile":"$WIRETRUSTEE_MGMT_API_CERT_FILE",
+        "CertKey":"$WIRETRUSTEE_MGMT_API_CERT_KEY_FILE"
     },
     "IdpManagerConfig": {
-        "Manager": "none",
+        "Manager": "none"
-        "Auth0ClientCredentials": {
-        "Audience": "<PASTE YOUR AUTH0 AUDIENCE HERE>",
-        "AuthIssuer": "<PASTE YOUR AUTH0 Auth Issuer HERE>",
-        "ClientId": "<PASTE YOUR AUTH0 Application Client ID HERE>",
-        "ClientSecret": "<PASTE YOUR AUTH0 Application Client Secret HERE>",
-         "GrantType": "client_credentials"
-         }
      }
 }

infrastructure_files/setup.env

@@ -1,4 +1,6 @@
-# e.g. app.mydomain.com
+# Dashboard domain and auth0 configuration
+
+# Dashboard domain. e.g. app.mydomain.com
 WIRETRUSTEE_DOMAIN=""
 # e.g. dev-24vkclam.us.auth0.com
 WIRETRUSTEE_AUTH0_DOMAIN=""
@@ -8,3 +10,42 @@ WIRETRUSTEE_AUTH0_CLIENT_ID=""
 WIRETRUSTEE_AUTH0_AUDIENCE=""
 # e.g. hello@mydomain.com
 WIRETRUSTEE_LETSENCRYPT_EMAIL=""
+
+## From this point, most settings are being done automatically, but you can edit if you need some customization
+
+# Management API
+
+# Management API port
+WIRETRUSTEE_MGMT_API_PORT=33071
+# Management API endpoint address, used by the Dashboard
+WIRETRUSTEE_MGMT_API_ENDPOINT=https://$WIRETRUSTEE_DOMAIN:$WIRETRUSTEE_MGMT_API_PORT
+# Management Certficate file path. These are generated by the Dashboard container
+WIRETRUSTEE_MGMT_API_CERT_FILE="/etc/letsencrypt/live/$WIRETRUSTEE_DOMAIN/fullchain.pem"
+# Management Certficate key file path.
+WIRETRUSTEE_MGMT_API_CERT_KEY_FILE="/etc/letsencrypt/live/$WIRETRUSTEE_DOMAIN/privkey.pem"
+
+# Turn credentials
+
+# User
+TURN_USER=self
+# Password. If empty, the configure.sh will generate one with openssl
+TURN_PASSWORD=
+# Min port
+TURN_MIN_PORT=49152
+# Max port
+TURN_MAX_PORT=65535
+
+# exports
+export WIRETRUSTEE_DOMAIN
+export WIRETRUSTEE_AUTH0_DOMAIN
+export WIRETRUSTEE_AUTH0_CLIENT_ID
+export WIRETRUSTEE_AUTH0_AUDIENCE
+export WIRETRUSTEE_LETSENCRYPT_EMAIL
+export WIRETRUSTEE_MGMT_API_PORT
+export WIRETRUSTEE_MGMT_API_ENDPOINT
+export WIRETRUSTEE_MGMT_API_CERT_FILE
+export WIRETRUSTEE_MGMT_API_CERT_KEY_FILE
+export TURN_USER
+export TURN_PASSWORD
+export TURN_MIN_PORT
+export TURN_MAX_PORT

infrastructure_files/turnserver.conf.tmpl

@@ -154,12 +154,12 @@ tls-listening-port=5349
 # Lower and upper bounds of the UDP relay endpoints:
 # (default values are 49152 and 65535)
 #
-min-port=49152
+min-port=$TURN_MIN_PORT
-max-port=65535
+max-port=$TURN_MAX_PORT
 
 # Uncomment to run TURN server in 'normal' 'moderate' verbose mode.
 # By default the verbose mode is off.
-verbose
+#verbose
 
 # Uncomment to run TURN server in 'extra' verbose mode.
 # This mode is very annoying and produces lots of output.
@@ -249,7 +249,7 @@ lt-cred-mech
 #user=username1:key1
 #user=username2:key2
 # OR:
-user=username1:password1
+user=$TURN_USER:$TURN_PASSWORD
 #user=username2:password2
 #
 # Keys must be generated by turnadmin utility. The key value depends