smegmesh

Disclaimer

Submitted to fill the requirements of Msci (Hons) Computer Science at the School of Computer Science, University of St Andrews.

License

This repository is licensed under the MIT License. See the LICENSE file for more details.

Overview

Distributed WireGuard mesh management. This tool helps to configure WireGuard networks in a mesh topology such that there is no single point of failure. The tool aims to set-up mesh networks with minimal knowledge and configuration of WireGuard.

The idea being that a node can take up one of two roles in the network, a peer or a client. A peer is publicly accessible and must have IPv6 forwarding enabled. Peer's responsibility is routing traffic on behalf of clients associated with it.

Whereas, a client hides behind a private endpoint in which all packets are routed through the peer. A client must enable the flat keepAliveWg to ensure that its associated peer learns about any NAT mappings that change.

IPv6 is used in the overlay to make use of the larger address space. A node hashes it's WireGuard public key to create an identifier (the last 64-bits of the IPv6 address) and the mesh-id is hashed into the first 64-bits of the IPv6 address to create the locator.

A node (both client and a peer) can be in multiple meshes at the same time. In which case the node can optionally choose to act as a bridge and forward packets between the two meshes. Through this it is possible to define complex topologies. To route between meshes multiple hops away a simple link-state protocol is adopted (similar to RIP) in which the path length (number of meshes) is used to determine the shortest path.

Redundant routing is possible to create multiple exit points to the same mesh network. In which case consistent hashing is performed to split traffic between the exit points.

Message Dissemination

A variant of the gossip protocol is used for message dissemination. Each peer in the network is ordered lexicographically ordered by their public key. The node with the lexicographically lowest public key is used as the leader of the mesh. Every heartBeatInterval disseminates a refresh message throughout the entirety of the group in order to prune nodes that may have prematurely died.

If after 3 * heartBeatInterval a node has not received a dissemination message then the node prunes the leader and expects one from the next lexicographically lowest public key.

To 'merge' updates and reconcile any conflicts a Conflict Free Replicated Data Type (CRDT) is implemented. Consisting of an add and remove set. Where a node is in the group if it is in the add set and there is either no entry in the remove set or the timestamp in the remove set has a lower vector clock value.

Performance

This prototype has been tested to a scale of 3000 peers in the network. Furthermore, the fault-tolerance has been tested to a scale 3000 nodes to the order of 20 seconds for the entire network and 12 seconds for the 99 percentile.

Installation

To build the project do: go build -v ./.... A Docker file is provided to get started.

To build with the Dockerfile: docker build -t smegmesh-base ./

Then run an example topology in the examples folder. For example: cd examples/simple && docker-compose up -d

Tools

Smegd

Smegmesh requires the daemon process to be running (smegd) which also takes a configuration.yaml file. An example yaml configuration file is provided in examples/simple/shared/configuration.

Smegctl

Smegctl is a CLI tool to create, join, visualise and administer networks.

Api

An api is provided to invoke functions to create, join, visualise and administer networks. This could be used to create an application that allows a user to configure the networks.

Dns

A dns server is provided to resolve an alias into an IPv6 address.