docs: keep up with changed reality.

ugly hack with relativ URLs because relref is apparently broken when
linking to section pages (_index.md) except for a few cases...

docs/content/_index.md

@@ -4,39 +4,46 @@ title = "zrepl - ZFS replication"
 
 # zrepl - ZFS replication
 
-`zrepl` is a tool for replicating ZFS filesystems.
+zrepl is a tool for replicating ZFS filesystems.
 
-{{% panel theme="danger" header="Important" %}}
+{{% notice info %}}
-`zrepl` as well as this documentation is still under active
+`zrepl` as well as this documentation is still under active development.
-development. Some of the features below are not implemented yet. Use & test at your own risk ;)
+Use & test at your own risk ;)
-{{% / panel  %}}
+{{% /notice %}}
+
+## Getting started
+
+The [5 minute tutorial setup](/tutorial/) gives you a first impression.
 
 ## Main Features
 
-* filesystem replication
+* Filesystem Replication
-    * local & over network (SSH)
+    * [x] Local & Remote
-    * push & pull mode
+    * [x] Pull mode
-    * snapshots & bookmarks support
+    * [ ] Push mode
-    * feature-negotiation for
+    * [x] Access control checks when pulling datasets
-        * resumable `send & receive`
+    * [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules
-        * compressed `send & receive`
+    * [ ] Bookmarks support
-        * raw encrypted `send & receive` (as soon as it is available)
+    * [ ] Feature-negotiation for
-    * access control checks when pulling datasets
+        * Resumable `send & receive`
-    * [flexible mappings]({{< ref "configuration/overview.md#mapping-filter-syntax" >}}) for filesystems
+        * Compressed `send & receive`
-* automatic snapshot creation
+        * Raw encrypted `send & receive` (as soon as it is available)
-    * periodic interval
+* Automatic snapshot creation
-* automatic snapshot pruning
+    * [x] Ensure fixed time interval between snapshots
-    * [Retention Grid]({{< ref "configuration/snapshots.md#retention-grid" >}})
+* Automatic snapshot pruning
+    * [x] <i class="fa fa-arrow-right" aria-hidden="true"></i> [Retention Grid]({{< ref "configuration/prune.md#retention-grid" >}})
+* Maintainable implementation in Go
+    * [x] Cross platform
+    * [x] Type safe & testable code
 
 ## Contributing
 
-`zrepl` is usable but nowhere near a stable release right now -  we are happy
+zrepl is usable but nowhere near a stable release right now -  we are happy about contributors!
-about contributors!
 
 * Explore the codebase
     * These docs live in the `docs/` subdirectory
-* Document non-obvious / confusing / plain broken things you encounter when using `zrepl` for the first time
+* Document any non-obvious / confusing / plain broken behavior you encounter when setting up zrepl for the first time
-* Check the *Issues* and *Projects* sections for things to do ;)
+* Check the *Issues* and *Projects* sections for things to do
 
 {{% panel header="<i class='fa fa-github'></i> Getting your code merged"%}}
 [The <i class='fa fa-github'></i> GitHub repository](https://github.com/zrepl/zrepl) is where all development happens.

docs/content/configuration/_index.md

@@ -3,4 +3,6 @@ title = "Configuration"
 alwaysopen = true
 +++
 
+{{% alert theme="warning" %}}Under Construction{{% /alert %}}
+
 {{% children description="true" %}}

docs/content/configuration/example_pull_backup.md

@@ -1,9 +0,0 @@
-+++
-title = "Example: Pull Backup"
-description = "Example configuration for a typical pull-backup scenario, e.g. server to server"
-+++
-
-Example configuration for a typical pull-backup scenario, e.g. server to server
-
-{{% alert theme="warning"%}}TBD{{% /alert %}}
-

docs/content/configuration/example_push_backup.md

@@ -1,8 +0,0 @@
-+++
-title = "Example: Push Backup"
-description = "Example configuration for a typical push-backup scenario, e.g. laptop to NAS"
-+++
-
-Example configuration for a typical push-backup scenario, e.g. laptop to NAS
-
-{{% alert theme="warning"%}}TBD{{% /alert %}}

docs/content/configuration/jobs.md

@@ -0,0 +1,12 @@
++++
+title = "Job Types"
+weight = 10
++++
+
+{{% alert theme="warning" %}}Under Construction{{% /alert %}}
+
+## Source
+
+## Pull
+
+## Local

docs/content/configuration/map_filter_syntax.md

@@ -1,60 +1,10 @@
 +++
-title = "Overview"
+title = "Mapping & Filter Syntax"
-weight = 100
+weight = 20
-description = "Configuration format, SSH authentication, etc."
+description = "How to specify mappings & filters"
 +++
 
-{{% panel header="Recommendation" %}}
+{{% alert theme="warning" %}}Under Construction{{% /alert %}}
-Keep the [sample configuration file](https://github.com/zrepl/zrepl/blob/master/cmd/sampleconf/zrepl.yml) open on the side while reading this document!
-{{% / panel %}}
-
-All configuration is managed in a single YAML file.<br />
-It is structured by sections roughly corresponding to `zrepl` subcommands:
-
-```yaml
-# REPLICATION
-# Remote zrepl instances where pull and push jobs connect to
-remotes: 
-  name_of_remote: #...
-# Push jobs (replication from local to remote)
-pushs: 
-  name_of_push_job: #...
-  name_of_other_push_job: #...
-# pull jobs (replication from remote to local & local to local)
-pulls: 
-  name_of_pull_job: #...
-# mapping incoming pushs to local datasets
-sinks: 
-  client_identity: #...
-# access control for remote pull jobs
-pull_acls: 
-  client_identity: #...
-
-# SNAPSHOT MANAGEMENT
-# Automatic snapshotting of filesystems
-autosnap:
-  name_of_autosnap_job: #... 
-# Automatic pruning of snapshots based on creation date
-prune:
-  name_of_prune_job: #... 
-```
-
-When using `zrepl(8)`, a *subcommand* is passed the *job name* as a positional argument:
-
-```yaml
-autosnap: # subcommand
-  db: # job name
-    prefix: zrepl_ 
-    interval: 10m
-    dataset_filter: {
-      "tank/db<": ok
-    }
-```
-```bash
-$ zrepl autosnap --config zrepl.yml db
-```
-
-Run `zrepl --help` for a list of subcommands and options.
 
 ## Mapping & Filter Syntax
 
@@ -140,10 +90,3 @@ pull_acls:
       "tank/usr/home<": ok,
     }
 ```
-
-## Next up
-
-* [Automating snapshot creation & pruning]({{< ref "configuration/snapshots.md" >}})
-
-* [Replicating filesystems]({{< ref "configuration/replication.md" >}})
-

docs/content/configuration/prune.md

@@ -0,0 +1,9 @@
++++
+title = "Pruning"
+description = "Automated pruning of snapshots"
+weight = 200
++++
+
+{{% alert theme="warning" %}}Under Construction{{% /alert %}}
+
+## Retention Grid

docs/content/configuration/snapshots.md

@@ -1,9 +0,0 @@
-+++
-title = "Snapshot Management"
-description = "Automated snapshot creation & pruning"
-weight = 200
-+++
-
-{{% alert theme="warning"%}}TBD{{% /alert %}}
-
-## Retention Grid

docs/content/configuration/transports.md

@@ -1,27 +1,11 @@
 +++
-title = "Filesystem Replication"
+title = "Transports"
-description = "Replicating filesystems with existing bookmarks & snapshots"
-weight = 300
 +++
 
-{{% alert theme="warning"%}}Under Construction{{% /alert %}}
+{{% alert theme="warning" %}}Under Construction{{% /alert %}}
 
-### Remotes
+## Stdinserver
 
-The `remotes` section specifies remote `zrepl` instances from which to pull from / push backups to:
-
-```yaml
-remotes:
-  offsite_backups:
-    transport:
-      ssh:
-        host: 192.168.122.6
-        user: root
-        port: 22
-        identity_file: /etc/zrepl/identities/offsite_backups
-```
-
-#### SSH Transport
 
 The SSH transport connects to the remote server using the SSH binary in
 `$PATH` and the parameters specified in the `zrepl` config file.
@@ -37,4 +21,3 @@ Check the examples for instructions on how to set this up on your machines!
 The environment variables of the underlying SSH process are cleared. `$SSH_AUTH_SOCK` will not be available. We suggest creating a separate, unencrypted SSH key.
 {{% / panel %}}
 
-

docs/content/impl/_index.md

@@ -2,6 +2,8 @@
 title = "Implementation Overview"
 +++
 
+{{% alert theme="warning" %}}Under Construction{{% /alert %}}
+
 The following design aspects may convince you that `zrepl` is superior to a hacked-together shell script solution.
 
 ## Language

docs/content/install/_index.md

@@ -0,0 +1,94 @@
++++
+title = "Installation"
+weight = 20
++++
+
+{{% notice note %}}
+Note: check out the [tutorial]({{< relref "tutorial/_index.md" >}}) if you want a first impression of zrepl.
+{{% /notice %}}
+
+## User Privileges
+
+It is possible to run zrepl as an unprivileged user in combination with
+[ZFS delegation](https://www.freebsd.org/doc/handbook/zfs-zfs-allow.html).
+
+Also, there is the possibility to run it in a jail on FreeBSD by delegating a dataset to the jail.
+
+However, until we get around documenting those setups, you will have to run zrepl as root or experiment yourself :)
+
+## Installation
+
+zrepl is currently not packaged on any operating system. Signed & versioned releases are planned but not available yet.
+
+Check out the sources yourself, compile and install to the zrepl user's `$PATH`.<br />
+**Note**: if the zrepl binary is not in `$PATH`, you will have to adjust the examples in the [tutorial]({{< relref "tutorial/_index.md" >}}).
+
+```bash
+# NOTE: you may want to checkout & build as an unprivileged user
+cd /root
+git clone https://github.com/zrepl/zrepl.git
+cd zrepl
+go build -o zrepl
+cp zrepl /usr/local/bin/zrepl
+rehash
+# see if it worked
+zrepl help
+```
+
+
+
+## Configuration & Runtime Directories
+
+### Main Configuration File
+
+The main configuration file lives in either
+
+* `/etc/zrepl/zrepl.yml`
+* `/usr/local/etc/zrepl/zrepl.yml`
+
+The file locations are check in that order. Alternative locations must be specified using a command line flag.
+
+### Runtime Directories
+
+**Source jobs** with `serve.type = stdinserver` require a private runtime directory. ([Why?]({{< relref "configuration/transports.md#stdinserver" >}}))
+
+The default is `/var/run/zrepl/stdinserver` and it must only be accessible by the user
+that runs `zrepl daemon` and `zrepl stdinserver`:
+
+```bash
+mkdir -p /var/run/zrepl/stdinserver
+chmod -R 0700 /var/run/zrepl
+```
+
+## Running the Daemon
+
+All work zrepl done is performed by a daemon process.
+
+There are no *rc(8)* or *systemd.service(5)* service definitions yet.
+
+The daemon does not fork and writes all log output to stderr.
+
+```bash
+zrepl --config /etc/zrepl/zrepl.yml daemon
+```
+
+FreeBSD ships with the *daemon(8)* utility which is also a good start for writing an *rc(8)* file:
+
+```bash
+daemon -o /var/log/zrepl.log \
+       -p /var/run/zrepl/daemon.pid \
+       zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon
+```
+
+{{% notice info %}}
+Make sure to read the first lines of log output after the daemon starts: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets
+in the runtime directory, it will complain but not terminate as other tasks such as taking periodic snapshots might still work.
+{{% / notice %}}
+
+### Restarting
+
+The daemon handles SIGINT and SIGTERM for graceful shutdown.
+
+Graceful shutdown means at worst that a job will not be rescheduled for the next interval.
+
+The daemon exits as soon as all jobs have reported shut down.

docs/content/tutorial/_index.md

@@ -0,0 +1,173 @@
+---
+title: "Tutorial"
+weight: 1
+---
+
+This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
+
+We assume the following scenario
+
+* Production server `prod1` with filesystems to back up
+    * `zroot/var/db`
+    * `zroot/usr/home` and all its child filesystems
+    * **except** `zroot/usr/home/paranoid` belonging to a user doing backups themselves
+* Backup server `backups` with
+    * Filesystem `storage/zrepl/pull/prod1` + children dedicated to backups of `prod1`
+
+Our backup solution should fulfill the following requirements:
+
+* Periodically snapshot the filesystems on `prod1` *every 10 minutes*
+* Incrementally replicate these snapshots to `storage/zrepl/pull/prod1/*` on `backups`
+* Keep only very few snapshots on `prod1` to save disk space
+* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on `backups`
+
+## Analysis
+
+We can model this situation as two jobs:
+
+* A **source job** on `prod1`
+    * Creates the snapshots
+    * Keeps a few snapshots that are also on `prod1` to enable incremental replication
+* A **pull job** on `prod1`
+    * Pulls the snapshots
+    * Fades out snapshots as they age
+
+{{%expand "Side note: why doesn't `backups` take the snapshots right before replication?" %}}
+After all, a little `ssh prod1 'zfs snapshot...'` wouldn't be so bad, right?
+
+As is the case with all distributed systems, the link between `prod1` and `backups` might be down for an hour or two.
+We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
+
+When the link comes up again, `backups` will happily catch up the 12 snapshots taken by `prod1` in the meantime, without
+a gap in our backup history.
+{{%/expand%}}
+
+## Install zrepl
+
+Follow the [OS-specific installation instructions](/install/) and come back here.
+
+## Configure `backups`
+
+We define a **pull job** named `pull_prod1` in the [main configuration file](/install/#main-configuration-file):
+
+```yaml
+jobs:
+- name: pull_prod1
+  type: pull
+  connect:
+    type: ssh+stdinserver
+    host: prod1.example.com
+    user: root
+    port: 22
+    identity_file: /etc/zrepl/ssh/prod1
+  interval: 10m
+  mapping: {
+    "<":"storage/zrepl/pull/prod1"
+  }
+  initial_repl_policy: most_recent
+  snapshot_prefix: zrepl_pull_backup_
+  prune:
+    policy: grid
+    grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
+```
+
+The `connect` section instructs zrepl to use the `stdinserver` transport: instead of directly exposing zrepl on `prod1`
+to the internet, `backups` starts the `zrepl stdinserver` on `prod1` via SSH.
+(You can learn more about what happens [here]({{< relref "configuration/transports.md#stdinserver" >}}), or just continue following this tutorial.)
+
+Thus, we need to create the SSH key pair `/etc/zrepl/ssh/prod1{,.pub}` and later pass the public part to `prod1`
+which will use it to authenticate `backups`. Execute the following commands on `backups` as the root user:
+
+```bash
+cd /etc/zrepl
+mkdir -p ssh
+chmod 0700 ssh
+ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/prod1
+```
+You can learn more about the [**pull job** format here]({{< relref "configuration/jobs.md#pull" >}}) but for now we are good to go.
+
+## Configure `prod1`
+
+We define a corresponding **source job** named `pull_backup` in the [main configuration file](/install/#main-configuration-file)
+`zrepl.yml`:
+
+```yaml
+jobs:
+
+- name: pull_backup
+  type: source
+  serve:
+    type: stdinserver
+    client_identity: backups.example.com
+  datasets: {
+    "zroot/var/db": "ok",
+    "zroot/usr/home<": "ok",
+    "zroot/usr/home/paranoid": "!",
+  }
+  snapshot_prefix: zrepl_pull_backup_
+  interval: 10m
+  prune:
+    policy: grid
+    grid: 1x1d(keep=all)
+
+```
+
+The `serve` section corresponds to the `connect` section in the configuration of `backups`.
+
+We need to allow the SSH key on `backups` to execute `zrepl stdinserver backups.example.com` on
+`prod1`. For good measure, we will in fact enforce that only this command can be executed.
+
+Open `/root/.ssh/authorized_keys` and add either of the the following lines, replacing BACKUPS_SSH_PUBKEY at the end
+of the line with the contents of `/etc/zrepl/ssh/prod1.pub` (note the **.pub** !) from `backups`.
+
+```
+# for OpenSSH >= 7.2
+command="zrepl stdinserver backups.example.com",restrict BACKUPS_SSH_PUBKEY
+# for older OpenSSH versions
+command="zrepl stdinserver backups.example.com",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc  BACKUPS_SSH_PUBKEY
+```
+
+{{% alert theme="info" %}}The entries **must** be on a single line, including the replaced BACKUPS_SSH_PUBKEY{{% /alert %}}
+
+Again, you can learn more about the [**source job** format here]({{< ref "configuration/jobs.md#source" >}}).
+
+## Apply Configuration Changes
+
+We need to restart the zrepl daemon on **both** `prod1` and `backups`.
+
+This is [OS-specific](/install/#restarting).
+
+## Watch it Work
+
+A common setup is to watch the log output and zfs list of snapshots on both machines.
+
+If you like tmux, here is a handy script that works on FreeBSD:
+
+```bash
+pkg install gnu-watch tmux
+tmux new-window
+tmux split-window "tail -f /var/log/zrepl.log"
+tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
+tmux select-layout tiled
+```
+
+The Linux equivalent might look like this
+
+```bash
+# make sure tmux is installed & let's assume you use systemd + journald
+tmux new-window
+tmux split-window "journalctl -f -u zrepl.service"
+tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
+tmux select-layout tiled
+```
+
+## Summary
+
+Congratulations, you have a working pull backup. Where to go next?
+
+* Read more about [configuration format, options & job types](/configuration/)
+* Learn about [implementation details](/impl/) of zrepl.
+
+
+
+

docs/themes/docdock

@@ -1 +1 @@
-Subproject commit f1a902a088f925e1478062d5327aa8e897752b13
+Subproject commit 9e429b2c209e6062e7bd6f53ced0a31d3b624448