mirror of
https://github.com/zrepl/zrepl.git
synced 2024-12-31 19:39:28 +01:00
docs: keep up with changed reality.
ugly hack with relativ URLs because relref is apparently broken when linking to section pages (_index.md) except for a few cases...
This commit is contained in:
Christian Schwarz
parent
6a05e101cf
commit
3fd9726719
@ -4,39 +4,46 @@ title = "zrepl - ZFS replication"
|
||||
|
||||
# zrepl - ZFS replication
|
||||
|
||||
`zrepl` is a tool for replicating ZFS filesystems.
|
||||
zrepl is a tool for replicating ZFS filesystems.
|
||||
|
||||
{{% panel theme="danger" header="Important" %}}
|
||||
`zrepl` as well as this documentation is still under active
|
||||
development. Some of the features below are not implemented yet. Use & test at your own risk ;)
|
||||
{{% / panel %}}
|
||||
{{% notice info %}}
|
||||
`zrepl` as well as this documentation is still under active development.
|
||||
Use & test at your own risk ;)
|
||||
{{% /notice %}}
|
||||
|
||||
## Getting started
|
||||
|
||||
The [5 minute tutorial setup](/tutorial/) gives you a first impression.
|
||||
|
||||
## Main Features
|
||||
|
||||
* filesystem replication
|
||||
* local & over network (SSH)
|
||||
* push & pull mode
|
||||
* snapshots & bookmarks support
|
||||
* feature-negotiation for
|
||||
* resumable `send & receive`
|
||||
* compressed `send & receive`
|
||||
* raw encrypted `send & receive` (as soon as it is available)
|
||||
* access control checks when pulling datasets
|
||||
* [flexible mappings]({{< ref "configuration/overview.md#mapping-filter-syntax" >}}) for filesystems
|
||||
* automatic snapshot creation
|
||||
* periodic interval
|
||||
* automatic snapshot pruning
|
||||
* [Retention Grid]({{< ref "configuration/snapshots.md#retention-grid" >}})
|
||||
|
||||
* Filesystem Replication
|
||||
* [x] Local & Remote
|
||||
* [x] Pull mode
|
||||
* [ ] Push mode
|
||||
* [x] Access control checks when pulling datasets
|
||||
* [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules
|
||||
* [ ] Bookmarks support
|
||||
* [ ] Feature-negotiation for
|
||||
* Resumable `send & receive`
|
||||
* Compressed `send & receive`
|
||||
* Raw encrypted `send & receive` (as soon as it is available)
|
||||
* Automatic snapshot creation
|
||||
* [x] Ensure fixed time interval between snapshots
|
||||
* Automatic snapshot pruning
|
||||
* [x] <i class="fa fa-arrow-right" aria-hidden="true"></i> [Retention Grid]({{< ref "configuration/prune.md#retention-grid" >}})
|
||||
* Maintainable implementation in Go
|
||||
* [x] Cross platform
|
||||
* [x] Type safe & testable code
|
||||
|
||||
## Contributing
|
||||
|
||||
`zrepl` is usable but nowhere near a stable release right now - we are happy
|
||||
about contributors!
|
||||
zrepl is usable but nowhere near a stable release right now - we are happy about contributors!
|
||||
|
||||
* Explore the codebase
|
||||
* These docs live in the `docs/` subdirectory
|
||||
* Document non-obvious / confusing / plain broken things you encounter when using `zrepl` for the first time
|
||||
* Check the *Issues* and *Projects* sections for things to do ;)
|
||||
* Document any non-obvious / confusing / plain broken behavior you encounter when setting up zrepl for the first time
|
||||
* Check the *Issues* and *Projects* sections for things to do
|
||||
|
||||
{{% panel header="<i class='fa fa-github'></i> Getting your code merged"%}}
|
||||
[The <i class='fa fa-github'></i> GitHub repository](https://github.com/zrepl/zrepl) is where all development happens.
|
||||
|
||||
@ -3,4 +3,6 @@ title = "Configuration"
|
||||
alwaysopen = true
|
||||
+++
|
||||
|
||||
{{% alert theme="warning" %}}Under Construction{{% /alert %}}
|
||||
|
||||
{{% children description="true" %}}
|
||||
|
||||
@ -1,9 +0,0 @@
|
||||
+++
|
||||
title = "Example: Pull Backup"
|
||||
description = "Example configuration for a typical pull-backup scenario, e.g. server to server"
|
||||
+++
|
||||
|
||||
Example configuration for a typical pull-backup scenario, e.g. server to server
|
||||
|
||||
{{% alert theme="warning"%}}TBD{{% /alert %}}
|
||||
|
||||
@ -1,8 +0,0 @@
|
||||
+++
|
||||
title = "Example: Push Backup"
|
||||
description = "Example configuration for a typical push-backup scenario, e.g. laptop to NAS"
|
||||
+++
|
||||
|
||||
Example configuration for a typical push-backup scenario, e.g. laptop to NAS
|
||||
|
||||
{{% alert theme="warning"%}}TBD{{% /alert %}}
|
||||
12
docs/content/configuration/jobs.md
Normal file
12
docs/content/configuration/jobs.md
Normal file
@ -0,0 +1,12 @@
|
||||
+++
|
||||
title = "Job Types"
|
||||
weight = 10
|
||||
+++
|
||||
|
||||
{{% alert theme="warning" %}}Under Construction{{% /alert %}}
|
||||
|
||||
## Source
|
||||
|
||||
## Pull
|
||||
|
||||
## Local
|
||||
@ -1,60 +1,10 @@
|
||||
+++
|
||||
title = "Overview"
|
||||
weight = 100
|
||||
description = "Configuration format, SSH authentication, etc."
|
||||
title = "Mapping & Filter Syntax"
|
||||
weight = 20
|
||||
description = "How to specify mappings & filters"
|
||||
+++
|
||||
|
||||
{{% panel header="Recommendation" %}}
|
||||
Keep the [sample configuration file](https://github.com/zrepl/zrepl/blob/master/cmd/sampleconf/zrepl.yml) open on the side while reading this document!
|
||||
{{% / panel %}}
|
||||
|
||||
All configuration is managed in a single YAML file.<br />
|
||||
It is structured by sections roughly corresponding to `zrepl` subcommands:
|
||||
|
||||
```yaml
|
||||
# REPLICATION
|
||||
# Remote zrepl instances where pull and push jobs connect to
|
||||
remotes:
|
||||
name_of_remote: #...
|
||||
# Push jobs (replication from local to remote)
|
||||
pushs:
|
||||
name_of_push_job: #...
|
||||
name_of_other_push_job: #...
|
||||
# pull jobs (replication from remote to local & local to local)
|
||||
pulls:
|
||||
name_of_pull_job: #...
|
||||
# mapping incoming pushs to local datasets
|
||||
sinks:
|
||||
client_identity: #...
|
||||
# access control for remote pull jobs
|
||||
pull_acls:
|
||||
client_identity: #...
|
||||
|
||||
# SNAPSHOT MANAGEMENT
|
||||
# Automatic snapshotting of filesystems
|
||||
autosnap:
|
||||
name_of_autosnap_job: #...
|
||||
# Automatic pruning of snapshots based on creation date
|
||||
prune:
|
||||
name_of_prune_job: #...
|
||||
```
|
||||
|
||||
When using `zrepl(8)`, a *subcommand* is passed the *job name* as a positional argument:
|
||||
|
||||
```yaml
|
||||
autosnap: # subcommand
|
||||
db: # job name
|
||||
prefix: zrepl_
|
||||
interval: 10m
|
||||
dataset_filter: {
|
||||
"tank/db<": ok
|
||||
}
|
||||
```
|
||||
```bash
|
||||
$ zrepl autosnap --config zrepl.yml db
|
||||
```
|
||||
|
||||
Run `zrepl --help` for a list of subcommands and options.
|
||||
{{% alert theme="warning" %}}Under Construction{{% /alert %}}
|
||||
|
||||
## Mapping & Filter Syntax
|
||||
|
||||
@ -140,10 +90,3 @@ pull_acls:
|
||||
"tank/usr/home<": ok,
|
||||
}
|
||||
```
|
||||
|
||||
## Next up
|
||||
|
||||
* [Automating snapshot creation & pruning]({{< ref "configuration/snapshots.md" >}})
|
||||
|
||||
* [Replicating filesystems]({{< ref "configuration/replication.md" >}})
|
||||
|
||||
9
docs/content/configuration/prune.md
Normal file
9
docs/content/configuration/prune.md
Normal file
@ -0,0 +1,9 @@
|
||||
+++
|
||||
title = "Pruning"
|
||||
description = "Automated pruning of snapshots"
|
||||
weight = 200
|
||||
+++
|
||||
|
||||
{{% alert theme="warning" %}}Under Construction{{% /alert %}}
|
||||
|
||||
## Retention Grid
|
||||
@ -1,9 +0,0 @@
|
||||
+++
|
||||
title = "Snapshot Management"
|
||||
description = "Automated snapshot creation & pruning"
|
||||
weight = 200
|
||||
+++
|
||||
|
||||
{{% alert theme="warning"%}}TBD{{% /alert %}}
|
||||
|
||||
## Retention Grid
|
||||
@ -1,27 +1,11 @@
|
||||
+++
|
||||
title = "Filesystem Replication"
|
||||
description = "Replicating filesystems with existing bookmarks & snapshots"
|
||||
weight = 300
|
||||
title = "Transports"
|
||||
+++
|
||||
|
||||
{{% alert theme="warning"%}}Under Construction{{% /alert %}}
|
||||
{{% alert theme="warning" %}}Under Construction{{% /alert %}}
|
||||
|
||||
### Remotes
|
||||
## Stdinserver
|
||||
|
||||
The `remotes` section specifies remote `zrepl` instances from which to pull from / push backups to:
|
||||
|
||||
```yaml
|
||||
remotes:
|
||||
offsite_backups:
|
||||
transport:
|
||||
ssh:
|
||||
host: 192.168.122.6
|
||||
user: root
|
||||
port: 22
|
||||
identity_file: /etc/zrepl/identities/offsite_backups
|
||||
```
|
||||
|
||||
#### SSH Transport
|
||||
|
||||
The SSH transport connects to the remote server using the SSH binary in
|
||||
`$PATH` and the parameters specified in the `zrepl` config file.
|
||||
@ -31,10 +15,9 @@ another instance of `zrepl` on the other side of the connection; You may be
|
||||
familiar with this concept from [git shell](https://git-scm.com/docs/git-shell)
|
||||
or [Borg Backup](https://borgbackup.readthedocs.io/en/stable/deployment.html).
|
||||
|
||||
Check the examples for instructions on how to set this up on your machines!
|
||||
Check the examples for instructions on how to set this up on your machines!
|
||||
|
||||
{{% panel %}}
|
||||
The environment variables of the underlying SSH process are cleared. `$SSH_AUTH_SOCK` will not be available. We suggest creating a separate, unencrypted SSH key.
|
||||
{{% / panel %}}
|
||||
|
||||
|
||||
@ -2,6 +2,8 @@
|
||||
title = "Implementation Overview"
|
||||
+++
|
||||
|
||||
{{% alert theme="warning" %}}Under Construction{{% /alert %}}
|
||||
|
||||
The following design aspects may convince you that `zrepl` is superior to a hacked-together shell script solution.
|
||||
|
||||
## Language
|
||||
|
||||
94
docs/content/install/_index.md
Normal file
94
docs/content/install/_index.md
Normal file
@ -0,0 +1,94 @@
|
||||
+++
|
||||
title = "Installation"
|
||||
weight = 20
|
||||
+++
|
||||
|
||||
{{% notice note %}}
|
||||
Note: check out the [tutorial]({{< relref "tutorial/_index.md" >}}) if you want a first impression of zrepl.
|
||||
{{% /notice %}}
|
||||
|
||||
## User Privileges
|
||||
|
||||
It is possible to run zrepl as an unprivileged user in combination with
|
||||
[ZFS delegation](https://www.freebsd.org/doc/handbook/zfs-zfs-allow.html).
|
||||
|
||||
Also, there is the possibility to run it in a jail on FreeBSD by delegating a dataset to the jail.
|
||||
|
||||
However, until we get around documenting those setups, you will have to run zrepl as root or experiment yourself :)
|
||||
|
||||
## Installation
|
||||
|
||||
zrepl is currently not packaged on any operating system. Signed & versioned releases are planned but not available yet.
|
||||
|
||||
Check out the sources yourself, compile and install to the zrepl user's `$PATH`.<br />
|
||||
**Note**: if the zrepl binary is not in `$PATH`, you will have to adjust the examples in the [tutorial]({{< relref "tutorial/_index.md" >}}).
|
||||
|
||||
```bash
|
||||
# NOTE: you may want to checkout & build as an unprivileged user
|
||||
cd /root
|
||||
git clone https://github.com/zrepl/zrepl.git
|
||||
cd zrepl
|
||||
go build -o zrepl
|
||||
cp zrepl /usr/local/bin/zrepl
|
||||
rehash
|
||||
# see if it worked
|
||||
zrepl help
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Configuration & Runtime Directories
|
||||
|
||||
### Main Configuration File
|
||||
|
||||
The main configuration file lives in either
|
||||
|
||||
* `/etc/zrepl/zrepl.yml`
|
||||
* `/usr/local/etc/zrepl/zrepl.yml`
|
||||
|
||||
The file locations are check in that order. Alternative locations must be specified using a command line flag.
|
||||
|
||||
### Runtime Directories
|
||||
|
||||
**Source jobs** with `serve.type = stdinserver` require a private runtime directory. ([Why?]({{< relref "configuration/transports.md#stdinserver" >}}))
|
||||
|
||||
The default is `/var/run/zrepl/stdinserver` and it must only be accessible by the user
|
||||
that runs `zrepl daemon` and `zrepl stdinserver`:
|
||||
|
||||
```bash
|
||||
mkdir -p /var/run/zrepl/stdinserver
|
||||
chmod -R 0700 /var/run/zrepl
|
||||
```
|
||||
|
||||
## Running the Daemon
|
||||
|
||||
All work zrepl done is performed by a daemon process.
|
||||
|
||||
There are no *rc(8)* or *systemd.service(5)* service definitions yet.
|
||||
|
||||
The daemon does not fork and writes all log output to stderr.
|
||||
|
||||
```bash
|
||||
zrepl --config /etc/zrepl/zrepl.yml daemon
|
||||
```
|
||||
|
||||
FreeBSD ships with the *daemon(8)* utility which is also a good start for writing an *rc(8)* file:
|
||||
|
||||
```bash
|
||||
daemon -o /var/log/zrepl.log \
|
||||
-p /var/run/zrepl/daemon.pid \
|
||||
zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon
|
||||
```
|
||||
|
||||
{{% notice info %}}
|
||||
Make sure to read the first lines of log output after the daemon starts: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets
|
||||
in the runtime directory, it will complain but not terminate as other tasks such as taking periodic snapshots might still work.
|
||||
{{% / notice %}}
|
||||
|
||||
### Restarting
|
||||
|
||||
The daemon handles SIGINT and SIGTERM for graceful shutdown.
|
||||
|
||||
Graceful shutdown means at worst that a job will not be rescheduled for the next interval.
|
||||
|
||||
The daemon exits as soon as all jobs have reported shut down.
|
||||
173
docs/content/tutorial/_index.md
Normal file
173
docs/content/tutorial/_index.md
Normal file
@ -0,0 +1,173 @@
|
||||
---
|
||||
title: "Tutorial"
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
|
||||
|
||||
We assume the following scenario
|
||||
|
||||
* Production server `prod1` with filesystems to back up
|
||||
* `zroot/var/db`
|
||||
* `zroot/usr/home` and all its child filesystems
|
||||
* **except** `zroot/usr/home/paranoid` belonging to a user doing backups themselves
|
||||
* Backup server `backups` with
|
||||
* Filesystem `storage/zrepl/pull/prod1` + children dedicated to backups of `prod1`
|
||||
|
||||
Our backup solution should fulfill the following requirements:
|
||||
|
||||
* Periodically snapshot the filesystems on `prod1` *every 10 minutes*
|
||||
* Incrementally replicate these snapshots to `storage/zrepl/pull/prod1/*` on `backups`
|
||||
* Keep only very few snapshots on `prod1` to save disk space
|
||||
* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on `backups`
|
||||
|
||||
## Analysis
|
||||
|
||||
We can model this situation as two jobs:
|
||||
|
||||
* A **source job** on `prod1`
|
||||
* Creates the snapshots
|
||||
* Keeps a few snapshots that are also on `prod1` to enable incremental replication
|
||||
* A **pull job** on `prod1`
|
||||
* Pulls the snapshots
|
||||
* Fades out snapshots as they age
|
||||
|
||||
{{%expand "Side note: why doesn't `backups` take the snapshots right before replication?" %}}
|
||||
After all, a little `ssh prod1 'zfs snapshot...'` wouldn't be so bad, right?
|
||||
|
||||
As is the case with all distributed systems, the link between `prod1` and `backups` might be down for an hour or two.
|
||||
We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
|
||||
|
||||
When the link comes up again, `backups` will happily catch up the 12 snapshots taken by `prod1` in the meantime, without
|
||||
a gap in our backup history.
|
||||
{{%/expand%}}
|
||||
|
||||
## Install zrepl
|
||||
|
||||
Follow the [OS-specific installation instructions](/install/) and come back here.
|
||||
|
||||
## Configure `backups`
|
||||
|
||||
We define a **pull job** named `pull_prod1` in the [main configuration file](/install/#main-configuration-file):
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
- name: pull_prod1
|
||||
type: pull
|
||||
connect:
|
||||
type: ssh+stdinserver
|
||||
host: prod1.example.com
|
||||
user: root
|
||||
port: 22
|
||||
identity_file: /etc/zrepl/ssh/prod1
|
||||
interval: 10m
|
||||
mapping: {
|
||||
"<":"storage/zrepl/pull/prod1"
|
||||
}
|
||||
initial_repl_policy: most_recent
|
||||
snapshot_prefix: zrepl_pull_backup_
|
||||
prune:
|
||||
policy: grid
|
||||
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
|
||||
```
|
||||
|
||||
The `connect` section instructs zrepl to use the `stdinserver` transport: instead of directly exposing zrepl on `prod1`
|
||||
to the internet, `backups` starts the `zrepl stdinserver` on `prod1` via SSH.
|
||||
(You can learn more about what happens [here]({{< relref "configuration/transports.md#stdinserver" >}}), or just continue following this tutorial.)
|
||||
|
||||
Thus, we need to create the SSH key pair `/etc/zrepl/ssh/prod1{,.pub}` and later pass the public part to `prod1`
|
||||
which will use it to authenticate `backups`. Execute the following commands on `backups` as the root user:
|
||||
|
||||
```bash
|
||||
cd /etc/zrepl
|
||||
mkdir -p ssh
|
||||
chmod 0700 ssh
|
||||
ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/prod1
|
||||
```
|
||||
You can learn more about the [**pull job** format here]({{< relref "configuration/jobs.md#pull" >}}) but for now we are good to go.
|
||||
|
||||
## Configure `prod1`
|
||||
|
||||
We define a corresponding **source job** named `pull_backup` in the [main configuration file](/install/#main-configuration-file)
|
||||
`zrepl.yml`:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
|
||||
- name: pull_backup
|
||||
type: source
|
||||
serve:
|
||||
type: stdinserver
|
||||
client_identity: backups.example.com
|
||||
datasets: {
|
||||
"zroot/var/db": "ok",
|
||||
"zroot/usr/home<": "ok",
|
||||
"zroot/usr/home/paranoid": "!",
|
||||
}
|
||||
snapshot_prefix: zrepl_pull_backup_
|
||||
interval: 10m
|
||||
prune:
|
||||
policy: grid
|
||||
grid: 1x1d(keep=all)
|
||||
|
||||
```
|
||||
|
||||
The `serve` section corresponds to the `connect` section in the configuration of `backups`.
|
||||
|
||||
We need to allow the SSH key on `backups` to execute `zrepl stdinserver backups.example.com` on
|
||||
`prod1`. For good measure, we will in fact enforce that only this command can be executed.
|
||||
|
||||
Open `/root/.ssh/authorized_keys` and add either of the the following lines, replacing BACKUPS_SSH_PUBKEY at the end
|
||||
of the line with the contents of `/etc/zrepl/ssh/prod1.pub` (note the **.pub** !) from `backups`.
|
||||
|
||||
```
|
||||
# for OpenSSH >= 7.2
|
||||
command="zrepl stdinserver backups.example.com",restrict BACKUPS_SSH_PUBKEY
|
||||
# for older OpenSSH versions
|
||||
command="zrepl stdinserver backups.example.com",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc BACKUPS_SSH_PUBKEY
|
||||
```
|
||||
|
||||
{{% alert theme="info" %}}The entries **must** be on a single line, including the replaced BACKUPS_SSH_PUBKEY{{% /alert %}}
|
||||
|
||||
Again, you can learn more about the [**source job** format here]({{< ref "configuration/jobs.md#source" >}}).
|
||||
|
||||
## Apply Configuration Changes
|
||||
|
||||
We need to restart the zrepl daemon on **both** `prod1` and `backups`.
|
||||
|
||||
This is [OS-specific](/install/#restarting).
|
||||
|
||||
## Watch it Work
|
||||
|
||||
A common setup is to watch the log output and zfs list of snapshots on both machines.
|
||||
|
||||
If you like tmux, here is a handy script that works on FreeBSD:
|
||||
|
||||
```bash
|
||||
pkg install gnu-watch tmux
|
||||
tmux new-window
|
||||
tmux split-window "tail -f /var/log/zrepl.log"
|
||||
tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
|
||||
tmux select-layout tiled
|
||||
```
|
||||
|
||||
The Linux equivalent might look like this
|
||||
|
||||
```bash
|
||||
# make sure tmux is installed & let's assume you use systemd + journald
|
||||
tmux new-window
|
||||
tmux split-window "journalctl -f -u zrepl.service"
|
||||
tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
|
||||
tmux select-layout tiled
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
Congratulations, you have a working pull backup. Where to go next?
|
||||
|
||||
* Read more about [configuration format, options & job types](/configuration/)
|
||||
* Learn about [implementation details](/impl/) of zrepl.
|
||||
|
||||
|
||||
|
||||
|
||||
2
docs/themes/docdock
vendored
2
docs/themes/docdock
vendored
@ -1 +1 @@
|
||||
Subproject commit f1a902a088f925e1478062d5327aa8e897752b13
|
||||
Subproject commit 9e429b2c209e6062e7bd6f53ced0a31d3b624448
|
||||
Loading…
Reference in New Issue
Block a user