extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-29 03:45:27 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs: index + tutorial rst adjustments

Browse Source
This commit is contained in:
Christian Schwarz 2017-11-09 21:17:09 +01:00
parent df181108b4
commit 2fe7f29d31
7 changed files with 94 additions and 64 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/configuration.rst
View File

@ -1,4 +1,6 @@
.. _configuration_toc:
************* *************
Configuration Configuration
************* *************

6
docs/configuration/jobs.rst
View File

@ -13,6 +13,8 @@ Check out the [tutorial]({{< relref "tutorial/_index.md" >}}) and {{< sampleconf
Whe receiving a filesystem, it is never mounted (`-u` flag) and `mountpoint=none` is set. Whe receiving a filesystem, it is never mounted (`-u` flag) and `mountpoint=none` is set.
This is temporary and being worked on {{< zrepl-issue 24 >}}. This is temporary and being worked on {{< zrepl-issue 24 >}}.
.. _job-source:
Source Job Source Job
---------- ----------
@ -47,6 +49,8 @@ Make sure you read the [prune policy documentation]({{< relref "configuration/pr
Example: {{< sampleconflink "pullbackup/productionhost.yml" >}} Example: {{< sampleconflink "pullbackup/productionhost.yml" >}}
.. _job-pull:
Pull Job Pull Job
-------- --------
@ -77,6 +81,8 @@ A pull job is the counterpart to a [source job]({{< relref "#source" >}}).
Example: {{< sampleconflink "pullbackup/backuphost.yml" >}} Example: {{< sampleconflink "pullbackup/backuphost.yml" >}}
.. _job-local:
Local Job Local Job
--------- ---------

2
docs/configuration/transports.rst
View File

@ -8,6 +8,8 @@ A transport provides an authenticated [`io.ReadWriteCloser`](https://golang.org/
Currently, only the `ssh+stdinserver` transport is supported. Currently, only the `ssh+stdinserver` transport is supported.
.. _transport-ssh+stdinserver:
`ssh+stdinserver` `ssh+stdinserver`
----------------- -----------------

2
docs/implementation.rst
View File

@ -1,3 +1,5 @@
.. _implementation_toc:
Implementation Overview Implementation Overview
======================= =======================

12
docs/index.rst
View File

@ -14,12 +14,13 @@ zrepl - ZFS replication
Getting started Getting started
~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~
The [5 minute tutorial setup]({{< relref "tutorial/_index.md" >}}) gives you a first impression. The :ref:`5 minute tutorial setup <tutorial>` gives you a first impression.
Main Features Main Features
~~~~~~~~~~~~~ ~~~~~~~~~~~~~
* Filesystem Replication * Filesystem Replication
* [x] Local & Remote * [x] Local & Remote
* [x] Pull mode * [x] Pull mode
* [ ] Push mode * [ ] Push mode
@ -27,17 +28,26 @@ Main Features
* [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules * [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules
* [x] Bookmarks support * [x] Bookmarks support
* [ ] Feature-negotiation for * [ ] Feature-negotiation for
* Resumable `send & receive` * Resumable `send & receive`
* Compressed `send & receive` * Compressed `send & receive`
* Raw encrypted `send & receive` (as soon as it is available) * Raw encrypted `send & receive` (as soon as it is available)
* Automatic snapshot creation * Automatic snapshot creation
* [x] Ensure fixed time interval between snapshots * [x] Ensure fixed time interval between snapshots
* Automatic snapshot [pruning]({{< ref "configuration/prune.md" >}}) * Automatic snapshot [pruning]({{< ref "configuration/prune.md" >}})
* [x] Age-based fading (grandfathering scheme) * [x] Age-based fading (grandfathering scheme)
* Flexible, detailed & structured [logging]({{< ref "configuration/logging.md" >}}) * Flexible, detailed & structured [logging]({{< ref "configuration/logging.md" >}})
* [x] `human`, `logfmt` and `json` formatting * [x] `human`, `logfmt` and `json` formatting
* [x] stdout, syslog and TCP (+TLS client auth) outlets * [x] stdout, syslog and TCP (+TLS client auth) outlets
* Maintainable implementation in Go * Maintainable implementation in Go
* [x] Cross platform * [x] Cross platform
* [x] Type safe & testable code * [x] Type safe & testable code

6
docs/installation.rst
View File

@ -1,3 +1,5 @@
.. _installation:
Installation Installation
============ ============
@ -36,6 +38,8 @@ Check out the sources yourself, fetch dependencies using dep, compile and instal
# see if it worked # see if it worked
zrepl help zrepl help
.. _mainconfigfile:
Configuration Files Configuration Files
------------------- -------------------
@ -78,6 +82,8 @@ There are no *rc(8)* or *systemd.service(5)* service definitions yet. Note the *
Example: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets Example: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets
in the runtime directory, it will emit an error message but not exit because other tasks such as periodic snapshots & pruning are of equal importance. in the runtime directory, it will emit an error message but not exit because other tasks such as periodic snapshots & pruning are of equal importance.
.. _install-restarting:
Restarting Restarting
~~~~~~~~~~ ~~~~~~~~~~

96
docs/tutorial.rst
View File

@ -1,3 +1,7 @@
.. |mainconfig| replace:: :ref:`main configuration file <mainconfigfile>`
.. _tutorial:
Tutorial Tutorial
======== ========
@ -5,58 +9,58 @@ Tutorial
This tutorial shows how zrepl can be used to implement a ZFS-based pull backup. This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
We assume the following scenario: We assume the following scenario:
* Production server `app-srv` with filesystems to back up: * Production server ``app-srv`` with filesystems to back up:
* `zroot/var/db` * ``zroot/var/db``
* `zroot/usr/home` and all its child filesystems * ``zroot/usr/home`` and all its child filesystems
* **except** `zroot/usr/home/paranoid` belonging to a user doing backups themselves * **except** ``zroot/usr/home/paranoid`` belonging to a user doing backups themselves
* Backup server `backup-srv` with * Backup server ``backup-srv`` with
* Filesystem `storage/zrepl/pull/app-srv` + children dedicated to backups of `app-srv` * Filesystem ``storage/zrepl/pull/app-srv`` + children dedicated to backups of ``app-srv``
Our backup solution should fulfill the following requirements: Our backup solution should fulfill the following requirements:
* Periodically snapshot the filesystems on `app-srv` *every 10 minutes* * Periodically snapshot the filesystems on ``app-srv`` *every 10 minutes*
* Incrementally replicate these snapshots to `storage/zrepl/pull/app-srv/*` on `backup-srv` * Incrementally replicate these snapshots to ``storage/zrepl/pull/app-srv/*`` on ``backup-srv``
* Keep only very few snapshots on `app-srv` to save disk space * Keep only very few snapshots on ``app-srv`` to save disk space
* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on `backup-srv` * Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on ``backup-srv``
Analysis Analysis
-------- --------
We can model this situation as two jobs: We can model this situation as two jobs:
* A **source job** on `app-srv` * A **source job** on ``app-srv``
* Creates the snapshots * Creates the snapshots
* Keeps a short history of snapshots to enable incremental replication to `backup-srv` * Keeps a short history of snapshots to enable incremental replication to ``backup-srv``
* Accepts connections from `backup-srv` * Accepts connections from ``backup-srv``
* A **pull job** on `backup-srv` * A **pull job** on ``backup-srv``
* Connects to the `zrepl daemon` process on `app-srv` * Connects to the ``zrepl daemon`` process on ``app-srv``
* Pulls the snapshots to `storage/zrepl/pull/app-srv/*` * Pulls the snapshots to ``storage/zrepl/pull/app-srv/*``
* Fades out snapshots in `storage/zrepl/pull/app-srv/*` as they age * Fades out snapshots in ``storage/zrepl/pull/app-srv/*`` as they age
Why doesn't the **pull job** create the snapshots before pulling? Why doesn't the **pull job** create the snapshots before pulling?
As is the case with all distributed systems, the link between `app-srv` and `backup-srv` might be down for an hour or two. As is the case with all distributed systems, the link between ``app-srv`` and ``backup-srv`` might be down for an hour or two.
We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage. We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
When the link comes up again, `backup-srv` will happily catch up the 12 snapshots taken by `app-srv` in the meantime, without When the link comes up again, ``backup-srv`` will happily catch up the 12 snapshots taken by ``app-srv`` in the meantime, without
a gap in our backup history. a gap in our backup history.
Install zrepl Install zrepl
------------- -------------
Follow the [OS-specific installation instructions]({{< relref "install/_index.md" >}}) and come back here. Follow the :ref:`OS-specific installation instructions <installation>` and come back here.
Configure `backup-srv` Configure ``backup-srv``
---------------------- ------------------------
We define a **pull job** named `pull_app-srv` in the [main configuration file]({{< relref "install/_index.md#configuration-files" >}} )::: We define a **pull job** named ``pull_app-srv`` in the |mainconfig|: ::
jobs: jobs:
- name: pull_app-srv - name: pull_app-srv
@ -77,10 +81,10 @@ We define a **pull job** named `pull_app-srv` in the [main configuration file]({
policy: grid policy: grid
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
The `connect` section instructs the zrepl daemon to use the `stdinserver` transport: The ``connect`` section instructs the zrepl daemon to use the ``stdinserver`` transport:
`backup-srv` will connect to the specified SSH server and expect `zrepl stdinserver CLIENT_IDENTITY` instead of the shell on the other side. ``backup-srv`` will connect to the specified SSH server and expect ``zrepl stdinserver CLIENT_IDENTITY`` instead of the shell on the other side.
It uses the private key specified at `connect.identity_file` which we still need to create::: It uses the private key specified at ``connect.identity_file`` which we still need to create:::
cd /etc/zrepl cd /etc/zrepl
mkdir -p ssh mkdir -p ssh
@ -88,15 +92,14 @@ It uses the private key specified at `connect.identity_file` which we still need
ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/identity ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/identity
Note that most use cases do not benefit from separate keypairs per remote endpoint. Note that most use cases do not benefit from separate keypairs per remote endpoint.
Thus, it is sufficient to create one keypair and use it for all `connect` directives on one host. Thus, it is sufficient to create one keypair and use it for all ``connect`` directives on one host.
Learn more about [stdinserver]({{< relref "configuration/transports.md#ssh-stdinserver" >}}) and the [**pull job** format]({{< relref "configuration/jobs.md#pull" >}}). Learn more about :ref:`transport-ssh+stdinserver` transport and the :ref:`pull job <job-pull>` format.
Configure `app-srv` Configure ``app-srv``
------------------- ---------------------
We define a corresponding **source job** named `pull_backup` in the [main configuration file]({{< relref "install/_index.md#configuration-files" >}}) We define a corresponding **source job** named ``pull_backup`` in the |mainconfig|:::
`zrepl.yml`:::
jobs: jobs:
- name: pull_backup - name: pull_backup
@ -116,12 +119,12 @@ We define a corresponding **source job** named `pull_backup` in the [main config
grid: 1x1d(keep=all) grid: 1x1d(keep=all)
The `serve` section corresponds to the `connect` section in the configuration of `backup-srv`. The ``serve`` section corresponds to the ``connect`` section in the configuration of ``backup-srv``.
We now want to authenticate `backup-srv` before allowing it to pull data. We now want to authenticate ``backup-srv`` before allowing it to pull data.
This is done by limiting SSH connections from `backup-srv` to execute the `stdinserver` subcommand. This is done by limiting SSH connections from ``backup-srv`` to execute the ``stdinserver`` subcommand.
Open `/root/.ssh/authorized_keys` and add either of the the following lines.:: Open ``/root/.ssh/authorized_keys`` and add either of the the following lines.::
# for OpenSSH >= 7.2 # for OpenSSH >= 7.2
command="zrepl stdinserver backup-srv.example.com",restrict CLIENT_SSH_KEY command="zrepl stdinserver backup-srv.example.com",restrict CLIENT_SSH_KEY
@ -130,31 +133,30 @@ Open `/root/.ssh/authorized_keys` and add either of the the following lines.::
.. ATTENTION:: .. ATTENTION::
Replace CLIENT_SSH_KEY with the contents of `/etc/zrepl/ssh/identity.pub` from `app-srv`. Replace CLIENT_SSH_KEY with the contents of ``/etc/zrepl/ssh/identity.pub`` from ``app-srv``.
Mind the trailing `.pub` in the filename. Mind the trailing ``.pub`` in the filename.
The entries **must** be on a single line, including the replaced CLIENT_SSH_KEY. The entries **must** be on a single line, including the replaced CLIENT_SSH_KEY.
.. HINT:: .. HINT::
You may need to adjust the `PermitRootLogin` option in `/etc/ssh/sshd_config` to `forced-commands-only` or higher for this to work. You may need to adjust the ``PermitRootLogin`` option in ``/etc/ssh/sshd_config`` to ``forced-commands-only`` or higher for this to work.
Refer to sshd_config(5) for details. Refer to sshd_config(5) for details.
The argument `backup-srv.example.com` is the client identity of `backup-srv` as defined in `jobs.serve.client_identity`. The argument ``backup-srv.example.com`` is the client identity of ``backup-srv`` as defined in ``jobs.serve.client_identity``.
Again, you both [stdinserver]({{< relref "configuration/transports.md#ssh-stdinserver" >}}) and the [**source job** format]({{< relref "configuration/jobs.md#source" >}}) are documented. Again, both :ref:`transport-ssh+stdinserver` transport and the :ref:`job-source` format are documented.
Apply Configuration Changes Apply Configuration Changes
--------------------------- ---------------------------
We need to restart the zrepl daemon on **both** `app-srv` and `backup-srv`. We need to restart the zrepl daemon on **both** ``app-srv`` and ``backup-srv``.
This is :ref:`OS-specific <install-restarting>`.
This is [OS-specific]({{< relref "install/_index.md#restarting" >}}).
Watch it Work Watch it Work
------------- -------------
A common setup is to `watch` the log output and `zfs list` of snapshots on both machines. A common setup is to ``watch`` the log output and ``zfs list`` of snapshots on both machines.
If you like tmux, here is a handy script that works on FreeBSD::: If you like tmux, here is a handy script that works on FreeBSD:::
@ -177,7 +179,7 @@ Summary
Congratulations, you have a working pull backup. Where to go next? Congratulations, you have a working pull backup. Where to go next?
* Read more about [configuration format, options & job types]({{< relref "configuration/_index.md" >}}) * Read more about :ref:`configuration format, options & job types <configuration_toc>`
* Learn about [implementation details]({{<relref "impl/_index.md" >}}) of zrepl. * Learn about :ref:`implementation details <implementation_toc>` of zrepl.