mirror of
https://github.com/zrepl/zrepl.git
synced 2024-11-25 09:54:47 +01:00
docs: index + tutorial rst adjustments
This commit is contained in:
parent
df181108b4
commit
2fe7f29d31
@ -1,4 +1,6 @@
|
|||||||
|
|
||||||
|
.. _configuration_toc:
|
||||||
|
|
||||||
*************
|
*************
|
||||||
Configuration
|
Configuration
|
||||||
*************
|
*************
|
||||||
|
@ -13,6 +13,8 @@ Check out the [tutorial]({{< relref "tutorial/_index.md" >}}) and {{< sampleconf
|
|||||||
Whe receiving a filesystem, it is never mounted (`-u` flag) and `mountpoint=none` is set.
|
Whe receiving a filesystem, it is never mounted (`-u` flag) and `mountpoint=none` is set.
|
||||||
This is temporary and being worked on {{< zrepl-issue 24 >}}.
|
This is temporary and being worked on {{< zrepl-issue 24 >}}.
|
||||||
|
|
||||||
|
.. _job-source:
|
||||||
|
|
||||||
Source Job
|
Source Job
|
||||||
----------
|
----------
|
||||||
|
|
||||||
@ -47,6 +49,8 @@ Make sure you read the [prune policy documentation]({{< relref "configuration/pr
|
|||||||
|
|
||||||
Example: {{< sampleconflink "pullbackup/productionhost.yml" >}}
|
Example: {{< sampleconflink "pullbackup/productionhost.yml" >}}
|
||||||
|
|
||||||
|
.. _job-pull:
|
||||||
|
|
||||||
Pull Job
|
Pull Job
|
||||||
--------
|
--------
|
||||||
|
|
||||||
@ -77,6 +81,8 @@ A pull job is the counterpart to a [source job]({{< relref "#source" >}}).
|
|||||||
|
|
||||||
Example: {{< sampleconflink "pullbackup/backuphost.yml" >}}
|
Example: {{< sampleconflink "pullbackup/backuphost.yml" >}}
|
||||||
|
|
||||||
|
.. _job-local:
|
||||||
|
|
||||||
Local Job
|
Local Job
|
||||||
---------
|
---------
|
||||||
|
|
||||||
|
@ -8,6 +8,8 @@ A transport provides an authenticated [`io.ReadWriteCloser`](https://golang.org/
|
|||||||
|
|
||||||
Currently, only the `ssh+stdinserver` transport is supported.
|
Currently, only the `ssh+stdinserver` transport is supported.
|
||||||
|
|
||||||
|
.. _transport-ssh+stdinserver:
|
||||||
|
|
||||||
`ssh+stdinserver`
|
`ssh+stdinserver`
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
|
@ -1,3 +1,5 @@
|
|||||||
|
.. _implementation_toc:
|
||||||
|
|
||||||
Implementation Overview
|
Implementation Overview
|
||||||
=======================
|
=======================
|
||||||
|
|
||||||
|
@ -14,32 +14,42 @@ zrepl - ZFS replication
|
|||||||
Getting started
|
Getting started
|
||||||
~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
The [5 minute tutorial setup]({{< relref "tutorial/_index.md" >}}) gives you a first impression.
|
The :ref:`5 minute tutorial setup <tutorial>` gives you a first impression.
|
||||||
|
|
||||||
Main Features
|
Main Features
|
||||||
~~~~~~~~~~~~~
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
* Filesystem Replication
|
* Filesystem Replication
|
||||||
* [x] Local & Remote
|
|
||||||
* [x] Pull mode
|
* [x] Local & Remote
|
||||||
* [ ] Push mode
|
* [x] Pull mode
|
||||||
* [x] Access control checks when pulling datasets
|
* [ ] Push mode
|
||||||
* [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules
|
* [x] Access control checks when pulling datasets
|
||||||
* [x] Bookmarks support
|
* [x] [Flexible mapping]({{< ref "configuration/map_filter_syntax.md" >}}) rules
|
||||||
* [ ] Feature-negotiation for
|
* [x] Bookmarks support
|
||||||
* Resumable `send & receive`
|
* [ ] Feature-negotiation for
|
||||||
* Compressed `send & receive`
|
|
||||||
* Raw encrypted `send & receive` (as soon as it is available)
|
* Resumable `send & receive`
|
||||||
|
* Compressed `send & receive`
|
||||||
|
* Raw encrypted `send & receive` (as soon as it is available)
|
||||||
|
|
||||||
* Automatic snapshot creation
|
* Automatic snapshot creation
|
||||||
* [x] Ensure fixed time interval between snapshots
|
|
||||||
|
* [x] Ensure fixed time interval between snapshots
|
||||||
|
|
||||||
* Automatic snapshot [pruning]({{< ref "configuration/prune.md" >}})
|
* Automatic snapshot [pruning]({{< ref "configuration/prune.md" >}})
|
||||||
* [x] Age-based fading (grandfathering scheme)
|
|
||||||
|
* [x] Age-based fading (grandfathering scheme)
|
||||||
|
|
||||||
* Flexible, detailed & structured [logging]({{< ref "configuration/logging.md" >}})
|
* Flexible, detailed & structured [logging]({{< ref "configuration/logging.md" >}})
|
||||||
* [x] `human`, `logfmt` and `json` formatting
|
|
||||||
* [x] stdout, syslog and TCP (+TLS client auth) outlets
|
* [x] `human`, `logfmt` and `json` formatting
|
||||||
|
* [x] stdout, syslog and TCP (+TLS client auth) outlets
|
||||||
|
|
||||||
* Maintainable implementation in Go
|
* Maintainable implementation in Go
|
||||||
* [x] Cross platform
|
|
||||||
* [x] Type safe & testable code
|
* [x] Cross platform
|
||||||
|
* [x] Type safe & testable code
|
||||||
|
|
||||||
Contributing
|
Contributing
|
||||||
~~~~~~~~~~~~
|
~~~~~~~~~~~~
|
||||||
|
@ -1,3 +1,5 @@
|
|||||||
|
.. _installation:
|
||||||
|
|
||||||
Installation
|
Installation
|
||||||
============
|
============
|
||||||
|
|
||||||
@ -36,6 +38,8 @@ Check out the sources yourself, fetch dependencies using dep, compile and instal
|
|||||||
# see if it worked
|
# see if it worked
|
||||||
zrepl help
|
zrepl help
|
||||||
|
|
||||||
|
.. _mainconfigfile:
|
||||||
|
|
||||||
Configuration Files
|
Configuration Files
|
||||||
-------------------
|
-------------------
|
||||||
|
|
||||||
@ -78,6 +82,8 @@ There are no *rc(8)* or *systemd.service(5)* service definitions yet. Note the *
|
|||||||
Example: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets
|
Example: if the daemon cannot create the [stdinserver]({{< relref "configuration/transports.md#stdinserver" >}}) sockets
|
||||||
in the runtime directory, it will emit an error message but not exit because other tasks such as periodic snapshots & pruning are of equal importance.
|
in the runtime directory, it will emit an error message but not exit because other tasks such as periodic snapshots & pruning are of equal importance.
|
||||||
|
|
||||||
|
.. _install-restarting:
|
||||||
|
|
||||||
Restarting
|
Restarting
|
||||||
~~~~~~~~~~
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
@ -1,3 +1,7 @@
|
|||||||
|
.. |mainconfig| replace:: :ref:`main configuration file <mainconfigfile>`
|
||||||
|
|
||||||
|
.. _tutorial:
|
||||||
|
|
||||||
Tutorial
|
Tutorial
|
||||||
========
|
========
|
||||||
|
|
||||||
@ -5,58 +9,58 @@ Tutorial
|
|||||||
This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
|
This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
|
||||||
We assume the following scenario:
|
We assume the following scenario:
|
||||||
|
|
||||||
* Production server `app-srv` with filesystems to back up:
|
* Production server ``app-srv`` with filesystems to back up:
|
||||||
|
|
||||||
* `zroot/var/db`
|
* ``zroot/var/db``
|
||||||
* `zroot/usr/home` and all its child filesystems
|
* ``zroot/usr/home`` and all its child filesystems
|
||||||
* **except** `zroot/usr/home/paranoid` belonging to a user doing backups themselves
|
* **except** ``zroot/usr/home/paranoid`` belonging to a user doing backups themselves
|
||||||
|
|
||||||
* Backup server `backup-srv` with
|
* Backup server ``backup-srv`` with
|
||||||
|
|
||||||
* Filesystem `storage/zrepl/pull/app-srv` + children dedicated to backups of `app-srv`
|
* Filesystem ``storage/zrepl/pull/app-srv`` + children dedicated to backups of ``app-srv``
|
||||||
|
|
||||||
Our backup solution should fulfill the following requirements:
|
Our backup solution should fulfill the following requirements:
|
||||||
|
|
||||||
* Periodically snapshot the filesystems on `app-srv` *every 10 minutes*
|
* Periodically snapshot the filesystems on ``app-srv`` *every 10 minutes*
|
||||||
* Incrementally replicate these snapshots to `storage/zrepl/pull/app-srv/*` on `backup-srv`
|
* Incrementally replicate these snapshots to ``storage/zrepl/pull/app-srv/*`` on ``backup-srv``
|
||||||
* Keep only very few snapshots on `app-srv` to save disk space
|
* Keep only very few snapshots on ``app-srv`` to save disk space
|
||||||
* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on `backup-srv`
|
* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on ``backup-srv``
|
||||||
|
|
||||||
Analysis
|
Analysis
|
||||||
--------
|
--------
|
||||||
|
|
||||||
We can model this situation as two jobs:
|
We can model this situation as two jobs:
|
||||||
|
|
||||||
* A **source job** on `app-srv`
|
* A **source job** on ``app-srv``
|
||||||
|
|
||||||
* Creates the snapshots
|
* Creates the snapshots
|
||||||
* Keeps a short history of snapshots to enable incremental replication to `backup-srv`
|
* Keeps a short history of snapshots to enable incremental replication to ``backup-srv``
|
||||||
* Accepts connections from `backup-srv`
|
* Accepts connections from ``backup-srv``
|
||||||
|
|
||||||
* A **pull job** on `backup-srv`
|
* A **pull job** on ``backup-srv``
|
||||||
|
|
||||||
* Connects to the `zrepl daemon` process on `app-srv`
|
* Connects to the ``zrepl daemon`` process on ``app-srv``
|
||||||
* Pulls the snapshots to `storage/zrepl/pull/app-srv/*`
|
* Pulls the snapshots to ``storage/zrepl/pull/app-srv/*``
|
||||||
* Fades out snapshots in `storage/zrepl/pull/app-srv/*` as they age
|
* Fades out snapshots in ``storage/zrepl/pull/app-srv/*`` as they age
|
||||||
|
|
||||||
|
|
||||||
Why doesn't the **pull job** create the snapshots before pulling?
|
Why doesn't the **pull job** create the snapshots before pulling?
|
||||||
|
|
||||||
As is the case with all distributed systems, the link between `app-srv` and `backup-srv` might be down for an hour or two.
|
As is the case with all distributed systems, the link between ``app-srv`` and ``backup-srv`` might be down for an hour or two.
|
||||||
We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
|
We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
|
||||||
|
|
||||||
When the link comes up again, `backup-srv` will happily catch up the 12 snapshots taken by `app-srv` in the meantime, without
|
When the link comes up again, ``backup-srv`` will happily catch up the 12 snapshots taken by ``app-srv`` in the meantime, without
|
||||||
a gap in our backup history.
|
a gap in our backup history.
|
||||||
|
|
||||||
Install zrepl
|
Install zrepl
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
Follow the [OS-specific installation instructions]({{< relref "install/_index.md" >}}) and come back here.
|
Follow the :ref:`OS-specific installation instructions <installation>` and come back here.
|
||||||
|
|
||||||
Configure `backup-srv`
|
Configure ``backup-srv``
|
||||||
----------------------
|
------------------------
|
||||||
|
|
||||||
We define a **pull job** named `pull_app-srv` in the [main configuration file]({{< relref "install/_index.md#configuration-files" >}} ):::
|
We define a **pull job** named ``pull_app-srv`` in the |mainconfig|: ::
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
- name: pull_app-srv
|
- name: pull_app-srv
|
||||||
@ -77,10 +81,10 @@ We define a **pull job** named `pull_app-srv` in the [main configuration file]({
|
|||||||
policy: grid
|
policy: grid
|
||||||
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
|
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
|
||||||
|
|
||||||
The `connect` section instructs the zrepl daemon to use the `stdinserver` transport:
|
The ``connect`` section instructs the zrepl daemon to use the ``stdinserver`` transport:
|
||||||
`backup-srv` will connect to the specified SSH server and expect `zrepl stdinserver CLIENT_IDENTITY` instead of the shell on the other side.
|
``backup-srv`` will connect to the specified SSH server and expect ``zrepl stdinserver CLIENT_IDENTITY`` instead of the shell on the other side.
|
||||||
|
|
||||||
It uses the private key specified at `connect.identity_file` which we still need to create:::
|
It uses the private key specified at ``connect.identity_file`` which we still need to create:::
|
||||||
|
|
||||||
cd /etc/zrepl
|
cd /etc/zrepl
|
||||||
mkdir -p ssh
|
mkdir -p ssh
|
||||||
@ -88,15 +92,14 @@ It uses the private key specified at `connect.identity_file` which we still need
|
|||||||
ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/identity
|
ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/identity
|
||||||
|
|
||||||
Note that most use cases do not benefit from separate keypairs per remote endpoint.
|
Note that most use cases do not benefit from separate keypairs per remote endpoint.
|
||||||
Thus, it is sufficient to create one keypair and use it for all `connect` directives on one host.
|
Thus, it is sufficient to create one keypair and use it for all ``connect`` directives on one host.
|
||||||
|
|
||||||
Learn more about [stdinserver]({{< relref "configuration/transports.md#ssh-stdinserver" >}}) and the [**pull job** format]({{< relref "configuration/jobs.md#pull" >}}).
|
Learn more about :ref:`transport-ssh+stdinserver` transport and the :ref:`pull job <job-pull>` format.
|
||||||
|
|
||||||
Configure `app-srv`
|
Configure ``app-srv``
|
||||||
-------------------
|
---------------------
|
||||||
|
|
||||||
We define a corresponding **source job** named `pull_backup` in the [main configuration file]({{< relref "install/_index.md#configuration-files" >}})
|
We define a corresponding **source job** named ``pull_backup`` in the |mainconfig|:::
|
||||||
`zrepl.yml`:::
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
- name: pull_backup
|
- name: pull_backup
|
||||||
@ -116,12 +119,12 @@ We define a corresponding **source job** named `pull_backup` in the [main config
|
|||||||
grid: 1x1d(keep=all)
|
grid: 1x1d(keep=all)
|
||||||
|
|
||||||
|
|
||||||
The `serve` section corresponds to the `connect` section in the configuration of `backup-srv`.
|
The ``serve`` section corresponds to the ``connect`` section in the configuration of ``backup-srv``.
|
||||||
|
|
||||||
We now want to authenticate `backup-srv` before allowing it to pull data.
|
We now want to authenticate ``backup-srv`` before allowing it to pull data.
|
||||||
This is done by limiting SSH connections from `backup-srv` to execute the `stdinserver` subcommand.
|
This is done by limiting SSH connections from ``backup-srv`` to execute the ``stdinserver`` subcommand.
|
||||||
|
|
||||||
Open `/root/.ssh/authorized_keys` and add either of the the following lines.::
|
Open ``/root/.ssh/authorized_keys`` and add either of the the following lines.::
|
||||||
|
|
||||||
# for OpenSSH >= 7.2
|
# for OpenSSH >= 7.2
|
||||||
command="zrepl stdinserver backup-srv.example.com",restrict CLIENT_SSH_KEY
|
command="zrepl stdinserver backup-srv.example.com",restrict CLIENT_SSH_KEY
|
||||||
@ -130,31 +133,30 @@ Open `/root/.ssh/authorized_keys` and add either of the the following lines.::
|
|||||||
|
|
||||||
.. ATTENTION::
|
.. ATTENTION::
|
||||||
|
|
||||||
Replace CLIENT_SSH_KEY with the contents of `/etc/zrepl/ssh/identity.pub` from `app-srv`.
|
Replace CLIENT_SSH_KEY with the contents of ``/etc/zrepl/ssh/identity.pub`` from ``app-srv``.
|
||||||
Mind the trailing `.pub` in the filename.
|
Mind the trailing ``.pub`` in the filename.
|
||||||
The entries **must** be on a single line, including the replaced CLIENT_SSH_KEY.
|
The entries **must** be on a single line, including the replaced CLIENT_SSH_KEY.
|
||||||
|
|
||||||
|
|
||||||
.. HINT::
|
.. HINT::
|
||||||
|
|
||||||
You may need to adjust the `PermitRootLogin` option in `/etc/ssh/sshd_config` to `forced-commands-only` or higher for this to work.
|
You may need to adjust the ``PermitRootLogin`` option in ``/etc/ssh/sshd_config`` to ``forced-commands-only`` or higher for this to work.
|
||||||
Refer to sshd_config(5) for details.
|
Refer to sshd_config(5) for details.
|
||||||
|
|
||||||
The argument `backup-srv.example.com` is the client identity of `backup-srv` as defined in `jobs.serve.client_identity`.
|
The argument ``backup-srv.example.com`` is the client identity of ``backup-srv`` as defined in ``jobs.serve.client_identity``.
|
||||||
|
|
||||||
Again, you both [stdinserver]({{< relref "configuration/transports.md#ssh-stdinserver" >}}) and the [**source job** format]({{< relref "configuration/jobs.md#source" >}}) are documented.
|
Again, both :ref:`transport-ssh+stdinserver` transport and the :ref:`job-source` format are documented.
|
||||||
|
|
||||||
Apply Configuration Changes
|
Apply Configuration Changes
|
||||||
---------------------------
|
---------------------------
|
||||||
|
|
||||||
We need to restart the zrepl daemon on **both** `app-srv` and `backup-srv`.
|
We need to restart the zrepl daemon on **both** ``app-srv`` and ``backup-srv``.
|
||||||
|
This is :ref:`OS-specific <install-restarting>`.
|
||||||
This is [OS-specific]({{< relref "install/_index.md#restarting" >}}).
|
|
||||||
|
|
||||||
Watch it Work
|
Watch it Work
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
A common setup is to `watch` the log output and `zfs list` of snapshots on both machines.
|
A common setup is to ``watch`` the log output and ``zfs list`` of snapshots on both machines.
|
||||||
|
|
||||||
If you like tmux, here is a handy script that works on FreeBSD:::
|
If you like tmux, here is a handy script that works on FreeBSD:::
|
||||||
|
|
||||||
@ -177,7 +179,7 @@ Summary
|
|||||||
|
|
||||||
Congratulations, you have a working pull backup. Where to go next?
|
Congratulations, you have a working pull backup. Where to go next?
|
||||||
|
|
||||||
* Read more about [configuration format, options & job types]({{< relref "configuration/_index.md" >}})
|
* Read more about :ref:`configuration format, options & job types <configuration_toc>`
|
||||||
* Learn about [implementation details]({{<relref "impl/_index.md" >}}) of zrepl.
|
* Learn about :ref:`implementation details <implementation_toc>` of zrepl.
|
||||||
|
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user