extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-25 01:44:43 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs: reflect changes in replication_rewrite branch

Browse Source
This commit is contained in:
Christian Schwarz 2018-10-11 17:46:26 +02:00
parent 125b561df3
commit 1643198713
16 changed files with 823 additions and 554 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

144
docs/changelog.rst
View File

@ -1,81 +1,103 @@
.. |break_config| replace:: **[BREAK]**
.. |break_config| replace:: **[CONFIG]**
.. |break| replace:: **[BREAK]**
.. |bugfix| replace:: [BUG]
.. |docs| replace:: [DOCS]
.. |feature| replace:: [FEATURE]
.. _changelog:
Changelog
=========
The changelog summarized bugfixes that are deemed relevant for users.
The changelog summarizes bugfixes that are deemed relevant for users and package maintainers.
Developers should consult the git commit log or GitHub issue tracker.
0.0.4 (unreleased)
e use the following annotations for classifying changes:
* |break_config| Change that breaks the config.
As a package maintainer, make sure to warn your users about config breakage somehow.
* |break| Change that breaks interoperability or persistent state representation with previous releases.
As a package maintainer, make sure to warn your users about config breakage somehow.
Note that even updating the package on both sides might not be sufficient, e.g. if persistent state needs to be migrated to a new format.
* |feature| Change that introduces new functionality.
* |bugfix| Change that fixes a bug, no regressions or incompatibilities expected.
* |docs| Change to the documentation.
0.1 (unreleased)
------------------
This release is a milestone for zrepl and required significant refactoring if not rewrites of substantial parts of the application.
It breaks both configuration and transport format, and thus requires manual intervention and updates on both sides of a replication setup.
Notes to Package Maintainer
~~~~~~~~~~~~~~~~~~~~~~~~~~~
* If the daemon crashes, the stack trace produced by the Go runtime and possibly diagnostic output of zrepl will be written to stderr.
This behavior is independent from the ``stdout`` outlet type.
Please make sure the stderr output of the daemon is captured to a file.
Rotation should not be necessary because stderr is not written to under normal circumstances.
To conserve precious stack traces, make sure that multiple service restarts do not directly discard previous stderr output.
* Make it obvious for users how to set the ``GOTRACEBACK`` environment variable to ``GOTRACEBACK=crash``.
This functionality will cause SIGABRT on panics and can be used to capture a coredump of the panicking process.
To that extend, make sure that your package build system, your OS's coredump collection and the Go delve debugger work together.
Use your build system to package the Go program in `this tutorial on Go coredumps and the delve debugger <https://rakyll.org/coredumps/>`_ , and make sure the symbol resolution etc. work on coredumps captured from the binary produced by your build system. (Special focus on symbol stripping, etc.)
Changes
~~~~~~~
* |feature| :issue:`55` : Push replication (see :ref:`push job <job-push>` and :ref:`sink job <job-sink>`)
* |feature| :ref:`TCP Transport <transport-tcp>`
* |feature| :ref:`TCP + TLS client authentication transport <transport-tcp+tlsclientauth>`
* |feature| :issue:`78` TODO MERGE COMMIT Replication protocol rewrite
* Uses ``github.com/problame/go-streamrpc`` for RPC layer
* |break| zrepl 0.1 and restart on both sides of a replication setup is required
* |feature| :issue:`83`: Improved error handling of network-level errors (zrepl retries instead of failing the entire job)
* |bugfix| :issue:`75` :issue:`81`: use connection timeouts and protocol-level heartbeats
* |break| |break_config|: mappings are no longer supported
* Receiving sides (``pull`` and ``sink`` job) specify a single ``root_fs``.
Received filesystems are then stored *per client* in ``${root_fs}/${client_identity}``.
* |feature| |break| |break_config| Manual snapshotting + triggering of replication
* |feature| :issue:`69`: include manually created snapshots in replication
* |break_config| ``manual`` and ``periodic`` :ref:`snapshotting types <job-snapshotting-spec>`
* |feature| ``zrepl wakeup JOB`` subcommand to trigger *just* replication
* |feature| |break| |break_config| New pruning system
* The active side of a replication (pull or push) decides what to prune for both sender and receiver.
The RPC protocol is used to execute the destroy operations on the remote side.
* New pruning policies (see :ref:`configuration documentation <prune>` )
* The decision what snapshots shall be pruned is now made based on *keep rules*
* |feature| :issue:`68`: keep rule ``not_replicated`` prevents divergence of sender and receiver
* |feature| |break| Bookmark pruning is no longer necessary
* Per filesystem, zrepl creates a single bookmark (``#zrepl_replication_cursor``) and moves it forward with the most recent successfully replicated snapshot on the receiving side.
* Old bookmarks created prior to zrepl 0.1 (named like their corresponding snapshot) must be deleted manually.
* |break_config| ``keep_bookmarks`` parameter of the ``grid`` keep rule has been removed
* |feature| ``zrepl status`` for live-updating replication progress (it's really cool!)
* |feature| :issue:`67`: Expose `Prometheus <https://prometheus.io>`_ metrics via HTTP (:ref:`config docs <monitoring-prometheus>`)
0.0.3
-----
* |break_config| Logging outlet types must be specified using the ``type`` instead of ``outlet`` key
* |break| :issue:`53`: CLI: ``zrepl control *`` subcommands have been made direct subcommands of ``zrepl *``
* |break_config| |feature| :issue:`34`: automatic bookmarking of snapshots
* |bugfix| :issue:`81` :issue:`77` : handle failed accepts correctly (``source`` job)
* Snapshots are automatically bookmarked and pruning of bookmarks **must** be configured.
* This breaks existing configuration: ``grid`` :ref:`prune policy <prune-retention-grid>` specifications require the new ``keep_bookmarks`` parameter.
* Make sure to understand the meaning bookmarks have for :ref:`maximum replication downtime <replication-downtime>`.
* Example: :sampleconf:`pullbackup/productionhost.yml`
.. |lastrelease| replace:: 0.0.3
* |break| :commit:`ccd062e`: ``ssh+stdinserver`` transport: changed protocol requires daemon restart on both sides
Previous Releases
-----------------
* The delicate procedure of talking to the serving-side zrepl daemon via the stdinserver proxy command now has better error handling.
* This includes handshakes between client+proxy and client + remote daemo, which is not implemented in previous versions of zrepl.
* The connecting side will therefore time out, with the message ``dial_timeout of 10s exceeded``.
* Both sides of a replication setup must be updated and restarted. Otherwise the connecting side will hang and not time out.
* |break_config| :commit:`2bfcfa5`: first outlet in ``global.logging`` is now used for logging meta-errors, for example problems encountered when writing to other outlets.
* |feature| :issue:`10`: ``zrepl control status`` subcommand
* Allows inspection of job activity per task and their log output at runtime.
* Supports ``--format raw`` option for JSON output, usable for monitoring from scripts.
* |feature| :commit:`d7f3fb9`: subcommand bash completions
* Package maintainers should install this as appropriate.
* |bugfix| :issue:`61`: fix excessive memory usage
* |bugfix| :issue:`8` and :issue:`56`: ``ssh+stdinserver`` transport properly reaps SSH child processes
* |bugfix| :commit:`cef63ac`: ``human`` format now prints non-string values correctly
* |bugfix| :issue:`26`: slow TCP outlets no longer block the daemon
* |docs| :issue:`64`: tutorial: document ``known_host`` file entry
0.0.2
-----
* |break_config| :commit:`b95260f`: ``global.logging`` is no longer a dictionary but a list
* |break_config| :commit:`3e647c1`: ``source`` job field ``datasets`` renamed to ``filesystems``
* **NOTE**: zrepl will parse missing ``filesystems`` field as an empty filter,
i.e. no filesystems are presented to the other side.
* |bugfix| :commit:`72d2885` fix aliasing bug with root `<` subtree wildcard
* Filesystems paths with final match at blank `s` subtree wildcard are now appended to the target path
* Non-root subtree wildcards, e.g. `zroot/foo/bar<` still map directrly onto the target path
* Support days (``d``) and weeks (``w``) in durations
* Docs
* Ditch Hugo, move to Python Sphinx
* Improve & simplify tutorial (single SSH key per installation)
* Document pruning policies
* Document job types
* Document logging
* Start updating implementation overview
.. NOTE::
Due to limitations in our documentation system, we only show the changelog since the last release and the time this documentation is built.
For the changelog of previous releases, use the version selection in the hosted version of these docs at `zrepl.github.io <https://zrepl.github.io>`_.
0.0.1
-----
* Initial release
W

2
docs/configuration.rst
View File

@ -10,7 +10,7 @@ Configuration
configuration/preface
configuration/jobs
configuration/transports
configuration/map_filter_syntax
configuration/filter_syntax
configuration/prune
configuration/logging
configuration/monitoring

67
docs/configuration/filter_syntax.rst Normal file
View File

@ -0,0 +1,67 @@
.. include:: ../global.rst.inc
.. _pattern-filter:
Filter Syntax
=============
For :ref:`source jobs <job-source>` and :ref:`push jobs <job-push>`, a filesystem filter must be defined (field ``filesystems``).
A filter takes a filesystem path (in the ZFS filesystem hierarchy) as parameter and returns ``true`` (pass) or ``false`` (block).
A filter is specified as a **YAML dictionary** with patterns as keys and booleans as values.
The following rules determine which result is chosen for a given filesystem path:
* More specific path patterns win over less specific ones
* Non-wildcard patterns (full path patterns) win over *subtree wildcards* (`<` at end of pattern)
* If the path in question does not match any pattern, the result is ``false``.
The **subtree wildcard** ``<`` means "the dataset left of ``<`` and all its children".
.. TIP::
You can try out patterns for a configured job using the ``zrepl test`` subcommand.
Examples
--------
Full Access
~~~~~~~~~~~
The following configuration will allow access to all filesystems.
::
jobs:
- type: source
filesystems: {
"<": true,
}
...
Fine-grained
~~~~~~~~~~~~
The following configuration demonstrates all rules presented above.
::
jobs:
- type: source
filesystems: {
"tank<": true, # rule 1
"tank/foo<": false, # rule 2
"tank/foo/bar": true, # rule 3
}
...
Which rule applies to given path, and what is the result?
::
tank/foo/bar/loo => 2 false
tank/bar => 1 true
tank/foo/bar => 3 true
zroot => NONE false
tank/var/log => 1 true

384
docs/configuration/jobs.rst
View File

@ -1,20 +1,106 @@
.. include:: ../global.rst.inc
.. |patient| replace:: :ref:`patient <job-term-patient>`
.. |serve-transport| replace:: :ref:`serve transport<transport-ssh+stdinserver-serve>`
.. |connect-transport| replace:: :ref:`connect transport<transport-ssh+stdinserver-connect>`
.. |mapping| replace:: :ref:`mapping <pattern-mapping>`
.. |filter| replace:: :ref:`filter <pattern-filter>`
.. |prune| replace:: :ref:`prune <prune>`
.. |serve-transport| replace:: :ref:`serve specification<transport>`
.. |connect-transport| replace:: :ref:`connect specification<transport>`
.. |snapshotting-spec| replace:: :ref:`snapshotting specification <job-snapshotting-spec>`
.. |pruning-spec| replace:: :ref:`pruning specification <prune>`
.. |filter-spec| replace:: :ref:`filter specification<pattern-filter>`
.. _job:
Job Types
=========
Job Types & Replication
=======================
Overview & Terminology
----------------------
A *job* is the unit of activity tracked by the zrepl daemon and configured in the |mainconfig|.
Every job has a unique ``name``, a ``type`` and type-dependent fields which are documented on this page.
Check out the :ref:`tutorial` and :sampleconf:`/` for examples on how job types are actually used.
Replication always happens between a pair of jobs: one is the **active side**, and one the **passive side**.
The active side executes the replication logic whereas the passive side responds to requests after checking the active side's permissions.
For communication, the active side connects to the passive side using a :ref:`transport <transport>` and starts issuing remote procedure calls (RPCs).
The following table shows how different job types can be combined to achieve both push and pull mode setups:
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
| Setup name | active side | passive side | use case |
+=======================+==============+==================================+===============================================+
| Push mode | ``push`` | ``sink`` | * Laptop backup |
| | | | * NAS behind NAT to offsite |
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
| Pull mode | ``pull`` | ``source`` | * Central backup-server for many nodes |
| | | | * Remote server to NAS behind NAT |
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
| Local replication | | ``push`` + ``sink`` in one config | * Backup FreeBSD boot pool |
| | | with :ref:`local transport <transport-local>` | |
+-----------------------+--------------+----------------------------------+-----------------------------------------------+
How the Active Side Works
~~~~~~~~~~~~~~~~~~~~~~~~~
The active side (:ref:`push <job-push>` and :ref:`pull <job-pull>` job) executes the replication and pruning logic:
* Wakeup because of finished snapshotting (``push`` job) or pull interval ticker (``pull`` job).
* Connect to the corresponding passive side using a :ref:`transport <transport>` and instantiate an RPC client.
* Replicate data from the sending to the receiving side.
* Prune on sender & receiver.
.. TIP::
The progress of the active side can be watched live using the ``zrepl status`` subcommand.
How the Passive Side Works
~~~~~~~~~~~~~~~~~~~~~~~~~~
The passive side (:ref:`sink <job-sink>` and :ref:`source <job-source>`) waits for connections from the corresponding active side,
using the transport listener type specified in the ``serve`` field of the job configuration.
Each transport listener provides a client's identity to the passive side job.
It uses the client identity for access control:
* The ``sink`` job only allows pushes to those ZFS filesystems to the active side that are located below ``root_fs/${client_identity}``.
* The ``source`` job has a whitelist of client identities that are allowed pull access.
.. TIP::
The implementation of the ``sink`` job requires that the connecting client identities be a valid ZFS filesystem name components.
How Replication Works
~~~~~~~~~~~~~~~~~~~~~
One of the major design goals of the replication module is to avoid any duplication of the nontrivial logic.
As such, the code works on abstract senders and receiver **endpoints**, where typically one will be implemented by a local program object and the other is an RPC client instance.
Regardless of push- or pull-style setup, the logic executes on the active side, i.e. in the ``push`` or ``pull`` job.
The following steps take place during replication and can be monitored using the ``zrepl status`` subcommand:
* Plan the replication:
* Compare sender and receiver filesystem snapshots
* Build the **replication plan**
* Per filesystem, compute a diff between sender and receiver snapshots
* Build a list of replication steps
* If possible, use incremental sends (``zfs send -i``)
* Otherwise, use full send of most recent snapshot on sender
* Give up on filesystems that cannot be replicated without data loss
* Retry on errors that are likely temporary (i.e. network failures).
* Give up on filesystems where a permanent error was received over RPC.
* Execute the plan
* Perform replication steps in the following order:
Among all filesystems with pending replication steps, pick the filesystem whose next replication step's snapshot is the oldest.
* After a successful replication step, update the replication cursor bookmark (see below)
The idea behind the execution order of replication steps is that if the sender snapshots all filesystems simultaneously at fixed intervals, the receiver will have all filesystems snapshotted at time ``T1`` before the first snapshot at ``T2 = T1 + $interval`` is replicated.
.. _replication-cursor-bookmark:
The **replication cursor bookmark** ``#zrepl_replication_cursor`` is kept per filesystem on the sending side of a replication setup:
It is a bookmark of the most recent successfully replicated snapshot to the receiving side.
It is is used by the :ref:`not_replicated <prune-keep-not-replicated>` keep rule to identify all snapshots that have not yet been replicated to the receiving side.
Regardless of whether that keep rule is used, the bookmark ensures that replication can always continue incrementally.
.. ATTENTION::
@ -22,12 +108,133 @@ Check out the :ref:`tutorial` and :sampleconf:`/` for examples on how job types
Whe receiving a filesystem, it is never mounted (`-u` flag) and `mountpoint=none` is set.
This is temporary and being worked on :issue:`24`.
.. _job-snapshotting-spec:
Taking Snaphots
---------------
The ``push`` and ``source`` jobs can automatically take periodic snapshots of the filesystems matched by the ``filesystems`` filter field.
The snapshot names are composed of a user-defined prefix followed by a UTC date formatted like ``20060102_150405_000``.
We use UTC because it will avoid name conflicts when switching time zones or between summer and winter time.
For ``push`` jobs, replication is automatically triggered after all filesystems have been snapshotted.
::
jobs:
- type: push
filesystems: {
"<": true,
"tmp": false
}
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
...
There is also a ``manual`` snapshotting type, which covers the following use cases:
* Existing infrastructure for automatic snapshots: you only want to use zrepl for replication.
* Run scripts before and after taking snapshots (like locking database tables).
We are working on better integration for this use case: see :issue:`74`.
Note that you will have to trigger replication manually using the ``zrepl wakeup JOB`` subcommand in that case.
::
jobs:
- type: push
filesystems: {
"<": true,
"tmp": false
}
snapshotting:
type: manual
...
.. _job-push:
Job Type ``push``
-----------------
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``push``
* - ``name``
- unique name of the job
* - ``connect``
- |connect-transport|
* - ``filesystems``
- |filter-spec| for filesystems to be snapshotted and pushed to the sink
* - ``snapshotting``
- |snapshotting-spec|
* - ``pruning``
- |pruning-spec|
Example config: :sampleconf:`/push.yml`
.. _job-sink:
Job Type ``sink``
-----------------
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``sink``
* - ``name``
- unique name of the job
* - ``serve``
- |serve-transport|
* - ``root_fs``
- ZFS dataset path are received to
``$root_fs/$client_identity``
Example config: :sampleconf:`/sink.yml`
.. _job-pull:
Job Type ``pull``
-----------------
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``pull``
* - ``name``
- unique name of the job
* - ``connect``
- |connect-transport|
* - ``root_fs``
- ZFS dataset path are received to
``$root_fs/$client_identity``
* - ``interval``
- Interval at which to pull from the source job
* - ``pruning``
- |pruning-spec|
Example config: :sampleconf:`/pull.yml`
.. _job-source:
Source Job
----------
Example: :sampleconf:`pullbackup/productionhost.yml`.
Job Type ``source``
-------------------
.. list-table::
:widths: 20 80
@ -42,142 +249,19 @@ Example: :sampleconf:`pullbackup/productionhost.yml`.
* - ``serve``
- |serve-transport|
* - ``filesystems``
- |filter| for filesystems to expose to client
* - ``snapshot_prefix``
- prefix for ZFS snapshots taken by this job
* - ``interval``
- snapshotting interval
* - ``prune``
- |prune| for versions of filesytems in ``filesystems``, versions prefixed with ``snapshot_prefix``
- |filter-spec| for filesystems to be snapshotted and exposed to connecting clients
* - ``snapshotting``
- |snapshotting-spec|
Example config: :sampleconf:`/source.yml`
.. _replication-local:
Local replication
-----------------
If you have the need for local replication (most likely between two local storage pools), you can use the :ref:`local transport type <transport-local>` to connect a local push job to a local sink job.
Example config: :sampleconf:`/local.yml`.
- Snapshotting Task (every ``interval``, |patient|)
- A snapshot of filesystems matched by ``filesystems`` is taken every ``interval`` with prefix ``snapshot_prefix``.
- A bookmark of that snapshot is created with the same name.
- The ``prune`` policy is evaluated for versions of filesystems matched by ``filesystems``, versions prefixed with ``snapshot_prefix``.
- Serve Task
- Wait for connections from pull job using ``serve``.
A source job is the counterpart to a :ref:`job-pull`.
Make sure you read the |prune| policy documentation.
.. _job-pull:
Pull Job
--------
Example: :sampleconf:`pullbackup/backuphost.yml`
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``pull``
* - ``name``
- unqiue name of the job
* - ``connect``
- |connect-transport|
* - ``interval``
- Interval between pull attempts
* - ``mapping``
- |mapping| for remote to local filesystems
* - ``snapshot_prefix``
- prefix snapshots must match to be considered for replication & pruning
* - ``prune``
- |prune| policy for versions of filesystems of local filesystems reachable by ``mapping``, versions prefixed with ``snapshot_prefix``
* Main Task (every ``interval``, |patient|)
#. A connection to the remote source job is established using the strategy in ``connect``
#. ``mapping`` maps filesystems presented by the remote side to local *target filesystems*
#. Those remote filesystems with a local *target filesystem* are replicated
#. Only snapshots with prefix ``snapshot_prefix`` are replicated.
#. If possible, incremental replication takes place.
#. If the local target filesystem does not exist, the most recent snapshot is sent fully (non-incremental).
#. On conflicts, an error is logged but replication of other filesystems with mapping continues.
#. The ``prune`` policy is evaluated for all *target filesystems*
A pull job is the counterpart to a :ref:`job-source`.
Make sure you read the |prune| policy documentation.
.. _job-local:
Local Job
---------
Example: :sampleconf:`localbackup/host1.yml`
.. list-table::
:widths: 20 80
:header-rows: 1
* - Parameter
- Comment
* - ``type``
- = ``local``
* - ``name``
- unqiue name of the job
* - ``mapping``
- |mapping| from source to target filesystem (both local)
* - ``snapshot_prefix``
- prefix for ZFS snapshots taken by this job
* - ``interval``
- snapshotting & replication interval
* - ``prune_lhs``
- pruning policy on left-hand-side (source)
* - ``prune_rhs``
- pruning policy on right-hand-side (target)
* Main Task (every ``interval``, |patient|)
#. Evaluate ``mapping`` for local filesystems, those with a *target filesystem* are called *mapped filesystems*.
#. Snapshot *mapped filesystems* with ``snapshot_prefix``.
#. Bookmark the snapshot created above.
#. Replicate *mapped filesystems* to their respective *target filesystems*:
#. Only snapshots with prefix ``snapshot_prefix`` are replicated.
#. If possible, incremental replication takes place.
#. If the *target filesystem* does not exist, the most recent snapshot is sent fully (non-incremental).
#. On conflicts, an error is logged but replication of other *mapped filesystems* continues.
#. The ``prune_lhs`` policy is triggered for all *mapped filesystems*
#. The ``prune_rhs`` policy is triggered for all *target filesystems*
A local job is combination of source & pull job executed on the same machine.
Terminology
-----------
task
A job consists of one or more tasks and a task consists of one or more steps.
Some tasks may be periodic while others wait for an event to occur.
patient task
.. _job-term-patient:
A patient task is supposed to execute some task every `interval`.
We call the start of the task an *invocation*.
* If the task completes in less than `interval`, the task is restarted at `last_invocation + interval`.
* Otherwise, a patient job
* logs a warning as soon as a task exceeds its configured `interval`
* waits for the last invocation to finish
* logs a warning with the effective task duration
* immediately starts a new invocation of the task
filesystem version
A snapshot or a bookmark.

25
docs/configuration/logging.rst
View File

@ -8,18 +8,17 @@ Logging
zrepl uses structured logging to provide users with easily processable log messages.
Logging outlets are configured in the ``global`` section of the |mainconfig|.
Check out :sampleconf:`random/logging_and_monitoring.yml` for an example on how to configure multiple outlets:
::
global:
logging:
- outlet: OUTLET_TYPE
- type: OUTLET_TYPE
level: MINIMUM_LEVEL
format: FORMAT
- outlet: OUTLET_TYPE
- type: OUTLET_TYPE
level: MINIMUM_LEVEL
format: FORMAT
@ -45,7 +44,7 @@ By default, the following logging configuration is used
global:
logging:
- outlet: "stdout"
- type: "stdout"
level: "warn"
format: "human"
@ -93,8 +92,8 @@ Formats
* - Format
- Description
* - ``human``
- emphasizes context by putting job, task, step and other context variables into brackets
before the actual message, followed by remaining fields in logfmt style|
- prints job and subsystem into brackets before the actual message,
followed by remaining fields in logfmt style
* - ``logfmt``
- `logfmt <https://brandur.org/logfmt>`_ output. zrepl uses `this Go package <https://github.com/go-logfmt/logfmt>`_.
* - ``json``
@ -118,7 +117,7 @@ Outlets are the destination for log entries.
* - Parameter
- Comment
* - ``outlet``
* - ``type``
- ``stdout``
* - ``level``
- minimum :ref:`log level <logging-levels>`
@ -126,9 +125,11 @@ Outlets are the destination for log entries.
- output :ref:`format <logging-formats>`
* - ``time``
- always include time in output (``true`` or ``false``)
* - ``color``
- colorize output according to log level (``true`` or ``false``)
Writes all log entries with minimum level ``level`` formatted by ``format`` to stdout.
If stdout is a tty, interactive usage is assumed and the current time is included in the output.
If stdout is a tty, interactive usage is assumed and both ``time`` and ``color`` are set to ``true``.
Can only be specified once.
@ -140,7 +141,7 @@ Can only be specified once.
* - Parameter
- Comment
* - ``outlet``
* - ``type``
- ``syslog``
* - ``level``
- minimum :ref:`log level <logging-levels>`
@ -163,7 +164,7 @@ Can only be specified once.
* - Parameter
- Comment
* - ``outlet``
* - ``type``
- ``tcp``
* - ``level``
- minimum :ref:`log level <logging-levels>`
@ -179,11 +180,9 @@ Can only be specified once.
- TLS config (see below)
Establishes a TCP connection to ``address`` and sends log messages with minimum level ``level`` formatted by ``format``.
If ``tls`` is not specified, an unencrypted connection is established.
If ``tls`` is specified, the TCP connection is secured with TLS + Client Authentication.
This is particularly useful in combination with log aggregation services that run on an other machine.
The latter is particularly useful in combination with log aggregation services.
.. list-table::
:widths: 10 90

108
docs/configuration/map_filter_syntax.rst
View File

@ -1,108 +0,0 @@
.. include:: ../global.rst.inc
Mapping & Filter Syntax
=======================
For various job types, a filesystem ``mapping`` or ``filter`` needs to be
specified.
Both have in common that they take a filesystem path (in the ZFS filesystem hierarchy)as parameters and return something.
Mappings return a *target filesystem* and filters return a *filter result*.
The pattern syntax is the same for mappings and filters and is documented in the following section.
Common Pattern Syntax
---------------------
A mapping / filter is specified as a **YAML dictionary** with patterns as keys and
results as values.
The following rules determine which result is chosen for a given filesystem path:
* More specific path patterns win over less specific ones
* Non-wildcard patterns (full path patterns) win over *subtree wildcards* (`<` at end of pattern)
The **subtree wildcard** ``<`` means "the dataset left of ``<`` and all its children".
Example
~~~~~~~
::
# Rule number and its pattern
1: tank< # tank and all its children
2: tank/foo/bar # full path pattern (no wildcard)
3: tank/foo< # tank/foo and all its children
# Which rule applies to given path?
tank/foo/bar/loo => 3
tank/bar => 1
tank/foo/bar => 2
zroot => NO MATCH
tank/var/log => 1
.. _pattern-mapping:
Mappings
--------
Mappings map a *source filesystem path* to a *target filesystem path*.
Per pattern, either a target filesystem path or ``"!"`` is specified as a result.
* If no pattern matches, there exists no target filesystem (``NO MATCH``).
* If the result is a ``"!"``, there exists no target filesystem (``NO MATCH``).
* If the pattern is a non-wildcard pattern, the source path is mapped to the target path on the right.
* If the pattern ends with a *subtree wildcard* (``<``), the source path is **prefix-trimmed** with the path specified left of ``<``.
* Note: this means that only for *wildcard-only* patterns (pattern= ``<`` ) is the source path simply appended to the target path.
The example is from the :sampleconf:`localbackup/host1.yml` example config.
::
jobs:
- name: mirror_local
type: local
mapping: {
"zroot/var/db<": "storage/backups/local/zroot/var/db",
"zroot/usr/home<": "storage/backups/local/zroot/usr/home",
"zroot/usr/home/paranoid": "!", #don't backup paranoid user
"zroot/poudriere/ports<": "!", #don't backup the ports trees
}
...
::
zroot/var/db => storage/backups/local/zroot/var/db
zroot/var/db/a/child => storage/backups/local/zroot/var/db/a/child
zroot/usr/home => storage/backups/local/zroot/usr/home
zroot/usr/home/paranoid => NOT MAPPED
zroot/usr/home/bob => storage/backups/local/zroot/usr/home/bob
zroot/usr/src => NOT MAPPED
zroot/poudriere/ports/2017Q3 => NOT MAPPED
zroot/poudriere/ports/HEAD => NOT MAPPED
.. TIP::
You can try out patterns for a configured job using the ``zrepl test`` subcommand.
.. _pattern-filter:
Filters
-------
Valid filter results: ``ok`` or ``!``.
The example below show the source job from the :ref:`tutorial <tutorial-configure-app-srv>`:
The corresponding pull job is allowed access to ``zroot/var/db``, ``zroot/usr/home`` + children except ``zroot/usr/home/paranoid``::
jobs:
- name: pull_backup
type: source
...
filesystems: {
"zroot/var/db": "ok",
"zroot/usr/home<": "ok",
"zroot/usr/home/paranoid": "!",
}
...

3
docs/configuration/monitoring.rst
View File

@ -6,7 +6,6 @@ Monitoring
==========
Monitoring endpoints are configured in the ``global.monitoring`` section of the |mainconfig|.
Check out :sampleconf:`random/logging_and_monitoring.yml` for examples.
.. _monitoring-prometheus:
@ -17,7 +16,7 @@ zrepl can expose `Prometheus metrics <https://prometheus.io/docs/instrumenting/e
The ``listen`` attribute is a `net.Listen <https://golang.org/pkg/net/#Listen>`_ string for tcp, e.g. ``:9091`` or ``127.0.0.1:9091``.
The Prometheues monitoring job appears in the ``zrepl control`` job list and may be specified **at most once**.
There is no stability guarantee on the exported metrics.
At the time of writing, there is no stability guarantee on the exported metrics.
::

2
docs/configuration/preface.rst
View File

@ -15,7 +15,7 @@ zrepl searches for its main configuration file in the following locations (in th
* ``/etc/zrepl/zrepl.yml``
* ``/usr/local/etc/zrepl/zrepl.yml``
The examples in the :ref:`tutorial` or the ``cmd/sampleconf`` directory should provide a good starting point.
The examples in the :ref:`tutorial` or the :sampleconf:`/` directory should provide a good starting point.
-------------------
Runtime Directories

155
docs/configuration/prune.rst
View File

@ -3,46 +3,96 @@
Pruning Policies
================
In zrepl, *pruning* means *destroying filesystem versions by some policy* where filesystem versions are bookmarks and snapshots.
In zrepl, *pruning* means *destroying snapshots*.
Pruning must happen on both sides of a replication or the systems would inevitable run out of disk space at some point.
A *pruning policy* takes a list of filesystem versions and decides for each whether it should be kept or destroyed.
Typically, the requirements to temporal resolution and maximum retention time differ per side.
For example, when using zrepl to back up a busy database server, you will want high temporal resolution (snapshots every 10 min) for the last 24h in case of administrative disasters, but cannot afford to store them for much longer because you might have high turnover volume in the database.
On the receiving side, you may have more disk space available, or need to comply with other backup retention policies.
The job context defines which snapshots are even considered for pruning, for example through the ``snapshot_prefix`` variable.
Check the respective :ref:`job definition <job>` for details.
zrepl uses a set of **keep rules** to determine which snapshots shall be kept per filesystem.
**A snapshot that is not kept by any rule is destroyed.**
The keep rules are **evaluated on the active side** (:ref:`push <job-push>` or :ref:`pull job <job-pull>`) of the replication setup, for both active and passive side, after replication completed or was determined to have failed permanently.
Currently, the :ref:`prune-retention-grid` is the only supported pruning policy.
Example Configuration:
::
jobs:
- type: push
name: ...
connect: ...
filesystems: {
"<": true,
"tmp": false
}
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
pruning:
keep_sender:
- type: not_replicated
# make sure manually created snapshots by the administrator are kept
- type: regex
regex: "^manual_.*"
- type: grid
grid: 1x1h(keep=all) | 24x1h | 14x1d
regex: "^zrepl_.*"
keep_receiver:
- type: grid
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
regex: "^zrepl_.*"
# manually created snapshots will be kept forever on receiver
.. TIP::
You can perform a dry-run of a job's pruning policy using the ``zrepl test`` subcommand.
.. _prune-retention-grid:
.. ATTENTION::
Retention Grid
--------------
It is currently not possible to define pruning on a source job.
The source job creates snapshots, which means that extended replication downtime will fill up the source's zpool with snapshots, since pruning is directed by the corresponding active side (pull job).
If this is a potential risk for you, consider using :ref:`push mode <job-push>`.
.. _prune-keep-not-replicated:
Policy ``not_replicated``
-------------------------
::
jobs:
- type: push
pruning:
keep_sender:
- type: not_replicated
...
``not_replicated`` keeps all snapshots that have not been replicated to the receiving side.
It only makes sense to specify this rule on a sender (source or push job).
The state required to evaluate this rule is stored in the :ref:`replication cursor bookmark <replication-cursor-bookmark>` on the sending side.
.. _prune-keep-retention-grid:
Policy ``grid``
---------------
::
jobs:
- name: pull_app-srv
type: pull
...
prune:
policy: grid
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
│ │
- type: pull
pruning:
keep_receiver:
- type: grid
regex: "^zrepl_.*"
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
│ │
└─ one hour interval
│
└─ 24 adjacent one-hour intervals
- name: pull_backup
type: source
interval: 10m
prune:
policy: grid
grid: 1x1d(keep=all)
keep_bookmarks: 144
│
└─ 24 adjacent one-hour intervals
...
The retention grid can be thought of as a time-based sieve:
The ``grid`` field specifies a list of adjacent time intervals:
@ -51,16 +101,13 @@ All intervals to its right describe time intervals further in the past.
Each interval carries a maximum number of snapshots to keep.
It is secified via ``(keep=N)``, where ``N`` is either ``all`` (all snapshots are kept) or a positive integer.
The default value is **1**.
Bookmarks are not affected by the above.
Instead, the ``keep_bookmarks`` field specifies the number of bookmarks to be kept per filesystem.
You only need to specify ``keep_bookmarks`` at the source-side of a replication setup since the destination side does not receive bookmarks.
You can specify ``all`` as a value to keep all bookmarks, but be warned that you should install some other way to prune unneeded ones then (see below).
The default value is **keep=1**.
The following procedure happens during pruning:
#. The list of snapshots eligible for pruning is sorted by ``creation``
#. The list of snapshots is filtered by the regular expression in ``regex``.
Only snapshots names that match the regex are considered for this rule, all others are not affected.
#. The filtered list of snapshots is sorted by ``creation``
#. The left edge of the first interval is aligned to the ``creation`` date of the youngest snapshot
#. A list of buckets is created, one for each interval
#. The list of snapshots is split up into the buckets.
@ -69,16 +116,42 @@ The following procedure happens during pruning:
#. the contained snapshot list is sorted by creation.
#. snapshots from the list, oldest first, are destroyed until the specified ``keep`` count is reached.
#. all remaining snapshots on the list are kept.
#. The list of bookmarks eligible for pruning is sorted by ``createtxg`` and the most recent ``keep_bookmarks`` bookmarks are kept.
.. _replication-downtime:
.. ATTENTION::
.. _prune-keep-last-n:
Policy ``last_n``
-----------------
::
jobs:
- type: push
pruning:
keep_receiver:
- type: last_n
count: 10
...
``last_n`` keeps the last ``count`` snapshots (last = youngest = most recent creation date).
.. _prune-keep-regex:
Policy ``regex``
----------------
::
jobs:
- type: push
pruning:
keep_receiver:
- type: regex
regex: "^(zrepl|manual)_.*"
...
``regex`` keeps all snapshots whose names are matched by the regular expressionin ``regex``.
Like all other regular expression fields in prune policies, zrepl uses Go's `regexp.Regexp <https://golang.org/pkg/regexp/#Compile>`_ Perl-compatible regular expressions (`Syntax <https://golang.org/pkg/regexp/syntax>`_).
Be aware that ``keep_bookmarks x interval`` (interval of the job level) controls the **maximum allowable replication downtime** between source and destination.
If replication does not work for whatever reason, source and destination will eventually run out of sync because the source will continue pruning snapshots.
The only recovery in that case is full replication, which may not always be viable due to disk space or traffic constraints.
Further note that while bookmarks consume a constant amount of disk space, listing them requires temporary dynamic **kernel memory** proportional to the number of bookmarks.
Thus, do not use ``all`` or an inappropriately high value without good reason.

197
docs/configuration/transports.rst
View File

@ -5,42 +5,156 @@
Transports
==========
A transport provides an authenticated `io.ReadWriteCloser <https://golang.org/pkg/io/#ReadWriteCloser>`_ to the RPC layer.
(An ``io.ReadWriteCloser`` is essentially a bidirectional reliable communication channel.)
The zrepl RPC layer uses **transports** to establish a single, bidirectional data stream between an active and passive job.
On the passive (serving) side, the transport also provides the **client identity** to the upper layers:
this string is used for access control and separation of filesystem sub-trees in :ref:`sink jobs <job-sink>`.
Transports are specified in the ``connect`` or ``serve`` section of a job definition.
Currently, only the ``ssh+stdinserver`` transport is supported.
.. ATTENTION::
The **client identities must be valid ZFS dataset path components**
because the :ref:`sink job <job-sink>` uses ``${root_fs}/${client_identity}`` to determine the client's subtree.
.. _transport-tcp:
``tcp`` Transport
-----------------
The ``tcp`` transport uses plain TCP, which means that the data is **not encrypted** on the wire.
Clients are identified by their IPv4 or IPv6 addresses, and the client identity is established through a mapping on the server.
This transport may also be used in conjunction with network-layer encryption and/or VPN tunnels to provide encryption on the wire.
To make the IP-based client authentication effective, such solutions should provide authenticated IP addresses.
Some options to consider:
* `WireGuard <https://www.wireguard.com/>`_: Linux-focussed, in-kernel TLS
* `OpenVPN <https://openvpn.net/>`_: Cross-platform VPN, uses tun on \*nix
* `IPSec <https://en.wikipedia.org/wiki/IPsec>`_: Properly standardized, in-kernel network-layer VPN
* `spiped <http://www.tarsnap.com/spiped.html>`_: think of it as an encrypted pipe between two servers
* SSH
* `sshuttle <https://sshuttle.readthedocs.io/en/stable/overview.html>`_: VPN-like solution, but using SSH
* `SSH port forwarding <https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding>`_: Systemd user unit & make it start before the zrepl service.
Serve
~~~~~
::
jobs:
- type: sink
serve:
type: tcp
listen: ":8888"
clients: {
"192.168.122.123" : "mysql01"
"192.168.122.123" : "mx01"
}
...
Connect
~~~~~~~
::
jobs:
- type: push
connect:
type: tcp
address: "10.23.42.23:8888"
dial_timeout: # optional, default 10s
...
.. _transport-tcp+tlsclientauth:
``tls`` Transport
-----------------
The ``tls`` transport uses TCP + TLS with client authentication using client certificates.
The client identity is the common name (CN) presented in the client certificate.
It is recommended to set up a dedicated CA infrastructure for this transport, e.g. using OpenVPN's `EasyRSA <https://github.com/OpenVPN/easy-rsa>`_.
The implementation uses `Go's TLS library <https://golang.org/pkg/crypto/tls/>`_.
Since Go binaries are statically linked, you or your distribution need to recompile zrepl when vulnerabilities in that library are disclosed.
All file paths are resolved relative to the zrepl daemon's working directory.
Specify absolute paths if you are unsure what directory that is (or find out from your init system).
Serve
~~~~~
::
jobs:
- type: sink
root_fs: "pool2/backup_laptops"
serve:
type: tls
listen: ":8888"
ca: /etc/zrepl/ca.crt
cert: /etc/zrepl/prod.crt
key: /etc/zrepl/prod.key
client_cns:
- "laptop1"
- "homeserver"
The ``ca`` field specified the certificate authority used to validate client certificates.
The ``client_cns`` list specifies a list of accepted client common names (which are also the client identities for this transport).
Connect
~~~~~~~
::
jobs:
- type: pull
connect:
type: tls
address: "server1.foo.bar:8888"
ca: /etc/zrepl/ca.crt
cert: /etc/zrepl/backupserver.crt
key: /etc/zrepl/backupserver.key
server_cn: "server1"
dial_timeout: # optional, default 10s
The ``ca`` field specifies the CA which signed the server's certificate (``serve.cert``).
The ``server_cn`` specifies the expected common name (CN) of the server's certificate.
It overrides the hostname specified in ``address``.
The connection fails if either do not match.
.. _transport-ssh+stdinserver:
``ssh+stdinserver`` Transport
-----------------------------
The way the ``ssh+stdinserver`` transport works is inspired by `git shell <https://git-scm.com/docs/git-shell>`_ and `Borg Backup <https://borgbackup.readthedocs.io/en/stable/deployment.html>`_.
``ssh+stdinserver`` is inspired by `git shell <https://git-scm.com/docs/git-shell>`_ and `Borg Backup <https://borgbackup.readthedocs.io/en/stable/deployment.html>`_.
It is provided by the Go package ``github.com/problame/go-netssh``.
The config excerpts are taken from the :ref:`tutorial` which you should complete before reading further.
.. _transport-ssh+stdinserver-serve:
Serve Mode
~~~~~~~~~~
Serve
~~~~~
::
jobs:
- name: pull_backup
type: source
- type: source
serve:
type: stdinserver
client_identity: backup-srv.example.com
client_identities:
- "client1"
- "client2"
...
The serving job opens a UNIX socket named after ``client_identity`` in the runtime directory, e.g. ``/var/run/zrepl/stdinserver/backup-srv.example.com``.
First of all, note that ``type=stdinserver`` in this case:
Currently, only ``connect.type=ssh+stdinserver`` can connect to a ``serve.type=stdinserver``, but we want to keep that option open for future extensions.
On the same machine, the ``zrepl stdinserver $client_identity`` command connects to that socket.
For example, ``zrepl stdinserver backup-srv.example.com`` connects to the UNIX socket ``/var/run/zrepl/stdinserver/backup-srv.example.com``.
The serving job opens a UNIX socket named after ``client_identity`` in the runtime directory.
In our example above, that is ``/var/run/zrepl/stdinserver/client1`` and ``/var/run/zrepl/stdinserver/client2``.
On the same machine, the ``zrepl stdinserver $client_identity`` command connects to ``/var/run/zrepl/stdinserver/$client_identity``.
It then passes its stdin and stdout file descriptors to the zrepl daemon via *cmsg(3)*.
zrepl daemon in turn combines them into an ``io.ReadWriteCloser``:
zrepl daemon in turn combines them into an object implementing ``net.Conn``:
a ``Write()`` turns into a write to stdout, a ``Read()`` turns into a read from stdin.
Interactive use of the ``stdinserver`` subcommand does not make much sense.
@ -54,8 +168,8 @@ This can be achieved with an entry in the ``authorized_keys`` file of the servin
# for older OpenSSH versions
command="zrepl stdinserver CLIENT_IDENTITY",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc CLIENT_SSH_KEY
* CLIENT_IDENTITY is substituted with ``backup-srv.example.com`` in our example
* CLIENT_SSH_KEY is substituted with the public part of the SSH keypair specified in the ``connect`` directive on the connecting host.
* CLIENT_IDENTITY is substituted with an entry from ``client_identities`` in our example
* CLIENT_SSH_KEY is substituted with the public part of the SSH keypair specified in the ``connect.identity_file`` directive on the connecting host.
.. NOTE::
@ -64,24 +178,24 @@ This can be achieved with an entry in the ``authorized_keys`` file of the servin
To recap, this is of how client authentication works with the ``ssh+stdinserver`` transport:
* Connections to the ``client_identity`` UNIX socket are blindly trusted by zrepl daemon.
* Thus, the runtime directory must be private to the zrepl user (checked by zrepl daemon)
* Connections to the ``/var/run/zrepl/stdinserver/${client_identity}`` UNIX socket are blindly trusted by zrepl daemon.
The connection client identity is the name of the socket, i.e. ``${client_identity}``.
* Thus, the runtime directory must be private to the zrepl user (this is checked by zrepl daemon)
* The admin of the host with the serving zrepl daemon controls the ``authorized_keys`` file.
* Thus, the administrator controls the mapping ``PUBKEY -> CLIENT_IDENTITY``.
.. _transport-ssh+stdinserver-connect:
Connect Mode
~~~~~~~~~~~~
Connect
~~~~~~~
::
jobs:
- name: pull_app-srv
type: pull
- type: pull
connect:
type: ssh+stdinserver
host: app-srv.example.com
host: prod.example.com
user: root
port: 22
identity_file: /etc/zrepl/ssh/identity
@ -102,15 +216,46 @@ The connecting zrepl daemon
#. The remote user, host and port correspond to those configured.
#. Further options can be specified using the ``options`` field, which appends each entry in the list to the command line using ``-o $entry``.
#. Wraps the pipe ends in an ``io.ReadWriteCloser`` and uses it for RPC.
#. Wraps the pipe ends in a ``net.Conn`` and returns it to the RPC layer.
As discussed in the section above, the connecting zrepl daemon expects that ``zrepl stdinserver $client_identity`` is executed automatically via an ``authorized_keys`` file entry.
The ``known_hosts`` file used by the ssh command must contain an entry for the serving host, e.g., ``app-srv.example.com`` in the example above.
The ``known_hosts`` file used by the ssh command must contain an entry for ``connect.host`` prior to starting zrepl.
Thus, run the following on the pulling host's command line (substituting ``connect.host``):
::
ssh -i /etc/zrepl/ssh/identity root@prod.example.com
.. NOTE::
The environment variables of the underlying SSH process are cleared. ``$SSH_AUTH_SOCK`` will not be available.
It is suggested to create a separate, unencrypted SSH key solely for that purpose.
.. _transport-local:
``local`` Transport
-------------------
The local transport can be used to implement :ref:`local replication <replication-local>`, i.e., push replication between a push and sink job defined in the same configuration file.
The ``listener_name`` is analogous to a hostname and must match between ``serve`` and ``connect``.
The ``client_identity`` is used by the sink as documented above.
::
jobs:
- type: sink
serve:
type: local
listener_name: localsink
...
- type: push
connect:
type: local
listener_name: localsink
client_identity: local_backup
...

25
docs/implementation.rst
View File

@ -5,9 +5,12 @@ Implementation Overview
.. WARNING::
Incomplete / under construction
Incomplete and possibly outdated.
Check out the :ref:`talks about zrepl <pr-talks>` at various conferences for up-to-date material.
Alternatively, have a `look at the source code <http://github.com/zrepl/zrepl>`_ ;)
The following design aspects may convince you that ``zrepl`` is superior to a hacked-together shell script solution.
Also check out the :ref:`talks about zrepl <pr-talks>` at various conferences.
Testability & Performance
-------------------------
@ -28,7 +31,7 @@ While it is tempting to just issue a few ``ssh remote 'zfs send ...' | zfs recv`
* The snapshot streams need to be compatible.
* Communication is still unidirectional. Thus, you will most likely
* either not take advantage of features such as *compressed send & recv*
* either not take advantage of advanced replication features such as *compressed send & recv*
* or issue additional ``ssh`` commands in advance to figure out what features are supported on the other side.
* Advanced logic in shell scripts is ugly to read, poorly testable and a pain to maintain.
@ -36,25 +39,21 @@ While it is tempting to just issue a few ``ssh remote 'zfs send ...' | zfs recv`
zrepl takes a different approach:
* Define an RPC protocol.
* Establish an encrypted, authenticated, bidirectional communication channel...
* ... with zrepl running at both ends of it.
* Establish an encrypted, authenticated, bidirectional communication channel.
* Run daemons on both sides of the setup and let them talk to each other.
This has several obvious benefits:
* No blank root shell access is given to the other side.
* Instead, an *authenticated* peer can *request* filesystem lists, snapshot streams, etc.
* Requests are then checked against job-specific ACLs, limiting a client to the filesystems it is actually allowed to replicate.
* The :ref:`transport mechanism <transport>` is decoupled from the remaining logic, keeping it extensible.
* An *authenticated* peer *requests* filesystem lists, snapshot streams, etc.
* The server decides which filesystems it exposes to which peers.
* The :ref:`transport mechanism <transport>` is decoupled from the remaining logic, which allows us to painlessly offer multiple transport mechanisms.
Protocol Implementation
~~~~~~~~~~~~~~~~~~~~~~~
zrepl implements its own RPC protocol.
This is mostly due to the fact that existing solutions do not provide efficient means to transport large amounts of data.
Package `github.com/zrepl/zrepl/rpc <https://github.com/zrepl/zrepl/tree/master/rpc>`_ builds a special-case handling around returning an ``io.Reader`` as part of a unary RPC call.
Measurements show only a single memory-to-memory copy of a snapshot stream is made using ``github.com/zrepl/zrepl/rpc``, and there is still potential for further optimizations.
zrepl uses a custom RPC protol because, at the time of writing, existing solutions like gRPC do not provide efficient means to transport large amounts of data, whose size is unknown at send time (= zfs send streams).
The package used is `github.com/problame/go-streamrpc <https://github.com/problame/go-streamrpc/tree/master>`_.
Logging & Transparency
----------------------

59
docs/index.rst
View File

@ -8,10 +8,15 @@
zrepl - ZFS replication
-----------------------
.. ATTENTION::
zrepl as well as this documentation is still under active development.
It is neither feature complete nor is there a stability guarantee on the configuration format.
Use & test at your own risk ;)
**zrepl** is a one-stop, integrated solution for ZFS replication.
.. raw:: html
<div style="margin-bottom: 1em; background: #2e3436; min-height: 6em; max-width: 100%">
<a href="https://raw.githubusercontent.com/wiki/zrepl/zrepl/zrepl_0.1_status.mp4" target="_new" >
<video title="zrepl status subcommand" loop autoplay style="width: 100%; display: block;" src="https://raw.githubusercontent.com/wiki/zrepl/zrepl/zrepl_0.1_status.mp4"></video>
</a>
</div>
Getting started
~~~~~~~~~~~~~~~
@ -21,42 +26,47 @@ The :ref:`10 minutes tutorial setup <tutorial>` gives you a first impression.
Main Features
~~~~~~~~~~~~~
* Filesystem Replication
* **Filesystem replication**
* [x] Local & Remote
* [x] Pull mode
* [ ] Push mode
* [x] Access control checks when pulling datasets
* [x] :ref:`Flexible mapping <pattern-mapping>` rules
* [x] Bookmarks support
* [ ] Feature-negotiation for
* [x] Pull & Push mode
* [x] Multiple transport :ref:`transports <transport>`: TCP, TCP + TLS client auth, SSH
* Resumable `send & receive`
* Compressed `send & receive`
* Raw encrypted `send & receive` (as soon as it is available)
* Advanced replication features
* Automatic snapshot creation
* [ ] Resumable send & receive
* [ ] Compressed send & receive
* [ ] Raw encrypted send & receive
* [x] Ensure fixed time interval between snapshots
* **Automatic snapshot management**
* Automatic snapshot :ref:`pruning <prune>`
* [x] Periodic filesystem snapshots
* [x] Flexible :ref:`pruning rule system <prune>`
* [x] Age-based fading (grandfathering scheme)
* [x] Age-based fading (grandfathering scheme)
* [x] Bookmarks to avoid divergence between sender and receiver
* Logging \& Monitoring
* **Sophisticated Monitoring & Logging**
* Detailed & structured :ref:`logging <logging>`
* [x] Live progress reporting via `zrepl status` :ref:`subcommand <usage>`
* [x] Comprehensive, structured :ref:`logging <logging>`
* ``human``, ``logfmt`` and ``json`` formatting
* stdout, syslog and TCP (+TLS client auth) outlets
* Prometheus :ref:`monitoring <monitoring>` endpoint
* [x] Prometheus :ref:`monitoring <monitoring>` endpoint
* Maintainable implementation in Go
* **Maintainable implementation in Go**
* [x] Cross platform
* [x] Type safe & testable code
.. ATTENTION::
zrepl as well as this documentation is still under active development.
There is no stability guarantee on the RPC protocol or configuration format,
but we do our best to document breaking changes in the :ref:`changelog`.
Contributing
~~~~~~~~~~~~
@ -88,6 +98,7 @@ Table of Contents
configuration
usage
implementation
pr
changelog
GitHub Repository & Issue Tracker <https://github.com/zrepl/zrepl>
pr

15
docs/pr.rst
View File

@ -1,8 +1,17 @@
.. _pr-talks:
Talks & Presentations
=====================
* Talk at EuroBSDCon2017 FreeBSD DevSummit (
`Slides <https://docs.google.com/presentation/d/1EmmeEvOXAWJHCVnOS9-TTsxswbcGKmeLWdY_6BH4w0Q/edit?usp=sharing>`_,
`Event <https://wiki.freebsd.org/DevSummit/201709>`_
* Talk at OpenZFS Developer Summit 2018 of pre-release 0.1 (
`25min Recording <https://www.youtube.com/watch?v=U4TUPQzZzPk&index=4&list=PLaUVvul17xSe0pC6sCirlZXYqICP09Y8z&t=0s>`__ ,
`Slides <https://docs.google.com/presentation/d/1HXJ_9Q0kiHffeoQ7PlFn1qAfUcWyzedRgFlvL3nUlMU/edit?usp=sharing>`__ ,
`Event <http://www.open-zfs.org/wiki/OpenZFS_Developer_Summit_2018>`__
)
* Talk at EuroBSDCon2017 FreeBSD DevSummit with live demo of zrepl 0.0.3 (
`55min Recording <https://www.youtube.com/watch?v=c1LKeyP1mos&t=2316s>`__,
`Slides <https://docs.google.com/presentation/d/1EmmeEvOXAWJHCVnOS9-TTsxswbcGKmeLWdY_6BH4w0Q/edit?usp=sharing>`__,
`Event <https://wiki.freebsd.org/DevSummit/201709>`__
)

13
docs/sphinxconf/conf.py
View File

@ -101,6 +101,17 @@ html_static_path = ['../_static']
html_logo = '../_static/zrepl.svg'
html_context = {
# https://github.com/rtfd/sphinx_rtd_theme/issues/205
# Add 'Edit on Github' link instead of 'View page source'
"display_github": True,
"github_user": "zrepl",
"github_repo": "zrepl",
"github_version": "master",
"conf_py_path": "/docs/",
"source_suffix": source_suffix,
}
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
@ -162,7 +173,7 @@ texinfo_documents = [
# http://www.sphinx-doc.org/en/stable/ext/extlinks.html
extlinks = {
'issue':('https://github.com/zrepl/zrepl/issues/%s', 'issue #'),
'sampleconf':('https://github.com/zrepl/zrepl/blob/master/cmd/sampleconf/%s', 'cmd/sampleconf/'),
'sampleconf':('https://github.com/zrepl/zrepl/blob/master/config/samples%s', 'config/samples'),
'commit':('https://github.com/zrepl/zrepl/commit/%s', 'commit '),
}

160
docs/tutorial.rst
View File

@ -9,164 +9,120 @@ Tutorial
This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
We assume the following scenario:
* Production server ``app-srv`` with filesystems to back up:
* Production server ``prod`` with filesystems to back up:
* ``zroot/var/db``
* ``zroot/usr/home`` and all its child filesystems
* **except** ``zroot/usr/home/paranoid`` belonging to a user doing backups themselves
* Backup server ``backup-srv`` with
* Backup server ``backups`` with
* Filesystem ``storage/zrepl/pull/app-srv`` + children dedicated to backups of ``app-srv``
* Filesystem ``storage/zrepl/pull/prod`` + children dedicated to backups of ``prod``
Our backup solution should fulfill the following requirements:
* Periodically snapshot the filesystems on ``app-srv`` *every 10 minutes*
* Incrementally replicate these snapshots to ``storage/zrepl/pull/app-srv/*`` on ``backup-srv``
* Keep only very few snapshots on ``app-srv`` to save disk space
* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on ``backup-srv``
* Periodically snapshot the filesystems on ``prod`` *every 10 minutes*
* Incrementally replicate these snapshots to ``storage/zrepl/pull/prod/*`` on ``backups``
* Keep only very few snapshots on ``prod`` to save disk space
* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on ``backups``
Analysis
--------
We can model this situation as two jobs:
* A **source job** on ``app-srv``
* A **source job** on ``prod``
* Creates the snapshots
* Keeps a short history of snapshots to enable incremental replication to ``backup-srv``
* Accepts connections from ``backup-srv``
* Keeps a short history of snapshots to enable incremental replication to ``backups``
* Accepts connections from ``backups``
* A **pull job** on ``backup-srv``
* A **pull job** on ``backups``
* Connects to the ``zrepl daemon`` process on ``app-srv``
* Pulls the snapshots to ``storage/zrepl/pull/app-srv/*``
* Fades out snapshots in ``storage/zrepl/pull/app-srv/*`` as they age
* Connects to the ``zrepl daemon`` process on ``prod``
* Pulls the snapshots to ``storage/zrepl/pull/prod/*``
* Fades out snapshots in ``storage/zrepl/pull/prod/*`` as they age
Why doesn't the **pull job** create the snapshots before pulling?
As is the case with all distributed systems, the link between ``app-srv`` and ``backup-srv`` might be down for an hour or two.
As is the case with all distributed systems, the link between ``prod`` and ``backups`` might be down for an hour or two.
We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
When the link comes up again, ``backup-srv`` will happily catch up the 12 snapshots taken by ``app-srv`` in the meantime, without
a gap in our backup history.
When the link comes up again, ``backups`` will catch up with the snapshots taken by ``prod`` in the meantime, without a gap in our backup history.
Install zrepl
-------------
Follow the :ref:`OS-specific installation instructions <installation>` and come back here.
Configure ``backup-srv``
------------------------
Configure server ``backups``
----------------------------
We define a **pull job** named ``pull_app-srv`` in the |mainconfig| on host ``backup-srv``: ::
We define a **pull job** named ``pull_prod`` in ``/etc/zrepl/zrepl.yml`` or ``/usr/local/etc/zrepl/zrepl.yml`` on host ``backups`` : ::
jobs:
- name: pull_app-srv
- name: pull_prod
type: pull
connect:
type: ssh+stdinserver
host: app-srv.example.com
user: root
port: 22
identity_file: /etc/zrepl/ssh/identity
type: tcp
address: "192.168.2.20:2342"
root_fs: "storage/zrepl/pull/prod"
interval: 10m
mapping: {
"<":"storage/zrepl/pull/app-srv"
}
snapshot_prefix: zrepl_pull_backup_
prune:
policy: grid
grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
pruning:
keep_sender:
- type: not_replicated
- type: last_n
count: 10
keep_receiver:
- type: grid
grid: 1x1h(keep=all) | 24x1h | 30x1d | 6x30d
regex: "^zrepl_"
interval: 10m
The ``connect`` section instructs the zrepl daemon to use the ``stdinserver`` transport:
``backup-srv`` will connect to the specified SSH server and expect ``zrepl stdinserver CLIENT_IDENTITY`` instead of the shell on the other side.
The ``connect`` section instructs the zrepl daemon to use plain TCP transport.
Check out the :ref:`transports <transport>` section for alternatives that support encryption.
It uses the private key specified at ``connect.identity_file`` which we still need to create: ::
.. _tutorial-configure-prod:
cd /etc/zrepl
mkdir -p ssh
chmod 0700 ssh
ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/identity
Configure server ``prod``
-------------------------
Note that most use cases do not benefit from separate keypairs per remote endpoint.
Thus, it is sufficient to create one keypair and use it for all ``connect`` directives on one host.
zrepl uses ssh's default ``known_hosts`` file, which must contain a host identification entry for ``app-srv.example.com``.
If that entry does not already exist, we need to generate it.
Run the following command, compare the host fingerprints, and confirm with yes if they match.
You will not be able to get a shell with the identity file we just generated, which is fine. ::
ssh -i /etc/zrepl/ssh/identity root@app-srv.example.com
Learn more about :ref:`transport-ssh+stdinserver` transport and the :ref:`pull job <job-pull>` format.
.. _tutorial-configure-app-srv:
Configure ``app-srv``
---------------------
We define a corresponding **source job** named ``pull_backup`` in the |mainconfig| on host ``app-srv``: ::
We define a corresponding **source job** named ``source_backups`` in ``/etc/zrepl/zrepl.yml`` or ``/usr/local/etc/zrepl/zrepl.yml`` on host ``prod`` : ::
jobs:
- name: pull_backup
- name: source_backups
type: source
serve:
type: stdinserver
client_identity: backup-srv.example.com
type: tcp
listen: ":2342"
clients: {
"192.168.2.10" : "backups"
}
filesystems: {
"zroot/var/db": "ok",
"zroot/usr/home<": "ok",
"zroot/usr/home/paranoid": "!",
"zroot/var/db:": true,
"zroot/usr/home<": true,
"zroot/usr/home/paranoid": false
}
snapshot_prefix: zrepl_pull_backup_
interval: 10m
prune:
policy: grid
grid: 1x1d(keep=all)
keep_bookmarks: 144
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
The ``serve`` section corresponds to the ``connect`` section in the configuration of ``backup-srv``.
We now want to authenticate ``backup-srv`` before allowing it to pull data.
This is done by limiting SSH connections from ``backup-srv`` to execute the ``stdinserver`` subcommand.
Open ``/root/.ssh/authorized_keys`` and add either of the the following lines.::
# for OpenSSH >= 7.2
command="zrepl stdinserver backup-srv.example.com",restrict CLIENT_SSH_KEY
# for older OpenSSH versions
command="zrepl stdinserver backup-srv.example.com",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc CLIENT_SSH_KEY
.. ATTENTION::
Replace CLIENT_SSH_KEY with the contents of ``/etc/zrepl/ssh/identity.pub`` from ``app-srv``.
Mind the trailing ``.pub`` in the filename.
The entries **must** be on a single line, including the replaced CLIENT_SSH_KEY.
.. HINT::
You may need to adjust the ``PermitRootLogin`` option in ``/etc/ssh/sshd_config`` to ``forced-commands-only`` or higher for this to work.
Refer to sshd_config(5) for details.
The argument ``backup-srv.example.com`` is the client identity of ``backup-srv`` as defined in ``jobs.serve.client_identity``.
Again, both :ref:`transport-ssh+stdinserver` transport and the :ref:`job-source` format are documented.
The ``serve`` section whitelists ``backups``'s IP address ``192.168.2.10`` and assigns it the client identity ``backups`` which will show up in the logs.
Again, check the :ref:`docs for encrypted transports <transport>`.
Apply Configuration Changes
---------------------------
We need to restart the zrepl daemon on **both** ``app-srv`` and ``backup-srv``.
We need to restart the zrepl daemon on **both** ``prod`` and ``backups``.
This is :ref:`OS-specific <usage-zrepl-daemon-restarting>`.
Watch it Work
-------------
Run ``zrepl control status`` to view the current activity of the configured jobs.
If a job encountered problems since it last left idle state, the output contains useful debug log.
Run ``zrepl status`` on ``prod`` to monitor the replication and pruning activity.
Additionally, you can check the detailed structured logs of the `zrepl daemon` process and use GNU *watch* to view the snapshots present on both machines.
@ -175,7 +131,7 @@ If you like tmux, here is a handy script that works on FreeBSD: ::
pkg install gnu-watch tmux
tmux new-window
tmux split-window "tail -f /var/log/zrepl.log"
tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_'"
tmux select-layout tiled
The Linux equivalent might look like this: ::
@ -183,7 +139,7 @@ The Linux equivalent might look like this: ::
# make sure tmux is installed & let's assume you use systemd + journald
tmux new-window
tmux split-window "journalctl -f -u zrepl.service"
tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_'"
tmux select-layout tiled
Summary

18
docs/usage.rst
View File

@ -10,8 +10,8 @@ CLI Overview
.. NOTE::
To avoid duplication, the zrepl binary is self-documenting:
invoke any subcommand at any level with the ``--help`` flag to get information on the subcommand, available flags, etc.
The zrepl binary is self-documenting:
run ``zrepl help`` for an overview of the available subcommands or ``zrepl SUBCOMMAND --help`` for information on available flags, etc.
.. list-table::
:widths: 30 70
@ -19,16 +19,18 @@ CLI Overview
* - Subcommand
- Description
* - ``zrepl help``
- show subcommand overview
* - ``zrepl daemon``
- run the daemon, required for all zrepl functionality
* - ``zrepl control``
- control / query the daemon
* - ``zrepl control status``
- show job activity / monitoring (``--format raw``)
* - ``zrepl test``
- test configuration, try pattern syntax, dry run pruning policy, etc.
* - ``zrepl status``
- show job activity, or with ``--raw`` for JSON output
* - ``zrepl stdinserver``
- see :ref:`transport-ssh+stdinserver`
* - ``zrepl wakeup JOB``
- manually trigger replication + pruning
* - ``zrepl configcheck``
- check if config can be parsed without errors
.. _usage-zrepl-daemon: