extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-24 17:35:01 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
problame/wip/tiered-snapshot-management
zrepl/docs/index.rst
Christian Schwarz 58c08c855f new features: {resumable,encrypted,hold-protected} send-recv, last-received-hold 
- **Resumable Send & Recv Support**
  No knobs required, automatically used where supported.
- **Hold-Protected Send & Recv**
  Automatic ZFS holds to ensure that we can always resume a replication step.
- **Encrypted Send & Recv Support** for OpenZFS native encryption.
  Configurable at the job level, i.e., for all filesystems a job is responsible for.
- **Receive-side hold on last received dataset**
  The counterpart to the replication cursor bookmark on the send-side.
  Ensures that incremental replication will always be possible between a sender and receiver.

Design Doc
----------

`replication/design.md` doc describes how we use ZFS holds and bookmarks to ensure that a single replication step is always resumable.

The replication algorithm described in the design doc introduces the notion of job IDs (please read the details on this design doc).
We reuse the job names for job IDs and use `JobID` type to ensure that a job name can be embedded into hold tags, bookmark names, etc.
This might BREAK CONFIG on upgrade.

Protocol Version Bump
---------------------

This commit makes backwards-incompatible changes to the replication/pdu protobufs.
Thus, bump the version number used in the protocol handshake.

Replication Cursor Format Change
--------------------------------

The new replication cursor bookmark format is: `#zrepl_CURSOR_G_${this.GUID}_J_${jobid}`
Including the GUID enables transaction-safe moving-forward of the cursor.
Including the job id enables that multiple sending jobs can send the same filesystem without interfering.
The `zrepl migrate replication-cursor:v1-v2` subcommand can be used to safely destroy old-format cursors once zrepl has created new-format cursors.

Changes in This Commit
----------------------

- package zfs
  - infrastructure for holds
  - infrastructure for resume token decoding
  - implement a variant of OpenZFS's `entity_namecheck` and use it for validation in new code
  - ZFSSendArgs to specify a ZFS send operation
    - validation code protects against malicious resume tokens by checking that the token encodes the same send parameters that the send-side would use if no resume token were available (i.e. same filesystem, `fromguid`, `toguid`)
  - RecvOptions support for `recv -s` flag
  - convert a bunch of ZFS operations to be idempotent
    - achieved through more differentiated error message scraping / additional pre-/post-checks

- package replication/pdu
  - add field for encryption to send request messages
  - add fields for resume handling to send & recv request messages
  - receive requests now contain `FilesystemVersion To` in addition to the filesystem into which the stream should be `recv`d into
    - can use `zfs recv $root_fs/$client_id/path/to/dataset@${To.Name}`, which enables additional validation after recv (i.e. whether `To.Guid` matched what we received in the stream)
    - used to set `last-received-hold`
- package replication/logic
  - introduce `PlannerPolicy` struct, currently only used to configure whether encrypted sends should be requested from the sender
  - integrate encryption and resume token support into `Step` struct

- package endpoint
  - move the concepts that endpoint builds on top of ZFS to a single file `endpoint/endpoint_zfs.go`
    - step-holds + step-bookmarks
    - last-received-hold
    - new replication cursor + old replication cursor compat code
  - adjust `endpoint/endpoint.go` handlers for
    - encryption
    - resumability
    - new replication cursor
    - last-received-hold

- client subcommand `zrepl holds list`: list all holds and hold-like bookmarks that zrepl thinks belong to it
- client subcommand `zrepl migrate replication-cursor:v1-v2`
2020-02-14 22:00:13 +01:00

137 lines
4.6 KiB
ReStructuredText
Raw Permalink Blame History

.. zrepl documentation master file, created by
sphinx-quickstart on Wed Nov 8 22:28:10 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
.. include:: global.rst.inc
|GitHub license| |Language: Go| |Twitter| |Donate via Patreon| |Donate via Liberapay| |Donate via PayPal|
zrepl - ZFS replication
-----------------------
**zrepl** is a one-stop, integrated solution for ZFS replication.
.. raw:: html
<div style="margin-bottom: 1em; background: #2e3436; color: white; font-size: 0.8em; min-height: 6em; max-width: 100%; overflow: auto;">
<pre>
Job: prod_to_backups
Type: push
Replication:
Attempt #1
Status: fan-out-filesystems
Progress: [=========================\----] 246.7 MiB / 264.7 MiB @ 11.5 MiB/s
zroot STEPPING (step 1/2, 624 B/1.2 KiB) next: @a => @b
zroot/ROOT DONE (step 2/2, 1.2 KiB/1.2 KiB)
zroot/ROOT/default STEPPING (step 1/2, 123.4 MiB/129.3 MiB) next: @a => @b
zroot/tmp STEPPING (step 1/2, 29.9 KiB/44.2 KiB) next: @a => @b
zroot/usr STEPPING (step 1/2, 624 B/1.2 KiB) next: @a => @b
zroot/usr/home STEPPING (step 1/2, 123.3 MiB/135.3 MiB) next: @a => @b
zroot/var STEPPING (step 1/2, 624 B/1.2 KiB) next: @a => @b
zroot/var/audit DONE (step 2/2, 1.2 KiB/1.2 KiB)
zroot/var/crash DONE (step 2/2, 1.2 KiB/1.2 KiB)
zroot/var/log STEPPING (step 1/2, 22.0 KiB/29.2 KiB) next: @a => @b
zroot/var/mail STEPPING (step 1/2, 624 B/1.2 KiB) next: @a => @b
Pruning Sender:
...
Pruning Receiver:
</pre>
</div>
Getting started
~~~~~~~~~~~~~~~
The :ref:`10 minutes tutorial setup <tutorial>` gives you a first impression.
Main Features
~~~~~~~~~~~~~
* **Filesystem replication**
* [x] Pull & Push mode
* [x] Multiple :ref:`transport modes <transport>`: TCP, TCP + TLS client auth, SSH
* Advanced replication features
* [x] Automatic retries for temporary network errors
* [x] Automatic resumable send & receive
* [x] Automatic ZFS holds during send & receive
* [x] Automatic bookmark \& hold management for guaranteed incremental send & recv
* [x] Encrypted raw send & receive to untrusted receivers (OpenZFS native encryption)
* [ ] Compressed send & receive
* **Automatic snapshot management**
* [x] Periodic :ref:`filesystem snapshots <job-snapshotting-spec>`
* [x] Support for :ref:`pre- and post-snapshot hooks <job-snapshotting-hooks>` with builtins for MySQL & Postgres
* [x] Flexible :ref:`pruning rule system <prune>`
* [x] Age-based fading (grandfathering scheme)
* [x] Bookmarks to avoid divergence between sender and receiver
* **Sophisticated Monitoring & Logging**
* [x] Live progress reporting via `zrepl status` :ref:`subcommand <usage>`
* [x] Comprehensive, structured :ref:`logging <logging>`
* ``human``, ``logfmt`` and ``json`` formatting
* stdout, syslog and TCP (+TLS client auth) outlets
* [x] Prometheus :ref:`monitoring <monitoring>` endpoint
* **Maintainable implementation in Go**
* [x] Cross platform
* [x] Dynamic feature checking
* [x] Type safe & testable code
.. ATTENTION::
zrepl as well as this documentation is still under active development.
There is no stability guarantee on the RPC protocol or configuration format,
but we do our best to document breaking changes in the :ref:`changelog`.
Contributing
~~~~~~~~~~~~
We are happy about any help we can get!
* :ref:`Financial Support <supporters>`
* Explore the codebase
* These docs live in the ``docs/`` subdirectory
* Document any non-obvious / confusing / plain broken behavior you encounter when setting up zrepl for the first time
* Check the *Issues* and *Projects* sections for things to do.
The `good first issues <https://github.com/zrepl/zrepl/labels/good%20first%20issue>`_ and `docs <https://github.com/zrepl/zrepl/labels/docs>`_ are suitable starting points.
.. admonition:: Development Workflow
:class: note
The `GitHub repository <https://github.com/zrepl/zrepl>`_ is where all development happens.
Make sure to read the `Developer Documentation section <https://github.com/zrepl/zrepl>`_ and open new issues or pull requests there.
Table of Contents
~~~~~~~~~~~~~~~~~
.. toctree::
:maxdepth: 2
:caption: Contents:
tutorial
installation
configuration
usage
pr
changelog
GitHub Repository & Issue Tracker <https://github.com/zrepl/zrepl>
supporters
Reference in New Issue View Git Blame Copy Permalink