Commit Graph

74 Commits

Author SHA1 Message Date
58c08c855f new features: {resumable,encrypted,hold-protected} send-recv, last-received-hold
- **Resumable Send & Recv Support**
  No knobs required, automatically used where supported.
- **Hold-Protected Send & Recv**
  Automatic ZFS holds to ensure that we can always resume a replication step.
- **Encrypted Send & Recv Support** for OpenZFS native encryption.
  Configurable at the job level, i.e., for all filesystems a job is responsible for.
- **Receive-side hold on last received dataset**
  The counterpart to the replication cursor bookmark on the send-side.
  Ensures that incremental replication will always be possible between a sender and receiver.

Design Doc
----------

`replication/design.md` doc describes how we use ZFS holds and bookmarks to ensure that a single replication step is always resumable.

The replication algorithm described in the design doc introduces the notion of job IDs (please read the details on this design doc).
We reuse the job names for job IDs and use `JobID` type to ensure that a job name can be embedded into hold tags, bookmark names, etc.
This might BREAK CONFIG on upgrade.

Protocol Version Bump
---------------------

This commit makes backwards-incompatible changes to the replication/pdu protobufs.
Thus, bump the version number used in the protocol handshake.

Replication Cursor Format Change
--------------------------------

The new replication cursor bookmark format is: `#zrepl_CURSOR_G_${this.GUID}_J_${jobid}`
Including the GUID enables transaction-safe moving-forward of the cursor.
Including the job id enables that multiple sending jobs can send the same filesystem without interfering.
The `zrepl migrate replication-cursor:v1-v2` subcommand can be used to safely destroy old-format cursors once zrepl has created new-format cursors.

Changes in This Commit
----------------------

- package zfs
  - infrastructure for holds
  - infrastructure for resume token decoding
  - implement a variant of OpenZFS's `entity_namecheck` and use it for validation in new code
  - ZFSSendArgs to specify a ZFS send operation
    - validation code protects against malicious resume tokens by checking that the token encodes the same send parameters that the send-side would use if no resume token were available (i.e. same filesystem, `fromguid`, `toguid`)
  - RecvOptions support for `recv -s` flag
  - convert a bunch of ZFS operations to be idempotent
    - achieved through more differentiated error message scraping / additional pre-/post-checks

- package replication/pdu
  - add field for encryption to send request messages
  - add fields for resume handling to send & recv request messages
  - receive requests now contain `FilesystemVersion To` in addition to the filesystem into which the stream should be `recv`d into
    - can use `zfs recv $root_fs/$client_id/path/to/dataset@${To.Name}`, which enables additional validation after recv (i.e. whether `To.Guid` matched what we received in the stream)
    - used to set `last-received-hold`
- package replication/logic
  - introduce `PlannerPolicy` struct, currently only used to configure whether encrypted sends should be requested from the sender
  - integrate encryption and resume token support into `Step` struct

- package endpoint
  - move the concepts that endpoint builds on top of ZFS to a single file `endpoint/endpoint_zfs.go`
    - step-holds + step-bookmarks
    - last-received-hold
    - new replication cursor + old replication cursor compat code
  - adjust `endpoint/endpoint.go` handlers for
    - encryption
    - resumability
    - new replication cursor
    - last-received-hold

- client subcommand `zrepl holds list`: list all holds and hold-like bookmarks that zrepl thinks belong to it
- client subcommand `zrepl migrate replication-cursor:v1-v2`
2020-02-14 22:00:13 +01:00
93ccdb8024 docs: prune: fix typo 2020-02-14 21:40:48 +01:00
501645f918 docs: filter syntax: reference 'snap' job type 2020-02-14 21:40:48 +01:00
5b50a66c6c daemon/snapper: refactor sync-up algorithm + warn about FSes awaiting first sync point
refs https://github.com/zrepl/zrepl/issues/256
2020-01-15 19:20:37 +01:00
d35e2400b2 transport/{TCP,TLS}: optional IP_FREEBIND / IP_BINDANY bind socketops
Allows to bind to an address even if it is not actually (yet or ever)
configured. Fixes #238

Rationale:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/#whatdoesthismeanformeadeveloper
2020-01-04 17:21:48 +01:00
7e743c74dc docs + samples: adjust ssh 'Compression' arg in examples 2019-11-20 18:19:16 +01:00
d2bc40f78d docs: transports: ssh: better copy-pastable connect section 2019-11-16 22:07:47 +01:00
ffe677e55a docs: snapshotting: command hook type: not the only hook type anymore 2019-10-13 12:16:31 +02:00
5c95c21727 transport/local: configurable dial_timeout for connect, default 2s 2019-09-29 19:05:54 +02:00
729c83ee72 pre- and post-snapshot hooks
* stack-based execution model, documented in documentation
* circbuf for capturing hook output
* built-in hooks for postgres and mysql
* refactor docs, too much info on the jobs page, too difficult
  to discover snapshotting & hooks

Co-authored-by: Ross Williams <ross@ross-williams.net>
Co-authored-by: Christian Schwarz <me@cschwarz.com>

fixes #74
2019-09-27 21:25:59 +02:00
2107483588 docs: jobs: correct root_fs field explanation 2019-03-21 12:51:39 +01:00
c2b05954e9 docs: jobs: explain multi-job & machine considerations
refs #136
refs #140
2019-03-21 12:51:07 +01:00
2f2e6e6a00 receiving side: placeholder as simple on|off property 2019-03-20 20:26:30 +01:00
6f7467e8d8 Merge branch 'InsanePrawn-master' into 'master' 2019-03-20 19:45:00 +01:00
f2a6193735 Merge branch 'problame/replication_refactor' 2019-03-20 19:44:15 +01:00
fb999c8617 docs: drop references to 'main config file'
were just confusing in the remaining places where they were used

fixes #127
2019-03-18 14:50:32 +01:00
53dc0c3c5e docs: document existence of zrepl:placeholder property
fixes #129
2019-03-18 14:50:32 +01:00
c9b812570d docs: changelog: document new rpc & retry behavior changes 2019-03-18 12:39:53 +01:00
dd673bf923 docs: condense snap job overview table row 2019-03-17 21:29:09 +01:00
e8c0d206ea docs: fix nitpicks 2019-03-17 20:54:47 +01:00
17818439a0 Merge branch 'problame/replication_refactor' into InsanePrawn-master 2019-03-17 17:33:51 +01:00
056be1185d dist: add grafana dashboard
fixes #116
2019-03-16 16:12:34 +01:00
da3ba50a2c Merge remote-tracking branch 'origin/master' into problame/replication_refactor 2019-03-16 14:48:01 +01:00
4ee00091d6 pull job: support manual-only invocation 2019-03-16 14:24:05 +01:00
34052d98d6 docs: move snap job below all replication-related job types 2019-03-15 22:00:29 +01:00
3543fbbb65 docs: clarify language & example of source-side pruning workaround 2019-03-15 22:00:29 +01:00
aff639e87a Merge remote-tracking branch 'origin/master' into InsanePrawn-master 2019-03-15 21:05:20 +01:00
78ec5aa716 Merge remote-tracking branch 'origin/master' into joshsouza-fix_peer_cert_chains 2019-03-15 18:37:11 +01:00
fc311a9fd6 syslog logging: support setting facility in config 2019-03-15 17:55:11 +01:00
a7993d18c6 transport/tls: clarify docs & error message language 2019-03-15 17:17:25 +01:00
5595cff6a6 Merge branch 'master' into fix_peer_cert_chains 2019-03-15 16:34:21 +01:00
796c5ad42d rpc rewrite: control RPCs using gRPC + separate RPC for data transfer
transport/ssh: update go-netssh to new version
    => supports CloseWrite and Deadlines
    => build: require Go 1.11 (netssh requires it)
2019-03-13 13:53:48 +01:00
f724480c7b Add documentation regarding using a certificate chain 2019-01-22 10:09:24 -08:00
5c5e8c0baf Documentation changes mostly as requested 2019-01-22 16:46:34 +01:00
1aae7b222f docs: fix confusing description of the role of client identity for sink jobs 2018-12-01 15:19:59 +01:00
d977796f18 Add SnapJob docs 2018-11-21 16:59:46 +01:00
5e1ea21f85 pruning: add 'Negate' option to KeepRegex and expose it in config 2018-11-16 12:21:54 +01:00
ad8be226fd fix small typo 2018-10-22 11:32:37 +02:00
9b3e5c38e2 docs: fix changelog + invocations of wakeup subcommand 2018-10-22 11:27:00 +02:00
7e1c5f5d1f docs: discourage use of ssh+stdinserver transport due to inferior error handling 2018-10-22 11:25:16 +02:00
a62b475f46 docs/transport/tls: document self-signed certs procedure for 2-machine setup 2018-10-21 22:20:07 +02:00
f13749380d docs: add warnings of changing semantics for manually created snapshots in 0.1 2018-10-13 18:34:37 +02:00
f6cf23779f docs: Remove stale TIP for dry-run zrepl test subcommand.
Won't make it to 0.1
2018-10-13 16:22:19 +02:00
92a1a6d2ca docs: fix wrong subcommand for configcheck 2018-10-13 16:22:19 +02:00
63169c51b7 add 'test filesystems' subcommand for testing filesystem filters 2018-10-13 16:22:19 +02:00
1643198713 docs: reflect changes in replication_rewrite branch 2018-10-11 18:03:18 +02:00
88de8ba8bb initial repl policy: get rid of unimplemented options 2018-08-25 22:23:47 +02:00
82ea535692 daemon: expose prometheus in new global.monitoring config section + document it
refs #67
2018-04-14 11:24:47 +02:00
9b803aad2d docs: tutorial: document known_hosts file setup
fixes #64
2018-04-01 14:58:04 +02:00
7f89372cfa docs: fix enumeration in ssh+stdinserver docs 2018-03-04 17:20:08 +01:00