Commit Graph

193 Commits

Author SHA1 Message Date
3ff1966cab docs/installation: use && for early exit if build-in-docker step fails 2020-02-17 18:02:04 +01:00
b8d9f4ba92 docs: supporters: update 2020-02-14 22:00:13 +01:00
58c08c855f new features: {resumable,encrypted,hold-protected} send-recv, last-received-hold
- **Resumable Send & Recv Support**
  No knobs required, automatically used where supported.
- **Hold-Protected Send & Recv**
  Automatic ZFS holds to ensure that we can always resume a replication step.
- **Encrypted Send & Recv Support** for OpenZFS native encryption.
  Configurable at the job level, i.e., for all filesystems a job is responsible for.
- **Receive-side hold on last received dataset**
  The counterpart to the replication cursor bookmark on the send-side.
  Ensures that incremental replication will always be possible between a sender and receiver.

Design Doc
----------

`replication/design.md` doc describes how we use ZFS holds and bookmarks to ensure that a single replication step is always resumable.

The replication algorithm described in the design doc introduces the notion of job IDs (please read the details on this design doc).
We reuse the job names for job IDs and use `JobID` type to ensure that a job name can be embedded into hold tags, bookmark names, etc.
This might BREAK CONFIG on upgrade.

Protocol Version Bump
---------------------

This commit makes backwards-incompatible changes to the replication/pdu protobufs.
Thus, bump the version number used in the protocol handshake.

Replication Cursor Format Change
--------------------------------

The new replication cursor bookmark format is: `#zrepl_CURSOR_G_${this.GUID}_J_${jobid}`
Including the GUID enables transaction-safe moving-forward of the cursor.
Including the job id enables that multiple sending jobs can send the same filesystem without interfering.
The `zrepl migrate replication-cursor:v1-v2` subcommand can be used to safely destroy old-format cursors once zrepl has created new-format cursors.

Changes in This Commit
----------------------

- package zfs
  - infrastructure for holds
  - infrastructure for resume token decoding
  - implement a variant of OpenZFS's `entity_namecheck` and use it for validation in new code
  - ZFSSendArgs to specify a ZFS send operation
    - validation code protects against malicious resume tokens by checking that the token encodes the same send parameters that the send-side would use if no resume token were available (i.e. same filesystem, `fromguid`, `toguid`)
  - RecvOptions support for `recv -s` flag
  - convert a bunch of ZFS operations to be idempotent
    - achieved through more differentiated error message scraping / additional pre-/post-checks

- package replication/pdu
  - add field for encryption to send request messages
  - add fields for resume handling to send & recv request messages
  - receive requests now contain `FilesystemVersion To` in addition to the filesystem into which the stream should be `recv`d into
    - can use `zfs recv $root_fs/$client_id/path/to/dataset@${To.Name}`, which enables additional validation after recv (i.e. whether `To.Guid` matched what we received in the stream)
    - used to set `last-received-hold`
- package replication/logic
  - introduce `PlannerPolicy` struct, currently only used to configure whether encrypted sends should be requested from the sender
  - integrate encryption and resume token support into `Step` struct

- package endpoint
  - move the concepts that endpoint builds on top of ZFS to a single file `endpoint/endpoint_zfs.go`
    - step-holds + step-bookmarks
    - last-received-hold
    - new replication cursor + old replication cursor compat code
  - adjust `endpoint/endpoint.go` handlers for
    - encryption
    - resumability
    - new replication cursor
    - last-received-hold

- client subcommand `zrepl holds list`: list all holds and hold-like bookmarks that zrepl thinks belong to it
- client subcommand `zrepl migrate replication-cursor:v1-v2`
2020-02-14 22:00:13 +01:00
93ccdb8024 docs: prune: fix typo 2020-02-14 21:40:48 +01:00
501645f918 docs: filter syntax: reference 'snap' job type 2020-02-14 21:40:48 +01:00
5b50a66c6c daemon/snapper: refactor sync-up algorithm + warn about FSes awaiting first sync point
refs https://github.com/zrepl/zrepl/issues/256
2020-01-15 19:20:37 +01:00
dd508280f0 docs: tutorial: minor typo + language fixes 2020-01-15 19:12:09 +01:00
d35e2400b2 transport/{TCP,TLS}: optional IP_FREEBIND / IP_BINDANY bind socketops
Allows to bind to an address even if it is not actually (yet or ever)
configured. Fixes #238

Rationale:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/#whatdoesthismeanformeadeveloper
2020-01-04 17:21:48 +01:00
47ed599db7 docs: add Void Linux to installation instructions 2019-12-28 12:43:53 +01:00
5e17d7ba80 docs: add recent supporters 2019-11-26 00:45:13 +01:00
0261dbfe3d docs: 0.2.1 changelog 2019-11-20 20:16:41 +01:00
7e743c74dc docs + samples: adjust ssh 'Compression' arg in examples 2019-11-20 18:19:16 +01:00
27db3e6f70 docs: supporters: update & add viz for different kinds of support 2019-11-16 22:11:07 +01:00
d2bc40f78d docs: transports: ssh: better copy-pastable connect section 2019-11-16 22:07:47 +01:00
6eda1f743f Fix typo in tutorial.rst 2019-11-05 09:57:30 -08:00
6787decef1 Add OmniOS (illumos distribution) to list of OSs 2019-11-05 09:49:57 -08:00
d56d45a2ab docs: install: apt: fix snippet display & link to packaging repo 2019-10-21 16:35:23 +02:00
fcf16a163a docs: install: apt snippet: idempotent, bash compat, multiarch compat
Co-authored-by: Janis Streib <me@janis-streib.de>
Co-authored-by: Christian Schwarz <me@cschwarz.com>
2019-10-21 16:21:51 +02:00
dc39c819a3 docs: add debian + ubuntu installation 2019-10-18 20:18:42 +02:00
3806e97404 docs: add copr repo for Fedora/CentOS
closes #229
2019-10-16 10:46:02 +02:00
b422e6f12e docs: installation: add Arch Linux 'from source' package 2019-10-13 12:33:27 +02:00
f8d5082bdd docs: remove outdated implementation references + remove 0.2-rc* from published docs 2019-10-13 12:26:39 +02:00
ffe677e55a docs: snapshotting: command hook type: not the only hook type anymore 2019-10-13 12:16:31 +02:00
ad77371e38 docs: include Arch Linux installation 2019-10-06 20:38:00 +02:00
3edfe535c6 docs: fix typo on index page 2019-10-05 14:59:51 +02:00
d3b99e8e39 Fix typo 2019-10-05 14:58:49 +02:00
3c03f21419 docs: SEPA hint, supporters, fix publish script 2019-10-03 11:57:19 +02:00
5c95c21727 transport/local: configurable dial_timeout for connect, default 2s 2019-09-29 19:05:54 +02:00
8af824df41 docs: promote monetary support in changelog 2019-09-29 19:04:53 +02:00
58ab25919e platformtest: dedicated pool per test, Makefile target, maintainer notice
fixes #216
fixes #211
2019-09-29 18:48:44 +02:00
215848f476 docs: 0.2 changelog 2019-09-28 17:50:07 +02:00
729c83ee72 pre- and post-snapshot hooks
* stack-based execution model, documented in documentation
* circbuf for capturing hook output
* built-in hooks for postgres and mysql
* refactor docs, too much info on the jobs page, too difficult
  to discover snapshotting & hooks

Co-authored-by: Ross Williams <ross@ross-williams.net>
Co-authored-by: Christian Schwarz <me@cschwarz.com>

fixes #74
2019-09-27 21:25:59 +02:00
07956c2299 zfs,endpoint: use zfs destroy batch syntax if available
refs #72
2019-09-14 13:43:46 +02:00
77d3a1ad4d build: drop go Dep, switch to modules, support Go 1.13
bump enumer to v1.1.1
bump golangci-lint to v1.17.1

no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output

fixes #112
2019-09-14 13:36:44 +02:00
a65d8f1f4c docs: fix requirements.txt & pin sphinxcontrib-versioning version 2019-09-08 00:47:39 +02:00
424234c2d1 docs: Patreon button + Supporters page 2019-09-07 23:16:57 +02:00
a0cf9cff2a docs: include homebrew installation 2019-08-21 12:47:24 +02:00
3c3606d516 docs: show donations/week for liberaypay 2019-06-29 12:08:15 +02:00
234a327a03 build: Linux arm64 support
* protoc zip fetching
* Makefile:
    * GOOS and GOARCH
    * run vet on all targets

Note: freebsd/arm64 is apparently not supported

fixes #180
refs #181
2019-06-23 15:25:26 +02:00
5138681c13 docs: 0.1.1 changelog 2019-04-06 12:36:06 +02:00
082335df5d docs: fix publish.sh branch whitelisting 2019-03-30 19:03:18 +01:00
be506661a5 docs: bump copyright 2019-03-30 19:01:34 +01:00
38385adf22 docs: update front-page zrepl status 2019-03-30 18:53:28 +01:00
bbcfb47d28 docs: changelog: more maintainer notices 2019-03-30 18:53:28 +01:00
5324f29693 docs: publish.sh: allow v0.1.0-rc4 2019-03-22 12:02:07 +01:00
b96a287a2f docs: adjust publish.sh script 2019-03-22 11:54:49 +01:00
0ab62f197a Merge branch 'problame/overlapping-dataset-hierarchy-improvements' into 'master'
closes #153
2019-03-22 11:02:57 +01:00
e809dbf45e Merge branch 'problame/social_and_donation_shields' 2019-03-21 21:44:29 +01:00
2107483588 docs: jobs: correct root_fs field explanation 2019-03-21 12:51:39 +01:00
c2b05954e9 docs: jobs: explain multi-job & machine considerations
refs #136
refs #140
2019-03-21 12:51:07 +01:00