Commit Graph

70 Commits

Author SHA1 Message Date
Christian Schwarz
10a14a8c50 [#307] add package trace, integrate it with logging, and adopt it throughout zrepl
package trace:

- introduce the concept of tasks and spans, tracked as linked list within ctx
    - see package-level docs for an overview of the concepts
    - **main feature 1**: unique stack of task and span IDs
        - makes it easy to follow a series of log entries in concurrent code
    - **main feature 2**: ability to produce a chrome://tracing-compatible trace file
        - either via an env variable or a `zrepl pprof` subcommand
        - this is not a CPU profile, we already have go pprof for that
        - but it is very useful to visually inspect where the
          replication / snapshotter / pruner spends its time
          ( fixes #307 )

usage in package daemon/logging:

- goal: every log entry should have a trace field with the ID stack from package trace

- make `logging.GetLogger(ctx, Subsys)` the authoritative `logger.Logger` factory function
    - the context carries a linked list of injected fields which
      `logging.GetLogger` adds to the logger it returns
    - `logging.GetLogger` also uses package `trace` to get the
      task-and-span-stack and injects it into the returned logger's fields
2020-05-19 11:30:02 +02:00
InsanePrawn
44bd354eae Spellcheck all files
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-02-24 16:06:09 +01:00
Christian Schwarz
58c08c855f new features: {resumable,encrypted,hold-protected} send-recv, last-received-hold
- **Resumable Send & Recv Support**
  No knobs required, automatically used where supported.
- **Hold-Protected Send & Recv**
  Automatic ZFS holds to ensure that we can always resume a replication step.
- **Encrypted Send & Recv Support** for OpenZFS native encryption.
  Configurable at the job level, i.e., for all filesystems a job is responsible for.
- **Receive-side hold on last received dataset**
  The counterpart to the replication cursor bookmark on the send-side.
  Ensures that incremental replication will always be possible between a sender and receiver.

Design Doc
----------

`replication/design.md` doc describes how we use ZFS holds and bookmarks to ensure that a single replication step is always resumable.

The replication algorithm described in the design doc introduces the notion of job IDs (please read the details on this design doc).
We reuse the job names for job IDs and use `JobID` type to ensure that a job name can be embedded into hold tags, bookmark names, etc.
This might BREAK CONFIG on upgrade.

Protocol Version Bump
---------------------

This commit makes backwards-incompatible changes to the replication/pdu protobufs.
Thus, bump the version number used in the protocol handshake.

Replication Cursor Format Change
--------------------------------

The new replication cursor bookmark format is: `#zrepl_CURSOR_G_${this.GUID}_J_${jobid}`
Including the GUID enables transaction-safe moving-forward of the cursor.
Including the job id enables that multiple sending jobs can send the same filesystem without interfering.
The `zrepl migrate replication-cursor:v1-v2` subcommand can be used to safely destroy old-format cursors once zrepl has created new-format cursors.

Changes in This Commit
----------------------

- package zfs
  - infrastructure for holds
  - infrastructure for resume token decoding
  - implement a variant of OpenZFS's `entity_namecheck` and use it for validation in new code
  - ZFSSendArgs to specify a ZFS send operation
    - validation code protects against malicious resume tokens by checking that the token encodes the same send parameters that the send-side would use if no resume token were available (i.e. same filesystem, `fromguid`, `toguid`)
  - RecvOptions support for `recv -s` flag
  - convert a bunch of ZFS operations to be idempotent
    - achieved through more differentiated error message scraping / additional pre-/post-checks

- package replication/pdu
  - add field for encryption to send request messages
  - add fields for resume handling to send & recv request messages
  - receive requests now contain `FilesystemVersion To` in addition to the filesystem into which the stream should be `recv`d into
    - can use `zfs recv $root_fs/$client_id/path/to/dataset@${To.Name}`, which enables additional validation after recv (i.e. whether `To.Guid` matched what we received in the stream)
    - used to set `last-received-hold`
- package replication/logic
  - introduce `PlannerPolicy` struct, currently only used to configure whether encrypted sends should be requested from the sender
  - integrate encryption and resume token support into `Step` struct

- package endpoint
  - move the concepts that endpoint builds on top of ZFS to a single file `endpoint/endpoint_zfs.go`
    - step-holds + step-bookmarks
    - last-received-hold
    - new replication cursor + old replication cursor compat code
  - adjust `endpoint/endpoint.go` handlers for
    - encryption
    - resumability
    - new replication cursor
    - last-received-hold

- client subcommand `zrepl holds list`: list all holds and hold-like bookmarks that zrepl thinks belong to it
- client subcommand `zrepl migrate replication-cursor:v1-v2`
2020-02-14 22:00:13 +01:00
Matthias Freund
4ba85c40cc daemon/job/build_jobs: fix validateReceivingSidesDoNotOverlap 2020-01-30 10:46:32 +01:00
Juergen Hoetzel
b3231d2bed daemon: fix typos in error messages
closes #255
2019-12-11 21:30:48 +01:00
Christian Schwarz
b5ff1a9926 snapper + client/status: snapshotting reports 2019-09-27 21:31:00 +02:00
Christian Schwarz
77d3a1ad4d build: drop go Dep, switch to modules, support Go 1.13
bump enumer to v1.1.1
bump golangci-lint to v1.17.1

no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output

fixes #112
2019-09-14 13:36:44 +02:00
Christian Schwarz
f31b54582f daemon/job: fix receiving-job root-fs overlap detection
fixup for 7756c9a5
fixes #160
2019-03-27 13:12:26 +01:00
Christian Schwarz
5b97953bfb run golangci-lint and apply suggested fixes 2019-03-27 13:12:26 +01:00
Christian Schwarz
afed762774 format source tree using goimports 2019-03-22 19:41:12 +01:00
Christian Schwarz
7756c9a55c config + job: forbid non-verlapping receiver root_fs
refs #136
refs #140
2019-03-21 12:07:55 +01:00
Christian Schwarz
158d1175e3 rename SinglePruner to LocalPruner 2019-03-17 21:18:25 +01:00
Christian Schwarz
b25da7b9b0 job: snap: comment fix 2019-03-17 21:07:42 +01:00
Christian Schwarz
5cd2593f52 job: snap: workaround for replication cursor requirement 2019-03-17 21:07:01 +01:00
Christian Schwarz
17818439a0 Merge branch 'problame/replication_refactor' into InsanePrawn-master 2019-03-17 17:33:51 +01:00
Christian Schwarz
4ee00091d6 pull job: support manual-only invocation 2019-03-16 14:24:05 +01:00
Christian Schwarz
aff639e87a Merge remote-tracking branch 'origin/master' into InsanePrawn-master 2019-03-15 21:05:20 +01:00
Christian Schwarz
07b43bffa4 replication: refactor driving logic (no more explicit state machine) 2019-03-13 15:00:40 +01:00
Christian Schwarz
796c5ad42d rpc rewrite: control RPCs using gRPC + separate RPC for data transfer
transport/ssh: update go-netssh to new version
    => supports CloseWrite and Deadlines
    => build: require Go 1.11 (netssh requires it)
2019-03-13 13:53:48 +01:00
InsanePrawn
160a3b6d32 more gofmt, drop snapjob.go_prefmt after it was accidentally added 2018-11-21 22:14:43 +01:00
InsanePrawn
3cef76d463 Refactor snapJob() to snapJobFromConfig() 2018-11-21 14:37:03 +01:00
InsanePrawn
e9564a7e5c Inlined a couple legacy leftover functions from the mode copypasta 2018-11-21 14:35:40 +01:00
InsanePrawn
d0f898751f Gofmt snapjob.go 2018-11-21 14:02:21 +01:00
InsanePrawn
22d9830baa Fix prometheus with multiple jobs 2018-11-21 04:26:03 +01:00
InsanePrawn
e10dc129de Make getPruner() private 2018-11-21 03:39:03 +01:00
InsanePrawn
dd11fc96db Touchups in job.go 2018-11-21 03:27:39 +01:00
InsanePrawn
7de3c0a09a Removed the references to a pruning 'side' in the singlepruner logging code and the snapjob prometheus thing. 2018-11-21 02:52:33 +01:00
InsanePrawn
141e49727c Missed a last reference to tasks 2018-11-21 02:51:23 +01:00
InsanePrawn
442d61918b remove most of the watchdog machinery 2018-11-21 02:42:13 +01:00
InsanePrawn
58dcc07430 Added SnapJobStatus 2018-11-21 02:08:39 +01:00
InsanePrawn
19d0916e34 remove snapMode, rename snap_ActiveSide to SnapJob 2018-11-21 01:54:56 +01:00
InsanePrawn
1265cc7934 pruned unused lines and comments ;) 2018-11-21 01:34:50 +01:00
InsanePrawn
3d2688e959 Ugly but working inital snapjob implementation 2018-11-20 19:30:15 +01:00
Christian Schwarz
98bc8d1717 daemon/job: explicit notice of ZREPL_JOB_WATCHDOG_TIMEOUT environment variable on cancellation 2018-10-22 11:03:31 +02:00
Christian Schwarz
94427d334b replication + pruner + watchdog: adjust timeouts based on practical experience 2018-10-21 18:37:57 +02:00
Christian Schwarz
190c7270d9 daemon/active + watchdog: simplify control flow using explicit ActiveSideState 2018-10-21 12:53:34 +02:00
Christian Schwarz
f704b28cad daemon/job: track active side state explicitly 2018-10-21 12:52:48 +02:00
Christian Schwarz
e63ac7d1bb pruner: log transitions to error state + log info to confirm pruning is done in active job 2018-10-19 17:23:00 +02:00
Christian Schwarz
69bfcb7bed daemon/active: implement watchdog to handle stuck replication / pruners
ActiveSide.do() can only run sequentially, i.e. we cannot run
replication and pruning in parallel. Why?

* go-streamrpc only allows one active request at a time
(this is bad design and should be fixed at some point)
* replication and pruning are implemented independently, but work on the
same resources (snapshots)

A: pruning might destroy a snapshot that is planned to be replicated
B: replication might replicate snapshots that should be pruned

We do not have any resource management / locking for A and B, but we
have a use case where users don't want their machine fill up with
snapshots if replication does not work.
That means we _have_ to run the pruners.

A further complication is that we cannot just cancel the replication
context after a timeout and move on to the pruner: it could be initial
replication and we don't know how long it will take.
(And we don't have resumable send & recv yet).

With the previous commits, we can implement the watchdog using context
cancellation.
Note that the 'MadeProgress()' calls can only be placed right before
non-error state transition. Otherwise, we could end up in a live-lock.
2018-10-19 17:23:00 +02:00
Christian Schwarz
82f0060eec Revert "daemon/job/active: push mode: awful hack for handling of concurrent snapshots + stale remote operation"
This reverts commit aeb87ffbcf.
2018-10-19 09:35:30 +02:00
Christian Schwarz
a5376913fd daemon/job: fix buildJob returning nil error on job uild error
Would show up as ugly nil-pointer-deref panic later during daemon
startup
2018-10-18 16:19:27 +02:00
Christian Schwarz
aeb87ffbcf daemon/job/active: push mode: awful hack for handling of concurrent snapshots + stale remote operation
We have the problem that there are legitimate use cases where a user
does not want their machine to fill up with snapshots, even if it means
unreplicated must be destroyed.  This can be expressed by *not*
configuring the keep rule `not_replicated` for the snapshot-creating
side.  This commit only addresses push mode because we don't support
pruning in the source job. We adivse users in the docs to use push mode
if they have above use case, so this is fine - at least for 0.1.

Ideally, the replication.Replication would communicate to the pruner
which snapshots are currently part of the replication plan, and then
we'd need some conflict resolution to determine whether it's more
important to destroy the snapshots or to replicate them (destroy should
win?).

However, we don't have the infrastructure for this yet (we could parse
the replication report, but that's just ugly).  And we want to get 0.1
out, so showtime for a dirty hack:

We start replication, and ideally, replication and pruning is done
before new snapshot have been taken. If so: great. However, what happens
if snapshots have been taken and we are not done with replication and /
or pruning?

* If replicatoin is making progress according to its state, let it run.
This covers the *important* situation of initial replication, where
replication may easily take longer than a single snapshotting interval.

* If replication is in an error state, cancel it through context
cancellation.
    * As with the pruner below, the main problem here is that
      status output will only contain "context cancelled" after the
      cancellation, instead of showing the reason why it was cancelled.
      Not nice, but oh well, the logs provide enough detail for this
      niche situation...

* If we are past replication, we're still pruning

* Leave the local (send-side) pruning alone.
Again, we only implement this hack for push, so we know sender is
local, and it will only fail hard, not retry.

* If the remote (receiver-side) pruner is in an error state, cancel it
through context cancellation.

* Otherwise, let it run.

Note that every time we "let it run", we tolerate a temporary excess of
snapshots, but given sufficiently aggressive timeouts and the assumption
that the snapshot interval is much greater than the timeouts, this is
not a significant problem in practice.
2018-10-12 22:47:06 +02:00
Christian Schwarz
d584e1ac54 daemon/job/active: fix race in updateTasks
If concurrent updates strictly modify *different* members of the tasks
struct, the copying + lock-drop still constitutes a race condition:
The last updater always wins and sets tasks to its copy + changes.
This eliminates the other updater's changes.
2018-10-12 22:15:07 +02:00
Christian Schwarz
89e0103abd move wakeup subcommand into signal subcommand and add reset subcommand 2018-10-12 20:50:56 +02:00
Christian Schwarz
f9d24d15ed move wakup mechanism into separate package 2018-10-12 12:44:40 +02:00
Christian Schwarz
be962998ba move serve and connecter into transports package 2018-10-11 21:21:46 +02:00
Christian Schwarz
125b561df3 rename root_dataset to root_fs for receiving-side jobs 2018-10-11 18:03:18 +02:00
Christian Schwarz
4e16952ad9 snapshotting: support 'periodic' and 'manual' mode
1. Change config format to support multiple types
   of snapshotting modes.
2. Implement a hacky way to support periodic or completely
   manual snaphots.

In manual mode, the user has to trigger replication using the wakeup
mechanism after they took snapshots using their own tooling.

As indicated by the comment, a more general solution would be desirable,
but we want to get the release out and 'manual' mode is a feature that
some people requested...
2018-10-11 15:59:23 +02:00
Christian Schwarz
75e42fd860 pruner: implement Report method + display in status command 2018-09-24 19:25:40 +02:00
Christian Schwarz
75ba5874a5 active side: track activities in Run() as atomically updated member 2018-09-24 19:23:53 +02:00