Commit Graph

24 Commits

Author SHA1 Message Date
Christian Schwarz
000d8bba66 hotfix: limit concurrency of zfs send & recv commands
ATM, the replication logic sends all dry-run requests in parallel,
which might overwhelm the ZFS pool on the sending side.
Since we use rpc/dataconn for dry sends, this also opens one TCP
connection per dry-run request.

Use a sempahore to limit the degree of concurrency where we know it is a
problem ATM.
As indicated by the comments, the cleaner solution would involve some
kind of 'resource exhaustion' error code.

refs #161
refs #164
2019-03-28 22:17:12 +01:00
Christian Schwarz
afed762774 format source tree using goimports 2019-03-22 19:41:12 +01:00
Christian Schwarz
3e71542c78 endpoint: serialize dataset hierarchy modification within a receiver job
refs #136
refs #140
2019-03-20 23:02:10 +01:00
Christian Schwarz
2f2e6e6a00 receiving side: placeholder as simple on|off property 2019-03-20 20:26:30 +01:00
Christian Schwarz
5eababe0b0 endpoing: receiver: return visitErr on traversal error
fixes #137
2019-03-16 14:47:34 +01:00
Christian Schwarz
d50e553ebb handle changes to placeholder state correctly
We assumed that `zfs recv -F FS` would basically replace FS inplace, leaving its children untouched.
That is in fact not the case, it only works if `zfs send -R` is set, which we don't do.

Thus, implement the required functionality manually.

This solves a `zfs recv` error that would occur when a filesystem previously created as placeholder on the receiving side becomes a non-placeholder filesystem (likely due to config change on the sending side):

  zfs send pool1/foo@1 | zfs recv -F pool1/bar
  cannot receive new filesystem stream:
  destination has snapshots (eg. pool1/bar)
  must destroy them to overwrite it
2019-03-13 18:46:04 +01:00
Christian Schwarz
c87759affe replication/driver: automatic retries on connectivity-related errors 2019-03-13 15:00:40 +01:00
Christian Schwarz
07b43bffa4 replication: refactor driving logic (no more explicit state machine) 2019-03-13 15:00:40 +01:00
Christian Schwarz
796c5ad42d rpc rewrite: control RPCs using gRPC + separate RPC for data transfer
transport/ssh: update go-netssh to new version
    => supports CloseWrite and Deadlines
    => build: require Go 1.11 (netssh requires it)
2019-03-13 13:53:48 +01:00
Christian Schwarz
3472145df6 pruner + proto change: better handling of missing replication cursor
- don't treat missing replication cursor as an error in protocol
- treat it as a per-fs planning error instead
2018-11-16 12:21:54 +01:00
Christian Schwarz
814fec60f0 endpoint + zfs: context cancellation of util.IOCommand instances (send & recv for now) 2018-10-19 16:12:21 +02:00
Christian Schwarz
6fcf0635a5 zfs: generalize dry send information for normal sends and with resume token
This is in preparation for resumable send & recv, thus we just don't use
the ResumeToken field for the time being.
2018-10-18 15:56:28 +02:00
Christian Schwarz
6889f441b2 endpoint: support remote ReplicationCursor endpoint 2018-09-24 12:36:10 +02:00
Christian Schwarz
9c86e03384 endpoint Remote: fix broken Send endpoint for DryRun=true 2018-09-24 12:36:10 +02:00
Christian Schwarz
82d51cd0dc go vet fix 2018-09-05 21:48:52 -07:00
Christian Schwarz
2c25f28972 simplify mapping & filtering in endpoints (re-rooting only) 2018-09-05 19:51:06 -07:00
Christian Schwarz
975fdee217 replication & pruning: ditch replicated-property, use bookmark as cursor instead
A bookmark with a well-known name is used to track which version was
last successfully received by the receiver.
The createtxg that can be retrieved from the bookmark using `zfs get` is
used to set the Replicated attribute of each snap on the sender:
If the snap's CreateTXG > the cursor's, it is not yet replicated,
otherwise it has been.

There is an optional config option to change the behvior to
`CreateTXG >= the cursor's`, and the implementation defaults to that.

The reason: While things work just fine with `CreateTXG > the cursor's`,
ZFS does not provide size estimates in a `zfs send` dry run
(see acd2418).
However, to enable the use case of keeping the snapshot only around for
the replication, the config flag exists.
2018-09-05 19:51:06 -07:00
Christian Schwarz
acd2418803 handle DryRun send size estimate errors with bookmarks 2018-09-05 17:41:25 -07:00
Christian Schwarz
0c4a3f8dc4 pruning/history: properly communicate via rpc if snapshot does not exist 2018-09-04 14:01:48 -07:00
Christian Schwarz
b95e983d0d bump go-streamrpc to 0.2, cleanup logging
logging should be user-friendly in INFO mode
2018-09-02 15:45:18 -07:00
Anton Schirg
b5957aca37 do dry runs in planning stage to estimate size of all sends 2018-08-30 12:59:16 +02:00
Anton Schirg
98f3f3dfd8 show expected size of current send
Needs to be changed to send sizes for all planned steps
2018-08-30 12:58:13 +02:00
Christian Schwarz
22ca80eb7e remote snapshot destruction & replication status zfs property 2018-08-30 11:51:47 +02:00
Christian Schwarz
c69ebd3806 WIP rewrite the daemon
cmd subdir does not build on purpose, it's only left in tree to grab old
code and move it to github.com/zrepl/zrepl/daemon
2018-08-27 22:22:44 +02:00