Commit Graph

525 Commits

Author SHA1 Message Date
Christian Schwarz
7f9eb62640 sink: concurrent connection handling 2018-09-18 22:44:00 +02:00
Christian Schwarz
6c3f442f13 daemon control / jsonclient: fix connection leak due to open request body
Also:
- Defensive measures in control http server (1s timeouts)
(prevent the leak, even if request body is not closed)
- Add prometheus metrics to track control socket latencies
(were used for debugging)
2018-09-13 12:44:46 +02:00
Christian Schwarz
fa47667f31 bring back prometheus metrics, with new metrics for replication state machine 2018-09-07 22:22:34 -07:00
Christian Schwarz
ab9446137f fix missing import of errors pacakge 2018-09-07 22:22:34 -07:00
Christian Schwarz
0c2ac3a168 pprof subcommand 2018-09-07 00:04:03 -07:00
Christian Schwarz
bf5099baac version subcommand: unified client & server 2018-09-06 23:52:11 -07:00
Christian Schwarz
7836ea36fc serve TLS: validate client CNs against whitelist in config file 2018-09-06 13:34:39 -07:00
Christian Schwarz
1edf020ce7 status command: better handling of 'nothing to do' Complete state 2018-09-06 11:46:02 -07:00
Christian Schwarz
c60ed78bc5 status subcommand: only draw one big progress bar of the entire replication
more details on progress per step in text form
2018-09-06 11:05:32 -07:00
Christian Schwarz
82d51cd0dc go vet fix 2018-09-05 21:48:52 -07:00
Christian Schwarz
0f75677e59 daemon/pruner: fix exercise (don't call it test) 2018-09-05 21:47:44 -07:00
Christian Schwarz
2c25f28972 simplify mapping & filtering in endpoints (re-rooting only) 2018-09-05 19:51:06 -07:00
Christian Schwarz
1323a30a0c zfs: ability to specify sources for zfsGet
fix use for Placeholder, leave rest as previous behavior
2018-09-05 19:51:06 -07:00
Christian Schwarz
975fdee217 replication & pruning: ditch replicated-property, use bookmark as cursor instead
A bookmark with a well-known name is used to track which version was
last successfully received by the receiver.
The createtxg that can be retrieved from the bookmark using `zfs get` is
used to set the Replicated attribute of each snap on the sender:
If the snap's CreateTXG > the cursor's, it is not yet replicated,
otherwise it has been.

There is an optional config option to change the behvior to
`CreateTXG >= the cursor's`, and the implementation defaults to that.

The reason: While things work just fine with `CreateTXG > the cursor's`,
ZFS does not provide size estimates in a `zfs send` dry run
(see acd2418).
However, to enable the use case of keeping the snapshot only around for
the replication, the config flag exists.
2018-09-05 19:51:06 -07:00
Christian Schwarz
acd2418803 handle DryRun send size estimate errors with bookmarks 2018-09-05 17:41:25 -07:00
Christian Schwarz
9eca269ad8 fixup 308e5e35fb: remove fprintf debug output 2018-09-05 08:35:31 -07:00
Christian Schwarz
c21222ef13 serve/tls: use handshake timeout from config 2018-09-05 08:32:59 -07:00
Christian Schwarz
6c988d0ebb add small subcommand to validate config 2018-09-05 08:32:38 -07:00
Christian Schwarz
4b39a18178 zfs: disable resume token test because it doesn't work in docker 2018-09-04 17:31:46 -07:00
Christian Schwarz
6c31c66562 hidden bashcomp command 2018-09-04 17:27:20 -07:00
Christian Schwarz
adab06405b make go vet happy 2018-09-04 17:25:10 -07:00
Christian Schwarz
52f0c0c33b update Makefile 2018-09-04 17:19:59 -07:00
Christian Schwarz
bfc631f6a6 fix broken pruner exercise (don't call it test...) 2018-09-04 17:19:59 -07:00
Christian Schwarz
1e27720b99 zfs: skip test with ZFS_BINARY mock (doesn't work in parallel) 2018-09-04 17:02:02 -07:00
Christian Schwarz
8eade3d20a replication/pdu: fix broken test 2018-09-04 17:01:46 -07:00
Christian Schwarz
308e5e35fb Multi-client servers + bring back stdinserver support 2018-09-04 16:43:55 -07:00
Christian Schwarz
e161347e47 Implement periodic snapshotting. 2018-09-04 16:43:55 -07:00
Christian Schwarz
754b253043 config: no-field for replication anymore
It's closer to the original config and we don't want users to specify
'filesystems' and similar multiple times in a single job definition.
2018-09-04 14:44:45 -07:00
Christian Schwarz
be57d6ce8e replication/diff: replace invalid comparison of CreateTXG with Creation 2018-09-04 14:01:48 -07:00
Christian Schwarz
4336af295f fixup 22ca80eb7e: scraping regex was broken and potentially mixed with stdout 2018-09-04 14:01:48 -07:00
Christian Schwarz
0c4a3f8dc4 pruning/history: properly communicate via rpc if snapshot does not exist 2018-09-04 14:01:48 -07:00
Christian Schwarz
8799108b55 fixup b95e983d0d: prunerFactory: fix duplicate logger fields 2018-09-03 13:19:56 -07:00
Christian Schwarz
03f9f81cb5 fixup 3d8e552c6a: validate streamrpc config in factory constructors 2018-09-03 13:17:53 -07:00
Christian Schwarz
2da0e51fda Update Gopkg.lock to latest versions of streamrpc and yaml-config 2018-09-02 15:49:17 -07:00
Christian Schwarz
f0860767f5 zfs: include stderr of command in ZFSError.Error()
Since we don't implement screen-scraping of ZFS output ATM, this is
better than nothing, as user's may be able to figure out what' sthe
problem from the logs / status reports.
2018-09-02 15:46:42 -07:00
Christian Schwarz
ad28fd1ecb replication: diff does not need special case for receiver/sender == nil 2018-09-02 15:46:42 -07:00
Christian Schwarz
3d8e552c6a streamrpc 0.3 + config from daemon/config 2018-09-02 15:46:42 -07:00
Christian Schwarz
d55a271ac7 WIP adopt updated yaml-config with 'fromdefaults' struct tag 2018-09-02 15:46:03 -07:00
Christian Schwarz
b95e983d0d bump go-streamrpc to 0.2, cleanup logging
logging should be user-friendly in INFO mode
2018-09-02 15:45:18 -07:00
Anton Schirg
f387e23214 fix: at least two snapshots were needed to start replication 2018-08-30 19:20:18 +02:00
Anton Schirg
32391adf4f build pruner in factory and check prune rules 2018-08-30 19:20:14 +02:00
Anton Schirg
c0a3e1f121 wrap error in buildJob with job name 2018-08-30 17:40:02 +02:00
Anton Schirg
5442d8e7d5 status: calculate max fs name length 2018-08-30 15:21:07 +02:00
Anton Schirg
48feaff054 fix some status display alignment 2018-08-30 15:21:07 +02:00
Christian Schwarz
acd2a68cfb fix build: bump yaml-config 2018-08-30 13:40:28 +02:00
Christian Schwarz
1690339440 colorized stdout logger if stdout is tty 2018-08-30 13:33:28 +02:00
Anton Schirg
b5957aca37 do dry runs in planning stage to estimate size of all sends 2018-08-30 12:59:16 +02:00
Anton Schirg
47d8a5a7cd status: only show active not all versions of active filesystem 2018-08-30 12:58:13 +02:00
Anton Schirg
583773025f nicer progress bar 2018-08-30 12:58:13 +02:00
Anton Schirg
98f3f3dfd8 show expected size of current send
Needs to be changed to send sizes for all planned steps
2018-08-30 12:58:13 +02:00