Commit Graph

76 Commits

Author SHA1 Message Date
Christian Schwarz
a6497b2c6e add platformtest: infrastructure for ZFS compatiblity testing 2019-09-14 13:43:46 +02:00
Christian Schwarz
07956c2299 zfs,endpoint: use zfs destroy batch syntax if available
refs #72
2019-09-14 13:43:46 +02:00
Christian Schwarz
e5f944c2f8 zfs: zfsGet: return *ZFSError on exec failure
refs #178
2019-09-07 20:12:46 +02:00
Christian Schwarz
d81a1818d6 endpoint: Receiver: only create placeholders below root_fs
fixes #195
2019-09-07 20:01:15 +02:00
Christian Schwarz
5b97953bfb run golangci-lint and apply suggested fixes 2019-03-27 13:12:26 +01:00
Christian Schwarz
afed762774 format source tree using goimports 2019-03-22 19:41:12 +01:00
Christian Schwarz
644d6f8ffb Merge branch 'problame/simplify-placeholder-property' into master
As became clear in #126 (comment) the current zrepl:placeholder user
property makes it hard to rename datasets because the placeholder status
is determined by computing a hash of the dataset path and comparing it
with the property value.

This PR

- changes zrepl:placeholder to a simple on|off property
- derives property status from the property source (local),avoiding the
  original problem of property inheritance in the original implementation
- provides a migration command that, based on the current zrepl
  configuration, migrates property values to the new on|off schema

refs #150
2019-03-21 21:37:40 +01:00
Christian Schwarz
ab38f24198 zfs: bookmark / replication cursor: handle spaces in ds names correctly
fixes #131
2019-03-21 17:03:26 +01:00
Christian Schwarz
7d9a1b7eae zfs: dry send: handle spaces in dataset names correctly
fixes #131
2019-03-21 17:03:19 +01:00
Christian Schwarz
2f2e6e6a00 receiving side: placeholder as simple on|off property 2019-03-20 20:26:30 +01:00
Christian Schwarz
d50e553ebb handle changes to placeholder state correctly
We assumed that `zfs recv -F FS` would basically replace FS inplace, leaving its children untouched.
That is in fact not the case, it only works if `zfs send -R` is set, which we don't do.

Thus, implement the required functionality manually.

This solves a `zfs recv` error that would occur when a filesystem previously created as placeholder on the receiving side becomes a non-placeholder filesystem (likely due to config change on the sending side):

  zfs send pool1/foo@1 | zfs recv -F pool1/bar
  cannot receive new filesystem stream:
  destination has snapshots (eg. pool1/bar)
  must destroy them to overwrite it
2019-03-13 18:46:04 +01:00
Christian Schwarz
8129ed91f1 zfs + replication: migrate dead zfs/diff_test.go to replication/logic/diff
(and remove the dead code from package zfs)
2019-03-13 16:39:10 +01:00
Christian Schwarz
796c5ad42d rpc rewrite: control RPCs using gRPC + separate RPC for data transfer
transport/ssh: update go-netssh to new version
    => supports CloseWrite and Deadlines
    => build: require Go 1.11 (netssh requires it)
2019-03-13 13:53:48 +01:00
Christian Schwarz
7ab51fad0d zfs: add 'received' property source, handle 'any' source correctly and use 'any' for placeholder FS detection
we want was first noticed in zfs 0.8rc1

Upstream doc PR: https://github.com/zfsonlinux/zfs/pull/8134
2018-11-16 13:07:13 +01:00
Christian Schwarz
3e359aaeda zfs: fixup 6fcf0635a5: broken test 2018-10-19 17:23:00 +02:00
Christian Schwarz
814fec60f0 endpoint + zfs: context cancellation of util.IOCommand instances (send & recv for now) 2018-10-19 16:12:21 +02:00
Christian Schwarz
6fcf0635a5 zfs: generalize dry send information for normal sends and with resume token
This is in preparation for resumable send & recv, thus we just don't use
the ResumeToken field for the time being.
2018-10-18 15:56:28 +02:00
Christian Schwarz
3c06235dca replication + zfs: leave From field instead of To field empty for initial send 2018-10-14 13:06:23 +02:00
Christian Schwarz
074f989547 Merge branch 'replication_rewrite' (in fact it's a 90% rewrite) 2018-10-13 16:26:23 +02:00
Christian Schwarz
fa47667f31 bring back prometheus metrics, with new metrics for replication state machine 2018-09-07 22:22:34 -07:00
Christian Schwarz
1323a30a0c zfs: ability to specify sources for zfsGet
fix use for Placeholder, leave rest as previous behavior
2018-09-05 19:51:06 -07:00
Christian Schwarz
975fdee217 replication & pruning: ditch replicated-property, use bookmark as cursor instead
A bookmark with a well-known name is used to track which version was
last successfully received by the receiver.
The createtxg that can be retrieved from the bookmark using `zfs get` is
used to set the Replicated attribute of each snap on the sender:
If the snap's CreateTXG > the cursor's, it is not yet replicated,
otherwise it has been.

There is an optional config option to change the behvior to
`CreateTXG >= the cursor's`, and the implementation defaults to that.

The reason: While things work just fine with `CreateTXG > the cursor's`,
ZFS does not provide size estimates in a `zfs send` dry run
(see acd2418).
However, to enable the use case of keeping the snapshot only around for
the replication, the config flag exists.
2018-09-05 19:51:06 -07:00
Christian Schwarz
acd2418803 handle DryRun send size estimate errors with bookmarks 2018-09-05 17:41:25 -07:00
Christian Schwarz
4b39a18178 zfs: disable resume token test because it doesn't work in docker 2018-09-04 17:31:46 -07:00
Christian Schwarz
1e27720b99 zfs: skip test with ZFS_BINARY mock (doesn't work in parallel) 2018-09-04 17:02:02 -07:00
Christian Schwarz
4336af295f fixup 22ca80eb7e: scraping regex was broken and potentially mixed with stdout 2018-09-04 14:01:48 -07:00
Christian Schwarz
f0860767f5 zfs: include stderr of command in ZFSError.Error()
Since we don't implement screen-scraping of ZFS output ATM, this is
better than nothing, as user's may be able to figure out what' sthe
problem from the logs / status reports.
2018-09-02 15:46:42 -07:00
Anton Schirg
f387e23214 fix: at least two snapshots were needed to start replication 2018-08-30 19:20:18 +02:00
Anton Schirg
98f3f3dfd8 show expected size of current send
Needs to be changed to send sizes for all planned steps
2018-08-30 12:58:13 +02:00
Anton Schirg
b2f01e454f bug in ZFSListFilesystemVersions? 2018-08-30 12:54:30 +02:00
Christian Schwarz
22ca80eb7e remote snapshot destruction & replication status zfs property 2018-08-30 11:51:47 +02:00
Christian Schwarz
666ead2646 make go vet happy 2018-08-26 14:51:20 +02:00
Christian Schwarz
e30ae972f4 gofmt 2018-08-25 21:30:25 +02:00
Christian Schwarz
1a8d2c5ebe replication: context support and propert closing of stale readers 2018-07-08 23:31:46 +02:00
Christian Schwarz
8cca0a8547 Initial working version
Summary:
* Logging is still bad
* test output in a lot of placed
* FIXMEs every where

Test Plan: None, just review

Differential Revision: https://phabricator.cschwarz.com/D2
2018-06-24 10:44:00 +02:00
Moritz Fago
302d77cf4a zfs: allow spaces in zfs names.
Space is a allowed character in zfs names accoring to
https://github.com/zfsonlinux/zfs/issues/439.
2018-06-20 14:14:04 +02:00
Christian Schwarz
fa6426f803 WIP: zfs: hacky resume token parsing 2018-05-02 21:26:56 +02:00
Christian Schwarz
0764f8824e zfs: prometheus metrics
refs #67
2018-04-05 22:12:25 +02:00
Christian Schwarz
75fd21e454 make generate: stringer was updated and now uses strconv instead of fmt
bd4635fd25 (diff-0415b5286e4cf3e373f349d917e5e039)
2018-04-01 15:30:04 +02:00
Christian Schwarz
3ba3648f0f zfs: use channel as iterator for ZFSList results
The old approach with ZFSList would keep the two-dimensional array of
lines and their fields in memory (for a short time), which could easily
consume 100s of MiB with > 10000 snapshots / bookmarks (see #34)

fixes #61
2018-02-18 13:28:46 +01:00
Christian Schwarz
5f2c14adab zfs: use custom datatype to pass ZFS properties in ZFSSet
refs #55
2018-01-05 18:42:10 +01:00
Christian Schwarz
ed68bffea5 bookmark every snapshot
replication logic already supports bookmarks \o/

refs #34
2017-11-13 10:59:46 +01:00
Christian Schwarz
8934d7cb64 zfs: was not returning error on invalid dataset path 2017-09-30 16:39:15 +02:00
Christian Schwarz
0cbee78b40 fix unreachable code & missing stringer-generated code 2017-09-30 16:31:55 +02:00
Christian Schwarz
4e45b4090b pull log output: optimize to be readable by humans 2017-08-06 18:28:05 +02:00
Christian Schwarz
cba083cadf Make zfs.DatasetPath json.Marshaler and json.Unmarshaler
Had to resort to using pointers to zfs.DatasetPath everywhere... Should
find a better solution for that.
2017-08-06 16:22:15 +02:00
Christian Schwarz
2ce07c9342 rework filters & mappings
config defines a single datastructure that can act both as a Map and as a Filter
(DatasetMapFilter)

Cleanup wildcard syntax along the way (also changes semantics).
2017-08-06 16:21:54 +02:00
Christian Schwarz
4732fdd4cc Implement placeholder filesystems.
Note the docs on the placeholder user property introduced with this
commit. The solution is not really satisfying but couldn't think of a
better one OTOMH
2017-08-06 16:21:54 +02:00
Christian Schwarz
9ab6f18f82 zfs: fix/update tests for diffs for createtxg & guid 2017-07-09 00:08:50 +02:00
Christian Schwarz
4b373fbd95 zfs & replication: explicit conflict types for FilesystemDiff + handling in repl 2017-07-08 13:13:16 +02:00