Christian Schwarz
3c06235dca
replication + zfs: leave From field instead of To field empty for initial send
2018-10-14 13:06:23 +02:00
Christian Schwarz
074f989547
Merge branch 'replication_rewrite' (in fact it's a 90% rewrite)
2018-10-13 16:26:23 +02:00
Christian Schwarz
fa47667f31
bring back prometheus metrics, with new metrics for replication state machine
2018-09-07 22:22:34 -07:00
Christian Schwarz
1323a30a0c
zfs: ability to specify sources for zfsGet
...
fix use for Placeholder, leave rest as previous behavior
2018-09-05 19:51:06 -07:00
Christian Schwarz
975fdee217
replication & pruning: ditch replicated-property, use bookmark as cursor instead
...
A bookmark with a well-known name is used to track which version was
last successfully received by the receiver.
The createtxg that can be retrieved from the bookmark using `zfs get` is
used to set the Replicated attribute of each snap on the sender:
If the snap's CreateTXG > the cursor's, it is not yet replicated,
otherwise it has been.
There is an optional config option to change the behvior to
`CreateTXG >= the cursor's`, and the implementation defaults to that.
The reason: While things work just fine with `CreateTXG > the cursor's`,
ZFS does not provide size estimates in a `zfs send` dry run
(see acd2418
2018-09-05 19:51:06 -07:00
Christian Schwarz
acd2418803
handle DryRun send size estimate errors with bookmarks
2018-09-05 17:41:25 -07:00
Christian Schwarz
4b39a18178
zfs: disable resume token test because it doesn't work in docker
2018-09-04 17:31:46 -07:00
Christian Schwarz
1e27720b99
zfs: skip test with ZFS_BINARY mock (doesn't work in parallel)
2018-09-04 17:02:02 -07:00
Christian Schwarz
4336af295f
fixup 22ca80eb7e: scraping regex was broken and potentially mixed with stdout
2018-09-04 14:01:48 -07:00
Christian Schwarz
f0860767f5
zfs: include stderr of command in ZFSError.Error()
...
Since we don't implement screen-scraping of ZFS output ATM, this is
better than nothing, as user's may be able to figure out what' sthe
problem from the logs / status reports.
2018-09-02 15:46:42 -07:00
Anton Schirg
f387e23214
fix: at least two snapshots were needed to start replication
2018-08-30 19:20:18 +02:00
Anton Schirg
98f3f3dfd8
show expected size of current send
...
Needs to be changed to send sizes for all planned steps
2018-08-30 12:58:13 +02:00
Anton Schirg
b2f01e454f
bug in ZFSListFilesystemVersions?
2018-08-30 12:54:30 +02:00
Christian Schwarz
22ca80eb7e
remote snapshot destruction & replication status zfs property
2018-08-30 11:51:47 +02:00
Christian Schwarz
666ead2646
make go vet happy
2018-08-26 14:51:20 +02:00
Christian Schwarz
e30ae972f4
gofmt
2018-08-25 21:30:25 +02:00
Christian Schwarz
1a8d2c5ebe
replication: context support and propert closing of stale readers
2018-07-08 23:31:46 +02:00
Christian Schwarz
8cca0a8547
Initial working version
...
Summary:
* Logging is still bad
* test output in a lot of placed
* FIXMEs every where
Test Plan: None, just review
Differential Revision: https://phabricator.cschwarz.com/D2
2018-06-24 10:44:00 +02:00
Moritz Fago
302d77cf4a
zfs: allow spaces in zfs names.
...
Space is a allowed character in zfs names accoring to
https://github.com/zfsonlinux/zfs/issues/439 .
2018-06-20 14:14:04 +02:00
Christian Schwarz
fa6426f803
WIP: zfs: hacky resume token parsing
2018-05-02 21:26:56 +02:00
Christian Schwarz
0764f8824e
zfs: prometheus metrics
...
refs #67
2018-04-05 22:12:25 +02:00
Christian Schwarz
75fd21e454
make generate: stringer was updated and now uses strconv instead of fmt
...
bd4635fd25 (diff-0415b5286e4cf3e373f349d917e5e039)
2018-04-01 15:30:04 +02:00
Christian Schwarz
3ba3648f0f
zfs: use channel as iterator for ZFSList results
...
The old approach with ZFSList would keep the two-dimensional array of
lines and their fields in memory (for a short time), which could easily
consume 100s of MiB with > 10000 snapshots / bookmarks (see #34 )
fixes #61
2018-02-18 13:28:46 +01:00
Christian Schwarz
5f2c14adab
zfs: use custom datatype to pass ZFS properties in ZFSSet
...
refs #55
2018-01-05 18:42:10 +01:00
Christian Schwarz
ed68bffea5
bookmark every snapshot
...
replication logic already supports bookmarks \o/
refs #34
2017-11-13 10:59:46 +01:00
Christian Schwarz
8934d7cb64
zfs: was not returning error on invalid dataset path
2017-09-30 16:39:15 +02:00
Christian Schwarz
0cbee78b40
fix unreachable code & missing stringer-generated code
2017-09-30 16:31:55 +02:00
Christian Schwarz
4e45b4090b
pull log output: optimize to be readable by humans
2017-08-06 18:28:05 +02:00
Christian Schwarz
cba083cadf
Make zfs.DatasetPath json.Marshaler and json.Unmarshaler
...
Had to resort to using pointers to zfs.DatasetPath everywhere... Should
find a better solution for that.
2017-08-06 16:22:15 +02:00
Christian Schwarz
2ce07c9342
rework filters & mappings
...
config defines a single datastructure that can act both as a Map and as a Filter
(DatasetMapFilter)
Cleanup wildcard syntax along the way (also changes semantics).
2017-08-06 16:21:54 +02:00
Christian Schwarz
4732fdd4cc
Implement placeholder filesystems.
...
Note the docs on the placeholder user property introduced with this
commit. The solution is not really satisfying but couldn't think of a
better one OTOMH
2017-08-06 16:21:54 +02:00
Christian Schwarz
9ab6f18f82
zfs: fix/update tests for diffs for createtxg & guid
2017-07-09 00:08:50 +02:00
Christian Schwarz
4b373fbd95
zfs & replication: explicit conflict types for FilesystemDiff + handling in repl
2017-07-08 13:13:16 +02:00
Christian Schwarz
655b3ab55f
implement automatic snapshotting feature
2017-07-02 00:02:33 +02:00
Christian Schwarz
8c8a6ee905
implement snapshot pruning feature
2017-07-02 00:02:33 +02:00
Christian Schwarz
c7f140a00f
zfs: support destroy
2017-07-01 23:19:31 +02:00
Christian Schwarz
c22190e981
zfs: extract filesystem version code to separate file & add filtering support
2017-07-01 23:19:31 +02:00
Christian Schwarz
d8adce6110
zfs: Support foo/bar/* globs
2017-05-20 19:50:24 +02:00
Christian Schwarz
ee570bb060
refactor: consolidate ForkReader-like implementations to IOCommand
2017-05-14 12:27:15 +02:00
Christian Schwarz
dd6dd60e98
Simplify "fork then io.Reader" abstractions
2017-05-13 15:23:37 +02:00
Christian Schwarz
54778c0374
zfs: remove erronous output from ForkReader
2017-05-13 15:23:28 +02:00
Christian Schwarz
c762502f6e
zfs: implement DatasetPath top down walk
2017-05-13 15:23:28 +02:00
Christian Schwarz
996be3704d
zfs: implement ZFSListFilesystemExists()
2017-05-07 12:26:41 +02:00
Christian Schwarz
6a420e872e
zfs: implement ZFSSet
2017-05-07 12:23:12 +02:00
Christian Schwarz
d37bdc3d7a
zfs: implement ZFSRecv
2017-05-07 12:22:57 +02:00
Christian Schwarz
030bd7affe
zfs: implement ZFSSend
2017-05-07 12:20:56 +02:00
Christian Schwarz
1a92717894
zfs: FilesytemDiff: len(IncrementalPath) < 2 means same most recent version
...
bonus: style + typo fixes
2017-05-07 12:20:56 +02:00
Christian Schwarz
d0c0e2510b
zfs: FilesystemDiff: support empty left list = no common ancestor
2017-05-07 12:20:56 +02:00
Christian Schwarz
cd8796aed4
rpc: Initial|IncrementalTransferRequest transfer zfs data structures
2017-05-07 12:20:56 +02:00
Christian Schwarz
44d20ac256
zfs: ZFSListMapping fix broken logic
2017-05-06 23:45:33 +02:00
Christian Schwarz
aa696dd0ce
zfs: use createtxg and guid properties for FilesystemVersion
2017-05-06 10:58:01 +02:00
Christian Schwarz
f46fb2f34f
zfs: filesystem versioning and diffing
2017-05-01 20:35:49 +02:00
Christian Schwarz
644d3dd06b
zfs: public ZFSList() function + consolidation of mapping code
2017-05-01 20:35:04 +02:00
Christian Schwarz
28f0b90c25
zfs: check for forbidden dataset path characters
2017-05-01 20:09:35 +02:00
Christian Schwarz
ac5a911c64
zfs: glob mapping for complete wildcard ("*")
2017-04-30 23:47:11 +02:00
Christian Schwarz
d9ecfc8eb4
Gofmt megacommit.
2017-04-26 20:29:54 +02:00
Christian Schwarz
00231ecb73
Implement config parser.
2017-04-26 19:57:40 +02:00
Christian Schwarz
40f3b530e1
Implement DatasetMapping + basic ZFS list functionality.
2017-04-26 17:39:16 +02:00
Christian Schwarz
123becbd22
Interface wireframe
2017-04-14 19:26:32 +02:00
