extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-22 08:23:50 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,088 Commits 48 Branches 23 Tags 3.6 MiB
problame/release-circleci-binaries
Commit Graph

121 Commits

Author SHA1 Message Date
Christian Schwarz
afed762774 format source tree using goimports 2019-03-22 19:41:12 +01:00
Christian Schwarz
644d6f8ffb Merge branch 'problame/simplify-placeholder-property' into master 
As became clear in #126 (comment) the current zrepl:placeholder user
property makes it hard to rename datasets because the placeholder status
is determined by computing a hash of the dataset path and comparing it
with the property value.

This PR

- changes zrepl:placeholder to a simple on|off property
- derives property status from the property source (local),avoiding the
  original problem of property inheritance in the original implementation
- provides a migration command that, based on the current zrepl
  configuration, migrates property values to the new on|off schema

refs #150
 2019-03-21 21:37:40 +01:00
Christian Schwarz
ab38f24198 zfs: bookmark / replication cursor: handle spaces in ds names correctly 
fixes #131
 2019-03-21 17:03:26 +01:00
Christian Schwarz
7d9a1b7eae zfs: dry send: handle spaces in dataset names correctly 
fixes #131
 2019-03-21 17:03:19 +01:00
Christian Schwarz
2f2e6e6a00 receiving side: placeholder as simple on|off property 2019-03-20 20:26:30 +01:00
Christian Schwarz
d50e553ebb handle changes to placeholder state correctly 
We assumed that `zfs recv -F FS` would basically replace FS inplace, leaving its children untouched.
That is in fact not the case, it only works if `zfs send -R` is set, which we don't do.

Thus, implement the required functionality manually.

This solves a `zfs recv` error that would occur when a filesystem previously created as placeholder on the receiving side becomes a non-placeholder filesystem (likely due to config change on the sending side):

  zfs send pool1/foo@1 | zfs recv -F pool1/bar
  cannot receive new filesystem stream:
  destination has snapshots (eg. pool1/bar)
  must destroy them to overwrite it
 2019-03-13 18:46:04 +01:00
Christian Schwarz
8129ed91f1 zfs + replication: migrate dead zfs/diff_test.go to replication/logic/diff 
(and remove the dead code from package zfs)
 2019-03-13 16:39:10 +01:00
Christian Schwarz
796c5ad42d rpc rewrite: control RPCs using gRPC + separate RPC for data transfer 
transport/ssh: update go-netssh to new version
    => supports CloseWrite and Deadlines
    => build: require Go 1.11 (netssh requires it)
 2019-03-13 13:53:48 +01:00
Christian Schwarz
7ab51fad0d zfs: add 'received' property source, handle 'any' source correctly and use 'any' for placeholder FS detection 
we want was first noticed in zfs 0.8rc1

Upstream doc PR: https://github.com/zfsonlinux/zfs/pull/8134
 2018-11-16 13:07:13 +01:00
Christian Schwarz
3e359aaeda zfs: fixup 6fcf0635a5: broken test 2018-10-19 17:23:00 +02:00
Christian Schwarz
814fec60f0 endpoint + zfs: context cancellation of util.IOCommand instances (send & recv for now) 2018-10-19 16:12:21 +02:00
Christian Schwarz
6fcf0635a5 zfs: generalize dry send information for normal sends and with resume token 
This is in preparation for resumable send & recv, thus we just don't use
the ResumeToken field for the time being.
 2018-10-18 15:56:28 +02:00
Christian Schwarz
3c06235dca replication + zfs: leave From field instead of To field empty for initial send 2018-10-14 13:06:23 +02:00
Christian Schwarz
074f989547 Merge branch 'replication_rewrite' (in fact it's a 90% rewrite) 2018-10-13 16:26:23 +02:00
Christian Schwarz
fa47667f31 bring back prometheus metrics, with new metrics for replication state machine 2018-09-07 22:22:34 -07:00
Christian Schwarz
1323a30a0c zfs: ability to specify sources for zfsGet 
fix use for Placeholder, leave rest as previous behavior
 2018-09-05 19:51:06 -07:00
Christian Schwarz
975fdee217 replication & pruning: ditch replicated-property, use bookmark as cursor instead 
A bookmark with a well-known name is used to track which version was
last successfully received by the receiver.
The createtxg that can be retrieved from the bookmark using `zfs get` is
used to set the Replicated attribute of each snap on the sender:
If the snap's CreateTXG > the cursor's, it is not yet replicated,
otherwise it has been.

There is an optional config option to change the behvior to
`CreateTXG >= the cursor's`, and the implementation defaults to that.

The reason: While things work just fine with `CreateTXG > the cursor's`,
ZFS does not provide size estimates in a `zfs send` dry run
(see acd2418).
However, to enable the use case of keeping the snapshot only around for
the replication, the config flag exists.
 2018-09-05 19:51:06 -07:00
Christian Schwarz
acd2418803 handle DryRun send size estimate errors with bookmarks 2018-09-05 17:41:25 -07:00
Christian Schwarz
4b39a18178 zfs: disable resume token test because it doesn't work in docker 2018-09-04 17:31:46 -07:00
Christian Schwarz
1e27720b99 zfs: skip test with ZFS_BINARY mock (doesn't work in parallel) 2018-09-04 17:02:02 -07:00
Christian Schwarz
4336af295f fixup 22ca80eb7e: scraping regex was broken and potentially mixed with stdout 2018-09-04 14:01:48 -07:00
Christian Schwarz
f0860767f5 zfs: include stderr of command in ZFSError.Error() 
Since we don't implement screen-scraping of ZFS output ATM, this is
better than nothing, as user's may be able to figure out what' sthe
problem from the logs / status reports.
 2018-09-02 15:46:42 -07:00
Anton Schirg
f387e23214 fix: at least two snapshots were needed to start replication 2018-08-30 19:20:18 +02:00
Anton Schirg
98f3f3dfd8 show expected size of current send 
Needs to be changed to send sizes for all planned steps
 2018-08-30 12:58:13 +02:00
Anton Schirg
b2f01e454f bug in ZFSListFilesystemVersions? 2018-08-30 12:54:30 +02:00
Christian Schwarz
22ca80eb7e remote snapshot destruction & replication status zfs property 2018-08-30 11:51:47 +02:00
Christian Schwarz
666ead2646 make go vet happy 2018-08-26 14:51:20 +02:00
Christian Schwarz
e30ae972f4 gofmt 2018-08-25 21:30:25 +02:00
Christian Schwarz
1a8d2c5ebe replication: context support and propert closing of stale readers 2018-07-08 23:31:46 +02:00
Christian Schwarz
8cca0a8547 Initial working version 
Summary:
* Logging is still bad
* test output in a lot of placed
* FIXMEs every where

Test Plan: None, just review

Differential Revision: https://phabricator.cschwarz.com/D2
 2018-06-24 10:44:00 +02:00
Moritz Fago
302d77cf4a zfs: allow spaces in zfs names. 
Space is a allowed character in zfs names accoring to
https://github.com/zfsonlinux/zfs/issues/439.
 2018-06-20 14:14:04 +02:00
Christian Schwarz
fa6426f803 WIP: zfs: hacky resume token parsing 2018-05-02 21:26:56 +02:00
Christian Schwarz
0764f8824e zfs: prometheus metrics 
refs #67
 2018-04-05 22:12:25 +02:00
Christian Schwarz
75fd21e454 make generate: stringer was updated and now uses strconv instead of fmt 
bd4635fd25 (diff-0415b5286e4cf3e373f349d917e5e039)
 2018-04-01 15:30:04 +02:00
Christian Schwarz
3ba3648f0f zfs: use channel as iterator for ZFSList results 
The old approach with ZFSList would keep the two-dimensional array of
lines and their fields in memory (for a short time), which could easily
consume 100s of MiB with > 10000 snapshots / bookmarks (see #34)

fixes #61
 2018-02-18 13:28:46 +01:00
Christian Schwarz
5f2c14adab zfs: use custom datatype to pass ZFS properties in ZFSSet 
refs #55
 2018-01-05 18:42:10 +01:00
Christian Schwarz
ed68bffea5 bookmark every snapshot 
replication logic already supports bookmarks \o/

refs #34
 2017-11-13 10:59:46 +01:00
Christian Schwarz
8934d7cb64 zfs: was not returning error on invalid dataset path 2017-09-30 16:39:15 +02:00
Christian Schwarz
0cbee78b40 fix unreachable code & missing stringer-generated code 2017-09-30 16:31:55 +02:00
Christian Schwarz
4e45b4090b pull log output: optimize to be readable by humans 2017-08-06 18:28:05 +02:00
Christian Schwarz
cba083cadf Make zfs.DatasetPath json.Marshaler and json.Unmarshaler 
Had to resort to using pointers to zfs.DatasetPath everywhere... Should
find a better solution for that.
 2017-08-06 16:22:15 +02:00
Christian Schwarz
2ce07c9342 rework filters & mappings 
config defines a single datastructure that can act both as a Map and as a Filter
(DatasetMapFilter)

Cleanup wildcard syntax along the way (also changes semantics).
 2017-08-06 16:21:54 +02:00
Christian Schwarz
4732fdd4cc Implement placeholder filesystems. 
Note the docs on the placeholder user property introduced with this
commit. The solution is not really satisfying but couldn't think of a
better one OTOMH
 2017-08-06 16:21:54 +02:00
Christian Schwarz
9ab6f18f82 zfs: fix/update tests for diffs for createtxg & guid 2017-07-09 00:08:50 +02:00
Christian Schwarz
4b373fbd95 zfs & replication: explicit conflict types for FilesystemDiff + handling in repl 2017-07-08 13:13:16 +02:00
Christian Schwarz
655b3ab55f implement automatic snapshotting feature 2017-07-02 00:02:33 +02:00
Christian Schwarz
8c8a6ee905 implement snapshot pruning feature 2017-07-02 00:02:33 +02:00
Christian Schwarz
c7f140a00f zfs: support destroy 2017-07-01 23:19:31 +02:00
Christian Schwarz
c22190e981 zfs: extract filesystem version code to separate file & add filtering support 2017-07-01 23:19:31 +02:00
Christian Schwarz
d8adce6110 zfs: Support foo/bar/* globs 2017-05-20 19:50:24 +02:00
Christian Schwarz
ee570bb060 refactor: consolidate ForkReader-like implementations to IOCommand 2017-05-14 12:27:15 +02:00
Christian Schwarz
dd6dd60e98 Simplify "fork then io.Reader" abstractions 2017-05-13 15:23:37 +02:00
Christian Schwarz
54778c0374 zfs: remove erronous output from ForkReader 2017-05-13 15:23:28 +02:00
Christian Schwarz
c762502f6e zfs: implement DatasetPath top down walk 2017-05-13 15:23:28 +02:00
Christian Schwarz
996be3704d zfs: implement ZFSListFilesystemExists() 2017-05-07 12:26:41 +02:00
Christian Schwarz
6a420e872e zfs: implement ZFSSet 2017-05-07 12:23:12 +02:00
Christian Schwarz
d37bdc3d7a zfs: implement ZFSRecv 2017-05-07 12:22:57 +02:00
Christian Schwarz
030bd7affe zfs: implement ZFSSend 2017-05-07 12:20:56 +02:00
Christian Schwarz
1a92717894 zfs: FilesytemDiff: len(IncrementalPath) < 2 means same most recent version 
bonus: style + typo fixes
 2017-05-07 12:20:56 +02:00
Christian Schwarz
d0c0e2510b zfs: FilesystemDiff: support empty left list = no common ancestor 2017-05-07 12:20:56 +02:00
Christian Schwarz
cd8796aed4 rpc: Initial|IncrementalTransferRequest transfer zfs data structures 2017-05-07 12:20:56 +02:00
Christian Schwarz
44d20ac256 zfs: ZFSListMapping fix broken logic 2017-05-06 23:45:33 +02:00
Christian Schwarz
aa696dd0ce zfs: use createtxg and guid properties for FilesystemVersion 2017-05-06 10:58:01 +02:00
Christian Schwarz
f46fb2f34f zfs: filesystem versioning and diffing 2017-05-01 20:35:49 +02:00
Christian Schwarz
644d3dd06b zfs: public ZFSList() function + consolidation of mapping code 2017-05-01 20:35:04 +02:00
Christian Schwarz
28f0b90c25 zfs: check for forbidden dataset path characters 2017-05-01 20:09:35 +02:00
Christian Schwarz
ac5a911c64 zfs: glob mapping for complete wildcard ("*") 2017-04-30 23:47:11 +02:00
Christian Schwarz
d9ecfc8eb4 Gofmt megacommit. 2017-04-26 20:29:54 +02:00
Christian Schwarz
00231ecb73 Implement config parser. 2017-04-26 19:57:40 +02:00
Christian Schwarz
40f3b530e1 Implement DatasetMapping + basic ZFS list functionality. 2017-04-26 17:39:16 +02:00
Christian Schwarz
123becbd22 Interface wireframe 2017-04-14 19:26:32 +02:00