extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-25 18:04:58 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
794 Commits 48 Branches 23 Tags 3.6 MiB
5b52e5e331
Commit Graph

53 Commits

Author SHA1 Message Date
Christian Schwarz
6ebd9f1037 zfs: recv: fix deadlock if streamCopier returns io.EOF 
Close the write end of the pipe
* before we start waiting on the error channels
* and after everything from streamCopier has been written to the pipe
 2020-02-14 21:42:03 +01:00
Christian Schwarz
99ab16d7be zfs: send: improve error reporting by capturing stderr 2020-02-14 21:42:03 +01:00
Juergen Hoetzel
d35e2400b2 transport/{TCP,TLS}: optional IP_FREEBIND / IP_BINDANY bind socketops 
Allows to bind to an address even if it is not actually (yet or ever)
configured. Fixes #238

Rationale:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/#whatdoesthismeanformeadeveloper
 2020-01-04 17:21:48 +01:00
Christian Schwarz
b9933f6cb2 platformtest: add zfsGet bookmark handling & replicationCursor tests 
This encodes the observation made in issue #230 :
In the ZFS version shipped in Ubuntu 16.04 where
`zfs get someprop a#bookmark` does not work.
 2019-10-14 17:54:14 +02:00
Christian Schwarz
0ba4b5eda6 zfs: helper for ZFSGet guid and createtxg 2019-10-14 17:54:14 +02:00
Christian Schwarz
a6497b2c6e add platformtest: infrastructure for ZFS compatiblity testing 2019-09-14 13:43:46 +02:00
Christian Schwarz
07956c2299 zfs,endpoint: use zfs destroy batch syntax if available 
refs #72
 2019-09-14 13:43:46 +02:00
Christian Schwarz
e5f944c2f8 zfs: zfsGet: return *ZFSError on exec failure 
refs #178
 2019-09-07 20:12:46 +02:00
Christian Schwarz
5b97953bfb run golangci-lint and apply suggested fixes 2019-03-27 13:12:26 +01:00
Christian Schwarz
afed762774 format source tree using goimports 2019-03-22 19:41:12 +01:00
Christian Schwarz
ab38f24198 zfs: bookmark / replication cursor: handle spaces in ds names correctly 
fixes #131
 2019-03-21 17:03:26 +01:00
Christian Schwarz
7d9a1b7eae zfs: dry send: handle spaces in dataset names correctly 
fixes #131
 2019-03-21 17:03:19 +01:00
Christian Schwarz
d50e553ebb handle changes to placeholder state correctly 
We assumed that `zfs recv -F FS` would basically replace FS inplace, leaving its children untouched.
That is in fact not the case, it only works if `zfs send -R` is set, which we don't do.

Thus, implement the required functionality manually.

This solves a `zfs recv` error that would occur when a filesystem previously created as placeholder on the receiving side becomes a non-placeholder filesystem (likely due to config change on the sending side):

  zfs send pool1/foo@1 | zfs recv -F pool1/bar
  cannot receive new filesystem stream:
  destination has snapshots (eg. pool1/bar)
  must destroy them to overwrite it
 2019-03-13 18:46:04 +01:00
Christian Schwarz
796c5ad42d rpc rewrite: control RPCs using gRPC + separate RPC for data transfer 
transport/ssh: update go-netssh to new version
    => supports CloseWrite and Deadlines
    => build: require Go 1.11 (netssh requires it)
 2019-03-13 13:53:48 +01:00
Christian Schwarz
7ab51fad0d zfs: add 'received' property source, handle 'any' source correctly and use 'any' for placeholder FS detection 
we want was first noticed in zfs 0.8rc1

Upstream doc PR: https://github.com/zfsonlinux/zfs/pull/8134
 2018-11-16 13:07:13 +01:00
Christian Schwarz
814fec60f0 endpoint + zfs: context cancellation of util.IOCommand instances (send & recv for now) 2018-10-19 16:12:21 +02:00
Christian Schwarz
6fcf0635a5 zfs: generalize dry send information for normal sends and with resume token 
This is in preparation for resumable send & recv, thus we just don't use
the ResumeToken field for the time being.
 2018-10-18 15:56:28 +02:00
Christian Schwarz
3c06235dca replication + zfs: leave From field instead of To field empty for initial send 2018-10-14 13:06:23 +02:00
Christian Schwarz
074f989547 Merge branch 'replication_rewrite' (in fact it's a 90% rewrite) 2018-10-13 16:26:23 +02:00
Christian Schwarz
fa47667f31 bring back prometheus metrics, with new metrics for replication state machine 2018-09-07 22:22:34 -07:00
Christian Schwarz
1323a30a0c zfs: ability to specify sources for zfsGet 
fix use for Placeholder, leave rest as previous behavior
 2018-09-05 19:51:06 -07:00
Christian Schwarz
975fdee217 replication & pruning: ditch replicated-property, use bookmark as cursor instead 
A bookmark with a well-known name is used to track which version was
last successfully received by the receiver.
The createtxg that can be retrieved from the bookmark using `zfs get` is
used to set the Replicated attribute of each snap on the sender:
If the snap's CreateTXG > the cursor's, it is not yet replicated,
otherwise it has been.

There is an optional config option to change the behvior to
`CreateTXG >= the cursor's`, and the implementation defaults to that.

The reason: While things work just fine with `CreateTXG > the cursor's`,
ZFS does not provide size estimates in a `zfs send` dry run
(see acd2418).
However, to enable the use case of keeping the snapshot only around for
the replication, the config flag exists.
 2018-09-05 19:51:06 -07:00
Christian Schwarz
acd2418803 handle DryRun send size estimate errors with bookmarks 2018-09-05 17:41:25 -07:00
Christian Schwarz
4336af295f fixup 22ca80eb7e: scraping regex was broken and potentially mixed with stdout 2018-09-04 14:01:48 -07:00
Christian Schwarz
f0860767f5 zfs: include stderr of command in ZFSError.Error() 
Since we don't implement screen-scraping of ZFS output ATM, this is
better than nothing, as user's may be able to figure out what' sthe
problem from the logs / status reports.
 2018-09-02 15:46:42 -07:00
Anton Schirg
98f3f3dfd8 show expected size of current send 
Needs to be changed to send sizes for all planned steps
 2018-08-30 12:58:13 +02:00
Christian Schwarz
22ca80eb7e remote snapshot destruction & replication status zfs property 2018-08-30 11:51:47 +02:00
Christian Schwarz
1a8d2c5ebe replication: context support and propert closing of stale readers 2018-07-08 23:31:46 +02:00
Christian Schwarz
8cca0a8547 Initial working version 
Summary:
* Logging is still bad
* test output in a lot of placed
* FIXMEs every where

Test Plan: None, just review

Differential Revision: https://phabricator.cschwarz.com/D2
 2018-06-24 10:44:00 +02:00
Moritz Fago
302d77cf4a zfs: allow spaces in zfs names. 
Space is a allowed character in zfs names accoring to
https://github.com/zfsonlinux/zfs/issues/439.
 2018-06-20 14:14:04 +02:00
Christian Schwarz
0764f8824e zfs: prometheus metrics 
refs #67
 2018-04-05 22:12:25 +02:00
Christian Schwarz
3ba3648f0f zfs: use channel as iterator for ZFSList results 
The old approach with ZFSList would keep the two-dimensional array of
lines and their fields in memory (for a short time), which could easily
consume 100s of MiB with > 10000 snapshots / bookmarks (see #34)

fixes #61
 2018-02-18 13:28:46 +01:00
Christian Schwarz
5f2c14adab zfs: use custom datatype to pass ZFS properties in ZFSSet 
refs #55
 2018-01-05 18:42:10 +01:00
Christian Schwarz
ed68bffea5 bookmark every snapshot 
replication logic already supports bookmarks \o/

refs #34
 2017-11-13 10:59:46 +01:00
Christian Schwarz
cba083cadf Make zfs.DatasetPath json.Marshaler and json.Unmarshaler 
Had to resort to using pointers to zfs.DatasetPath everywhere... Should
find a better solution for that.
 2017-08-06 16:22:15 +02:00
Christian Schwarz
2ce07c9342 rework filters & mappings 
config defines a single datastructure that can act both as a Map and as a Filter
(DatasetMapFilter)

Cleanup wildcard syntax along the way (also changes semantics).
 2017-08-06 16:21:54 +02:00
Christian Schwarz
655b3ab55f implement automatic snapshotting feature 2017-07-02 00:02:33 +02:00
Christian Schwarz
c7f140a00f zfs: support destroy 2017-07-01 23:19:31 +02:00
Christian Schwarz
ee570bb060 refactor: consolidate ForkReader-like implementations to IOCommand 2017-05-14 12:27:15 +02:00
Christian Schwarz
dd6dd60e98 Simplify "fork then io.Reader" abstractions 2017-05-13 15:23:37 +02:00
Christian Schwarz
54778c0374 zfs: remove erronous output from ForkReader 2017-05-13 15:23:28 +02:00
Christian Schwarz
6a420e872e zfs: implement ZFSSet 2017-05-07 12:23:12 +02:00
Christian Schwarz
d37bdc3d7a zfs: implement ZFSRecv 2017-05-07 12:22:57 +02:00
Christian Schwarz
030bd7affe zfs: implement ZFSSend 2017-05-07 12:20:56 +02:00
Christian Schwarz
cd8796aed4 rpc: Initial|IncrementalTransferRequest transfer zfs data structures 2017-05-07 12:20:56 +02:00
Christian Schwarz
aa696dd0ce zfs: use createtxg and guid properties for FilesystemVersion 2017-05-06 10:58:01 +02:00
Christian Schwarz
644d3dd06b zfs: public ZFSList() function + consolidation of mapping code 2017-05-01 20:35:04 +02:00
Christian Schwarz
28f0b90c25 zfs: check for forbidden dataset path characters 2017-05-01 20:09:35 +02:00
Christian Schwarz
ac5a911c64 zfs: glob mapping for complete wildcard ("*") 2017-04-30 23:47:11 +02:00
Christian Schwarz
d9ecfc8eb4 Gofmt megacommit. 2017-04-26 20:29:54 +02:00