extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-24 17:35:01 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
problame/zync
zrepl/go.mod

47 lines
2.3 KiB
Modula-2
Raw Permalink Normal View History

build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
module github.com/zrepl/zrepl
go 1.12
require (
github.com/fatih/color v1.7.0
github.com/gdamore/tcell v1.2.0
[#307] add package trace, integrate it with logging, and adopt it throughout zrepl package trace: - introduce the concept of tasks and spans, tracked as linked list within ctx - see package-level docs for an overview of the concepts - **main feature 1**: unique stack of task and span IDs - makes it easy to follow a series of log entries in concurrent code - **main feature 2**: ability to produce a chrome://tracing-compatible trace file - either via an env variable or a `zrepl pprof` subcommand - this is not a CPU profile, we already have go pprof for that - but it is very useful to visually inspect where the replication / snapshotter / pruner spends its time ( fixes #307 ) usage in package daemon/logging: - goal: every log entry should have a trace field with the ID stack from package trace - make `logging.GetLogger(ctx, Subsys)` the authoritative `logger.Logger` factory function - the context carries a linked list of injected fields which `logging.GetLogger` adds to the logger it returns - `logging.GetLogger` also uses package `trace` to get the task-and-span-stack and injects it into the returned logger's fields
2020-04-11 15:49:41 +02:00
github.com/gitchander/permutation v0.0.0-20181107151852-9e56b92e9909
build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy tweaked go mod tidy: see comment in go.mod
2019-10-26 13:55:03 +02:00
github.com/go-logfmt/logfmt v0.4.0
pre- and post-snapshot hooks * stack-based execution model, documented in documentation * circbuf for capturing hook output * built-in hooks for postgres and mysql * refactor docs, too much info on the jobs page, too difficult to discover snapshotting & hooks Co-authored-by: Ross Williams <ross@ross-williams.net> Co-authored-by: Christian Schwarz <me@cschwarz.com> fixes #74
2019-07-26 21:12:21 +02:00
github.com/go-sql-driver/mysql v1.4.1-0.20190907122137-b2c03bcae3d4
build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy tweaked go mod tidy: see comment in go.mod
2019-10-26 13:55:03 +02:00
github.com/golang/protobuf v1.3.2
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
github.com/google/uuid v1.1.1
experiment: zync: an rsync-like tool for ZFS datasets (re-uses zrepl abstractions) Local sync and sync over SSH seems to work. The sshdirect transport might be interesting / re-usable for zrepl as well, although we'd still need some kind of locking in that case.
2021-01-14 01:31:08 +01:00
github.com/hashicorp/yamux v0.0.0-20200609203250-aecfd211c9ce
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
github.com/jinzhu/copier v0.0.0-20170922082739-db4671f3a9b8
github.com/kr/pretty v0.1.0
pre- and post-snapshot hooks * stack-based execution model, documented in documentation * circbuf for capturing hook output * built-in hooks for postgres and mysql * refactor docs, too much info on the jobs page, too difficult to discover snapshotting & hooks Co-authored-by: Ross Williams <ross@ross-williams.net> Co-authored-by: Christian Schwarz <me@cschwarz.com> fixes #74
2019-07-26 21:12:21 +02:00
github.com/lib/pq v1.2.0
transports/ssh: bump go-netssh to improve dial errors from go-netssh changelog: dial: better error handling if ssh command exits with non-zero exit status SSHError.Error() relied on go-rwccmd behavior of returning io.EOF if the ssh binary exited with status code 0. We no longe ruse go-rwccmd => capture Stderr ourselves using zrepl's circlog (depending on zrepl is not pretty, but since this package is supposedly only used by zrepl ATM, this is fine) refs https://github.com/zrepl/zrepl/issues/237
2019-10-26 14:32:07 +02:00
github.com/mattn/go-colorable v0.1.4 // indirect
github.com/mattn/go-isatty v0.0.8
build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy tweaked go mod tidy: see comment in go.mod
2019-10-26 13:55:03 +02:00
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // // go1.12 mod tidy adds this dependency as 'indirect', but go1.13 mod tidy removes it if the trailing comment is 'indirect' => add this comment to make the build work without changing go.mod on both go1.12 and go1.13
github.com/modern-go/reflect2 v1.0.1 // go1.12 mod tidy adds this dependency as 'indirect', but go1.13 mod tidy removes it if the trailing comment is 'indirect' => add this comment to make the build work without changing go.mod on both go1.12 and go1.13
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
github.com/montanaflynn/stats v0.5.0
transports/ssh: bump go-netssh to improve dial errors from go-netssh changelog: dial: better error handling if ssh command exits with non-zero exit status SSHError.Error() relied on go-rwccmd behavior of returning io.EOF if the ssh binary exited with status code 0. We no longe ruse go-rwccmd => capture Stderr ourselves using zrepl's circlog (depending on zrepl is not pretty, but since this package is supposedly only used by zrepl ATM, this is fine) refs https://github.com/zrepl/zrepl/issues/237
2019-10-26 14:32:07 +02:00
github.com/onsi/ginkgo v1.10.2 // indirect
github.com/onsi/gomega v1.7.0 // indirect
build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy tweaked go mod tidy: see comment in go.mod
2019-10-26 13:55:03 +02:00
github.com/pkg/errors v0.8.1
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
github.com/pkg/profile v1.2.1
transport/ssh: bump go-netssh version to fix ssh client process leaks fixes #322
2020-06-01 13:49:56 +02:00
github.com/problame/go-netssh v0.0.0-20200601114649-26439f9f0dc5
build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy tweaked go mod tidy: see comment in go.mod
2019-10-26 13:55:03 +02:00
github.com/prometheus/client_golang v1.2.1
[#307] add package trace, integrate it with logging, and adopt it throughout zrepl package trace: - introduce the concept of tasks and spans, tracked as linked list within ctx - see package-level docs for an overview of the concepts - **main feature 1**: unique stack of task and span IDs - makes it easy to follow a series of log entries in concurrent code - **main feature 2**: ability to produce a chrome://tracing-compatible trace file - either via an env variable or a `zrepl pprof` subcommand - this is not a CPU profile, we already have go pprof for that - but it is very useful to visually inspect where the replication / snapshotter / pruner spends its time ( fixes #307 ) usage in package daemon/logging: - goal: every log entry should have a trace field with the ID stack from package trace - make `logging.GetLogger(ctx, Subsys)` the authoritative `logger.Logger` factory function - the context carries a linked list of injected fields which `logging.GetLogger` adds to the logger it returns - `logging.GetLogger` also uses package `trace` to get the task-and-span-stack and injects it into the returned logger's fields
2020-04-11 15:49:41 +02:00
github.com/prometheus/common v0.7.0
transports/ssh: bump go-netssh to improve dial errors from go-netssh changelog: dial: better error handling if ssh command exits with non-zero exit status SSHError.Error() relied on go-rwccmd behavior of returning io.EOF if the ssh binary exited with status code 0. We no longe ruse go-rwccmd => capture Stderr ourselves using zrepl's circlog (depending on zrepl is not pretty, but since this package is supposedly only used by zrepl ATM, this is fine) refs https://github.com/zrepl/zrepl/issues/237
2019-10-26 14:32:07 +02:00
github.com/sergi/go-diff v1.0.1-0.20180205163309-da645544ed44 // go1.12 thinks it needs this
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
github.com/spf13/cobra v0.0.2
build: go.mod: bump go-netssh and drop go-critic replaces (go-netssh vendored util/circlog, so the circular dep is gone) fixes build failure reported by @poetterl-ric ``` make ZREPL_VERSION=0.2.1 zrepl-bin GO111MODULE=on go build -mod=readonly -ldflags "-X github.com/zrepl/zrepl/version.zreplVersion=0.2.1" -o "artifacts/zrepl-linux-amd64" go: github.com/problame/go-netssh@v0.0.0-20191026123024-f34099f4f6b1 requires github.com/zrepl/zrepl@v0.2.0 requires github.com/golangci/lint-1@v0.0.0-20181222135242-d2cdd8c08219: invalid version: git fetch --unshallow -f origin in /builddir/go/pkg/mod/cache/vcs/ca789ff49d608cda239a48837cfeea6e9dcdb2bce20051383910eef46b623a33: exit status 128: fatal: git fetch-pack: expected shallow list ```
2019-12-09 13:37:19 +01:00
github.com/spf13/pflag v1.0.5
pre- and post-snapshot hooks * stack-based execution model, documented in documentation * circbuf for capturing hook output * built-in hooks for postgres and mysql * refactor docs, too much info on the jobs page, too difficult to discover snapshotting & hooks Co-authored-by: Ross Williams <ross@ross-williams.net> Co-authored-by: Christian Schwarz <me@cschwarz.com> fixes #74
2019-07-26 21:12:21 +02:00
github.com/stretchr/testify v1.4.0
[#307] add package trace, integrate it with logging, and adopt it throughout zrepl package trace: - introduce the concept of tasks and spans, tracked as linked list within ctx - see package-level docs for an overview of the concepts - **main feature 1**: unique stack of task and span IDs - makes it easy to follow a series of log entries in concurrent code - **main feature 2**: ability to produce a chrome://tracing-compatible trace file - either via an env variable or a `zrepl pprof` subcommand - this is not a CPU profile, we already have go pprof for that - but it is very useful to visually inspect where the replication / snapshotter / pruner spends its time ( fixes #307 ) usage in package daemon/logging: - goal: every log entry should have a trace field with the ID stack from package trace - make `logging.GetLogger(ctx, Subsys)` the authoritative `logger.Logger` factory function - the context carries a linked list of injected fields which `logging.GetLogger` adds to the logger it returns - `logging.GetLogger` also uses package `trace` to get the task-and-span-stack and injects it into the returned logger's fields
2020-04-11 15:49:41 +02:00
github.com/willf/bitset v1.1.10
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
github.com/yudai/gojsondiff v0.0.0-20170107030110-7b1b7adf999d
transports/ssh: bump go-netssh to improve dial errors from go-netssh changelog: dial: better error handling if ssh command exits with non-zero exit status SSHError.Error() relied on go-rwccmd behavior of returning io.EOF if the ssh binary exited with status code 0. We no longe ruse go-rwccmd => capture Stderr ourselves using zrepl's circlog (depending on zrepl is not pretty, but since this package is supposedly only used by zrepl ATM, this is fine) refs https://github.com/zrepl/zrepl/issues/237
2019-10-26 14:32:07 +02:00
github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // go1.12 thinks it needs this
new features: {resumable,encrypted,hold-protected} send-recv, last-received-hold - **Resumable Send & Recv Support** No knobs required, automatically used where supported. - **Hold-Protected Send & Recv** Automatic ZFS holds to ensure that we can always resume a replication step. - **Encrypted Send & Recv Support** for OpenZFS native encryption. Configurable at the job level, i.e., for all filesystems a job is responsible for. - **Receive-side hold on last received dataset** The counterpart to the replication cursor bookmark on the send-side. Ensures that incremental replication will always be possible between a sender and receiver. Design Doc ---------- `replication/design.md` doc describes how we use ZFS holds and bookmarks to ensure that a single replication step is always resumable. The replication algorithm described in the design doc introduces the notion of job IDs (please read the details on this design doc). We reuse the job names for job IDs and use `JobID` type to ensure that a job name can be embedded into hold tags, bookmark names, etc. This might BREAK CONFIG on upgrade. Protocol Version Bump --------------------- This commit makes backwards-incompatible changes to the replication/pdu protobufs. Thus, bump the version number used in the protocol handshake. Replication Cursor Format Change -------------------------------- The new replication cursor bookmark format is: `#zrepl_CURSOR_G_${this.GUID}_J_${jobid}` Including the GUID enables transaction-safe moving-forward of the cursor. Including the job id enables that multiple sending jobs can send the same filesystem without interfering. The `zrepl migrate replication-cursor:v1-v2` subcommand can be used to safely destroy old-format cursors once zrepl has created new-format cursors. Changes in This Commit ---------------------- - package zfs - infrastructure for holds - infrastructure for resume token decoding - implement a variant of OpenZFS's `entity_namecheck` and use it for validation in new code - ZFSSendArgs to specify a ZFS send operation - validation code protects against malicious resume tokens by checking that the token encodes the same send parameters that the send-side would use if no resume token were available (i.e. same filesystem, `fromguid`, `toguid`) - RecvOptions support for `recv -s` flag - convert a bunch of ZFS operations to be idempotent - achieved through more differentiated error message scraping / additional pre-/post-checks - package replication/pdu - add field for encryption to send request messages - add fields for resume handling to send & recv request messages - receive requests now contain `FilesystemVersion To` in addition to the filesystem into which the stream should be `recv`d into - can use `zfs recv $root_fs/$client_id/path/to/dataset@${To.Name}`, which enables additional validation after recv (i.e. whether `To.Guid` matched what we received in the stream) - used to set `last-received-hold` - package replication/logic - introduce `PlannerPolicy` struct, currently only used to configure whether encrypted sends should be requested from the sender - integrate encryption and resume token support into `Step` struct - package endpoint - move the concepts that endpoint builds on top of ZFS to a single file `endpoint/endpoint_zfs.go` - step-holds + step-bookmarks - last-received-hold - new replication cursor + old replication cursor compat code - adjust `endpoint/endpoint.go` handlers for - encryption - resumability - new replication cursor - last-received-hold - client subcommand `zrepl holds list`: list all holds and hold-like bookmarks that zrepl thinks belong to it - client subcommand `zrepl migrate replication-cursor:v1-v2`
2019-09-11 17:19:17 +02:00
github.com/zrepl/yaml-config v0.0.0-20191220194647-cbb6b0cf4bdd
experiment: zync: an rsync-like tool for ZFS datasets (re-uses zrepl abstractions) Local sync and sync over SSH seems to work. The sshdirect transport might be interesting / re-usable for zrepl as well, although we'd still need some kind of locking in that case.
2021-01-14 01:31:08 +01:00
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy tweaked go mod tidy: see comment in go.mod
2019-10-26 13:55:03 +02:00
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
golang.org/x/sync v0.0.0-20190423024810-112230192c58
build: fix freebsd/aarch64 by updating golang:sys dependency This was original reported here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242456 And originally fixed upstream here: https://go.googlesource.com/sys/+/33540a1f6037
2020-02-01 06:23:47 +01:00
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037
[#321] platformtest: generate test case list + coverage tooling
2020-05-24 17:43:42 +02:00
golang.org/x/tools v0.0.0-20190206041539-40960b6deb8e
endpoint: refactor, fix stale holds on initial replication failure, zfs-abstractions subcmd, more efficient ZFS queries The motivation for this recatoring are based on two independent issues: - @JMoVS found that the changes merged as part of #259 slowed his OS X based installation down significantly. Analysis of the zfs command logging introduced in #296 showed that `zfs holds` took most of the execution time, and they pointed out that not all of those `zfs holds` invocations were actually necessary. I.e.: zrepl was inefficient about retrieving information from ZFS. - @InsanePrawn found that failures on initial replication would lead to step holds accumulating on the sending side, i.e. they would never be cleaned up in the HintMostRecentCommonAncestor RPC handler. That was because we only sent that RPC if there was a most recent common ancestor detected during replication planning. @InsanePrawn prototyped an implementation of a `zrepl zfs-abstractions release` command to mitigate the situation. As part of that development work and back-and-forth with @problame, it became evident that the abstractions that #259 built on top of zfs in package endpoint (step holds, replication cursor, last-received-hold), were not well-represented for re-use in the `zrepl zfs-abstractions release` subocommand prototype. This commit refactors package endpoint to address both of these issues: - endpoint abstractions now share an interface `Abstraction` that, among other things, provides a uniform `Destroy()` method. However, that method should not be destroyed directly but instead the package-level `BatchDestroy` function should be used in order to allow for a migration to zfs channel programs in the future. - endpoint now has a query facitilty (`ListAbstractions`) which is used to find on-disk - step holds and bookmarks - replication cursors (v1, v2) - last-received-holds By describing the query in a struct, we can centralized the retrieval of information via the ZFS CLI and only have to be clever once. We are "clever" in the following ways: - When asking for hold-based abstractions, we only run `zfs holds` on snapshot that have `userrefs` > 0 - To support this functionality, add field `UserRefs` to zfs.FilesystemVersion and retrieve it anywhere we retrieve zfs.FilesystemVersion from ZFS. - When asking only for bookmark-based abstractions, we only run `zfs list -t bookmark`, not with snapshots. - Currently unused (except for CLI) per-filesystem concurrent lookup - Option to only include abstractions with CreateTXG in a specified range - refactor `endpoint`'s various ZFS info retrieval methods to use `ListAbstractions` - rename the `zrepl holds list` command to `zrepl zfs-abstractions list` - make `zrepl zfs-abstractions list` consume endpoint.ListAbstractions - Add a `ListStale` method which, given a query template, lists stale holds and bookmarks. - it uses replication cursor has different modes - the new `zrepl zfs-abstractions release-{all,stale}` commands can be used to remove abstractions of package endpoint - Adjust HintMostRecentCommonAncestor RPC for stale-holds cleanup: - send it also if no most recent common ancestor exists between sender and receiver - have the sender clean up its abstractions when it receives the RPC with no most recent common ancestor, using `ListStale` - Due to changed semantics, bump the protocol version. - Adjust HintMostRecentCommonAncestor RPC for performance problems encountered by @JMoVS - by default, per (job,fs)-combination, only consider cleaning step holds in the createtxg range `[last replication cursor,conservatively-estimated-receive-side-version)` - this behavior ensures resumability at cost proportional to the time that replication was donw - however, as explained in a comment, we might leak holds if the zrepl daemon stops running - that trade-off is acceptable because in the presumably rare this might happen the user has two tools at their hand: - Tool 1: run `zrepl zfs-abstractions release-stale` - Tool 2: use env var `ZREPL_ENDPOINT_SENDER_HINT_MOST_RECENT_STEP_HOLD_CLEANUP_MODE` to adjust the lower bound of the createtxg range (search for it in the code). The env var can also be used to disable hold-cleanup on the send-side entirely. supersedes closes #293 supersedes closes #282 fixes #280 fixes #278 Additionaly, we fixed a couple of bugs: - zfs: fix half-nil error reporting of dataset-does-not-exist for ZFSListChan and ZFSBookmark - endpoint: Sender's `HintMostRecentCommonAncestor` handler would not check whether access to the specified filesystem was allowed.
2020-03-26 23:43:17 +01:00
gonum.org/v1/gonum v0.7.0 // indirect
build: drop go Dep, switch to modules, support Go 1.13 bump enumer to v1.1.1 bump golangci-lint to v1.17.1 no `go mod tidy` because 1.13 and 1.12 seem to alter each other's output fixes #112
2019-09-03 14:45:18 +02:00
google.golang.org/grpc v1.17.0
)
experiment: zync: an rsync-like tool for ZFS datasets (re-uses zrepl abstractions) Local sync and sync over SSH seems to work. The sshdirect transport might be interesting / re-usable for zrepl as well, although we'd still need some kind of locking in that case.
2021-01-14 01:31:08 +01:00
replace github.com/problame/go-netssh => /home/cs/zrepl/go-netssh
Reference in New Issue Copy Permalink