extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-12-02 21:37:50 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,047 Commits 48 Branches 23 Tags 3.6 MiB
bdd0c9a199
Commit Graph

1047 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Christian Schwarz
9a4763ceee client/status: notify user if size estimation is imprecise 
There's plenty of room for improvement here.
For example, detect if we're past the last step without size estimation
and compute the remaining sum of bytes to be replicated from there on.
 2020-02-14 21:42:03 +01:00
Christian Schwarz
f8200a6386 replication/driver: regulate per-filesystem step planning through stepQueue 
before this patch, we'd have unbounded parallelism for ListFilesystemVersions RPCs
 2020-02-14 21:42:03 +01:00
Christian Schwarz
e35320f8ee zfs: make StreamCopier wrapper for io.ReadCloser public 2020-02-14 21:42:03 +01:00
Christian Schwarz
5b52e5e331 rpc: fix missing logger context vars in control connection handlers 
use ctxInterceptor in gRPC interceptors
also panic if the unimplemented stream interceptor is used
 2020-02-14 21:42:03 +01:00
Christian Schwarz
6ebd9f1037 zfs: recv: fix deadlock if streamCopier returns io.EOF 
Close the write end of the pipe
* before we start waiting on the error channels
* and after everything from streamCopier has been written to the pipe
 2020-02-14 21:42:03 +01:00
Christian Schwarz
99ab16d7be zfs: send: improve error reporting by capturing stderr 2020-02-14 21:42:03 +01:00
Christian Schwarz
e675b35cda platformtest: harness: run filter + summary 2020-02-14 21:42:01 +01:00
Christian Schwarz
ddd7acec49 platformtest: harness: refactor + support SkipNow 2020-02-14 21:40:48 +01:00
Christian Schwarz
e7aa08564b platformtest: harness: fix FailNow (harness wouldn't detect FailNow) 
(harness checks for recover() != nil)
 2020-02-14 21:40:48 +01:00
Christian Schwarz
93ccdb8024 docs: prune: fix typo 2020-02-14 21:40:48 +01:00
Christian Schwarz
501645f918 docs: filter syntax: reference 'snap' job type 2020-02-14 21:40:48 +01:00
Matthias Freund
4fd9d30c98 build: add linux/armhf target 2020-02-10 12:09:11 +01:00
Ben Woods
e2b9c16959 build: fix freebsd/aarch64 
by updating golang:sys dependency

This was original reported here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242456

And originally fixed upstream here:
https://go.googlesource.com/sys/+/33540a1f6037
 2020-02-01 12:53:59 +01:00
Matthias Freund
4ba85c40cc daemon/job/build_jobs: fix validateReceivingSidesDoNotOverlap 2020-01-30 10:46:32 +01:00
Matthias Freund
cca95f613b client/status: add ETA calculation 2020-01-26 14:45:01 +01:00
Christian Schwarz
5b50a66c6c daemon/snapper: refactor sync-up algorithm + warn about FSes awaiting first sync point 
refs https://github.com/zrepl/zrepl/issues/256
 2020-01-15 19:20:37 +01:00
Christian Schwarz
dd508280f0 docs: tutorial: minor typo + language fixes 2020-01-15 19:12:09 +01:00
Christian Schwarz
2927d0ca15 rpc: use grpchelper package, add grpc.KeepaliveEnforcementPolicy, fix 'transport is closing' error 
Symptom: zrepl log message:

    rpc error: code = Unavailable desc = transport is closing

Underlying Problem:

* rpc.NewServer was not using grpchelper.NewServer and not setting Server KeepaliveParams by itself
* and even grpchelper.NewServer didn't set a KeepaliveEnforcementPolicy
* However, KeepaliveEnforcementPolicy is necessary if the client keepalive is configured with non-default values
* .. which grpchelper.ClientConn does, and that method is used by rpc.NewClient

* rpc.Client was sending pings
* lacking server-side KeepaliveEnforcementPolicy caused grpc-hard-coded `pingStrikes` counter to go past limit of 2:
  021bd5734e/internal/transport/http2_server.go (L726)

How was this debugged:
* GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info PATH=/root/mockpath:$PATH zrepl daemon
* with a patch on grpc package to get more log messages on pingStrikes increases:

    diff --git a/internal/transport/http2_server.go b/internal/transport/http2_server.go
    index 8b04b039..f68f55ea 100644
    --- a/internal/transport/http2_server.go
    +++ b/internal/transport/http2_server.go
    @@ -214,6 +214,7 @@ func newHTTP2Server(conn net.Conn, config *ServerConfig) (_ ServerTransport, err
            if kep.MinTime == 0 {
                    kep.MinTime = defaultKeepalivePolicyMinTime
            }
    +       errorf("effective keepalive enforcement policy: %#v", kep)
            done := make(chan struct{})
            t := &http2Server{
                    ctx:               context.Background(),
    @@ -696,6 +697,7 @@ func (t *http2Server) handlePing(f *http2.PingFrame) {
            t.controlBuf.put(pingAck)

            now := time.Now()
    +       errorf("transport:ping handlePing, last ping %s ago", now.Sub(t.lastPingAt))
            defer func() {
                    t.lastPingAt = now
            }()
    @@ -713,11 +715,13 @@ func (t *http2Server) handlePing(f *http2.PingFrame) {
                    // Keepalive shouldn't be active thus, this new ping should
                    // have come after at least defaultPingTimeout.
                    if t.lastPingAt.Add(defaultPingTimeout).After(now) {
    +                       errorf("transport:ping strike ns < 1 && !t.kep.PermitWithoutStream")
                            t.pingStrikes++
                    }
            } else {
                    // Check if keepalive policy is respected.
                    if t.lastPingAt.Add(t.kep.MinTime).After(now) {
    +                       errorf("transport:ping strike !(ns < 1 && !t.kep.PermitWithoutStream) kep.MinTime=%s ns=%d", t.kep.MinTime, ns)
                            t.pingStrikes++
                    }
            }

fixes #181
 2020-01-04 21:10:41 +01:00
Juergen Hoetzel
d35e2400b2 transport/{TCP,TLS}: optional IP_FREEBIND / IP_BINDANY bind socketops 
Allows to bind to an address even if it is not actually (yet or ever)
configured. Fixes #238

Rationale:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/#whatdoesthismeanformeadeveloper
 2020-01-04 17:21:48 +01:00
Frans Bergman
47ed599db7 docs: add Void Linux to installation instructions 2019-12-28 12:43:53 +01:00
Christian Schwarz
f899f4cbe4 build: go.mod: bump go-netssh and drop go-critic replaces 
(go-netssh vendored util/circlog, so the circular dep is gone)

fixes build failure reported by @poetterl-ric

```
  make ZREPL_VERSION=0.2.1 zrepl-bin
    GO111MODULE=on go build -mod=readonly -ldflags "-X github.com/zrepl/zrepl/version.zreplVersion=0.2.1" -o "artifacts/zrepl-linux-amd64"
    go: github.com/problame/go-netssh@v0.0.0-20191026123024-f34099f4f6b1 requires
            github.com/zrepl/zrepl@v0.2.0 requires
            github.com/golangci/lint-1@v0.0.0-20181222135242-d2cdd8c08219: invalid version: git fetch --unshallow -f origin in /builddir/go/pkg/mod/cache/vcs/ca789ff49d608cda239a48837cfeea6e9dcdb2bce20051383910eef46b623a33: exit status 128:
            fatal: git fetch-pack: expected shallow list
```
 2019-12-28 12:42:33 +01:00
Juergen Hoetzel
b3231d2bed daemon: fix typos in error messages 
closes #255
 2019-12-11 21:30:48 +01:00
Christian Schwarz
c24c327151 build: fix build.Dockerfile + integrate into CircleCI 
fixup of 080f2c0616
fixup of 4994b7a9ea
 2019-11-28 15:19:46 +01:00
Christian Schwarz
5e17d7ba80 docs: add recent supporters 2019-11-26 00:45:13 +01:00
Christian Schwarz
0261dbfe3d docs: 0.2.1 changelog 2019-11-20 20:16:41 +01:00
Christian Schwarz
4301f741db dist/systemd: remove @privileged from SystemCallFilter + cleanup comments 
fixes #237
 2019-11-20 18:44:14 +01:00
Christian Schwarz
7e743c74dc docs + samples: adjust ssh 'Compression' arg in examples 2019-11-20 18:19:16 +01:00
Christian Schwarz
ad0b055245 daemon/prometheus: fix crash if listener cannot be created 
refs #238

  zrepl version=v0.2.0-11-gdc39c81 GOOS=linux GOARCH=amd64 Compiler=gc
 starting daemon
 [pull_source]: starting job
 [_prometheus]: starting job
 [connection_loss_tidyup]: starting job
 [connection_loss_tidyup]: wait for wakeups
 [_control]: starting job
 [_prometheus]: cannot listen err="listen tcp 10.0.0.200:9091: bind: cannot assign requested add
 [_prometheus]: job exited
 panic: runtime error: invalid memory address or nil pointer dereference
         panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x81ea4d]
 goroutine 25 [running]:
 net/http.(*onceCloseListener).close(...)
         /usr/local/go/src/net/http/server.go:3330
 sync.(*Once).doSlow(0xc00018b060, 0xc0000c7bc0)
         /usr/local/go/src/sync/once.go:66 +0xe3
 sync.(*Once).Do(...)
         /usr/local/go/src/sync/once.go:57
 net/http.(*onceCloseListener).Close(0xc00018b050, 0xc0003a6000, 0xe0)
         /usr/local/go/src/net/http/server.go:3326 +0x77
 panic(0xb1d1c0, 0x11d7d90)
         /usr/local/go/src/runtime/panic.go:679 +0x1b2
 net/http.(*onceCloseListener).Accept(0xc00018b050, 0xc000120020, 0xb0fd20, 0x11d7ce0, 0xbee6e0)
         <autogenerated>:1 +0x32
 net/http.(*Server).Serve(0xc0003a6000, 0x0, 0x0, 0x0, 0x0)
         /usr/local/go/src/net/http/server.go:2896 +0x286
 net/http.Serve(...)
         /usr/local/go/src/net/http/server.go:2468
 github.com/zrepl/zrepl/daemon.(*prometheusJob).Run(0xc000109940, 0xd19ec0, 0xc00018a930)
         /go/src/github.com/zrepl/zrepl/daemon/prometheus.go:75 +0x23e
 github.com/zrepl/zrepl/daemon.(*jobs).start.func1(0xc0000f68c0, 0xd22c40, 0xc000116ee0, 0xd1ba4
         /go/src/github.com/zrepl/zrepl/daemon/daemon.go:220 +0x121
 created by github.com/zrepl/zrepl/daemon.(*jobs).start
         /go/src/github.com/zrepl/zrepl/daemon/daemon.go:216 +0x52e
 zrepl.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
 zrepl.service: Failed with result 'exit-code'.
 2019-11-16 22:11:13 +01:00
Christian Schwarz
27db3e6f70 docs: supporters: update & add viz for different kinds of support 2019-11-16 22:11:07 +01:00
Christian Schwarz
4994b7a9ea rpc/dataconn + build: support GOOS={solaris,illumos} 2019-11-16 22:07:47 +01:00
Christian Schwarz
080f2c0616 build: Makefile: refactor cross-builds + release, add i386 targets 2019-11-16 22:07:47 +01:00
Christian Schwarz
d469cc04b6 transports/ssh: bump go-netssh to improve dial errors 
from go-netssh changelog:

    dial: better error handling if ssh command exits with non-zero exit status

    SSHError.Error() relied on go-rwccmd behavior of returning io.EOF if the
    ssh binary exited with status code 0.

    We no longe ruse go-rwccmd => capture Stderr ourselves using zrepl's
    circlog (depending on zrepl is not pretty, but since this package is supposedly
    only used by zrepl ATM, this is fine)

    refs https://github.com/zrepl/zrepl/issues/237
 2019-11-16 22:07:47 +01:00
Christian Schwarz
e0a25d04ac build: Makefile: set GO111MODULE=on for all go commands 2019-11-16 22:07:47 +01:00
Christian Schwarz
b0f2c79944 build: go mods: split build deps into subgomod, bump prometheus to 1.2.1, tweaked go mod tidy 
tweaked go mod tidy: see comment in go.mod
 2019-11-16 22:07:47 +01:00
Christian Schwarz
d2bc40f78d docs: transports: ssh: better copy-pastable connect section 2019-11-16 22:07:47 +01:00
Christian Schwarz
9e54f11960 dist/systemd: fix ssh-transport: create stdinserver runtime directory 
tested to work on Debian Stretch

refs #237
 2019-11-16 22:07:38 +01:00
Andy Fiddaman
6eda1f743f Fix typo in tutorial.rst 2019-11-05 09:57:30 -08:00
Andy Fiddaman
6787decef1 Add OmniOS (illumos distribution) to list of OSs 2019-11-05 09:49:57 -08:00
Christian Schwarz
d56d45a2ab docs: install: apt: fix snippet display & link to packaging repo 2019-10-21 16:35:23 +02:00
Christian Schwarz
fcf16a163a docs: install: apt snippet: idempotent, bash compat, multiarch compat 
Co-authored-by: Janis Streib <me@janis-streib.de>
Co-authored-by: Christian Schwarz <me@cschwarz.com>
 2019-10-21 16:21:51 +02:00
Christian Schwarz
dc39c819a3 docs: add debian + ubuntu installation 2019-10-18 20:18:42 +02:00
Christian Schwarz
1048b09487 build: include config examples and dist in noarch tarball 2019-10-18 20:18:42 +02:00
Richard Poettler
3806e97404 docs: add copr repo for Fedora/CentOS 
closes #229
 2019-10-16 10:46:02 +02:00
chenhao
c396f9508a zfs: replace hard coded zfs command in ZFSDestroy 
fixes #231
 2019-10-16 10:22:53 +02:00
Christian Schwarz
b9933f6cb2 platformtest: add zfsGet bookmark handling & replicationCursor tests 
This encodes the observation made in issue #230 :
In the ZFS version shipped in Ubuntu 16.04 where
`zfs get someprop a#bookmark` does not work.
 2019-10-14 17:54:14 +02:00
Christian Schwarz
0ba4b5eda6 zfs: helper for ZFSGet guid and createtxg 2019-10-14 17:54:14 +02:00
Christian Schwarz
18d2c350de platformtest: harness: -failure.stop-and-keep-pool mode, prettier logging 2019-10-14 17:54:14 +02:00
Christian Schwarz
f8f9fd11cd platformtest: logging-related refactorings 2019-10-14 17:32:58 +02:00
John Ramsden
b422e6f12e docs: installation: add Arch Linux 'from source' package 2019-10-13 12:33:27 +02:00
Christian Schwarz
f8d5082bdd docs: remove outdated implementation references + remove 0.2-rc* from published docs 2019-10-13 12:26:39 +02:00