Commit Graph

1102 Commits

Author SHA1 Message Date
Christian Schwarz
a58ce74ed0 implement new 'zrepl status'
Primary goals:

- Scrollable output ( fixes #245 )
- Sending job signals from status view
- Filtering of output by filesystem

Implementation:

- original TUI framework: github.com/rivo/tview
- but: tview is quasi-unmaintained, didn't support some features
- => use fork https://gitlab.com/tslocum/cview
- however, don't buy into either too much to avoid lock-in

- instead: **port over the existing status UI drawing code
  and adjust it to produce strings instead of directly
  drawing into the termbox buffer**

Co-authored-by: Calistoc <calistoc@protonmail.com>
Co-authored-by: InsanePrawn <insane.prawny@gmail.com>

fixes #245
fixes #220
2021-03-14 18:24:25 +01:00
Calistoc
2c8c2cfa14 add signal 'snapshot', rename existing signal 'wakeup' to 'replication' 2021-03-14 18:16:23 +01:00
Christian Schwarz
0ceea1b792 replication: simplify parallel replication variables & expose them in config
closes #140
2021-03-14 17:30:10 +01:00
Christian Schwarz
07f2bfff6a build: bump release build Go version + quickcheck default to Go 1.16 2021-02-20 17:29:17 +01:00
Christian Schwarz
b9ee14f6d7 endpoint.ReceiverConfig: remove obsolete TODO comment 2021-02-20 17:20:58 +01:00
InsanePrawn
393fc10a69 [#285] support setting zfs send / recv flags in the config (send: -wLcepbS, recv: -ox)
Co-authored-by: Christian Schwarz <me@cschwarz.com>
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>

closes #285
closes #276
closes #24
2021-02-20 17:20:45 +01:00
Christian Schwarz
1c937e58f7 zfs.NilBool: document its purpose and move it to its own package 'nodefault' 2021-02-20 17:04:57 +01:00
Christian Schwarz
70bbdfe760 zfs: ResumeToken: parse embedok, largeblockok, savedok if available
Developed for #285 but ultimately not used for it.
2021-02-20 17:04:57 +01:00
Christian Schwarz
efe7b17d21 Update to protobuf v1.25 and grpc 1.35; bump CI to go1.12
From:
github.com/golang/protobuf v1.3.2
google.golang.org/grpc v1.17.0

To:
github.com/golang/protobuf v1.4.3
google.golang.org/grpc v1.35.0
google.golang.org/protobuf v1.25.0

About the two protobuf packages:
https://developers.google.com/protocol-buffers/docs/reference/go/faq
> Version v1.4.0 and higher of github.com/golang/protobuf wrap the new
implementation and permit programs to adopt the new API incrementally. For
example, the well-known types defined in github.com/golang/protobuf/ptypes are
simply aliases of those defined in the newer module. Thus,
google.golang.org/protobuf/types/known/emptypb and
github.com/golang/protobuf/ptypes/empty may be used interchangeably.

Notable Code Changes in zrepl:
- generate protobufs now contain a mutex so we can't copy them by value
  anymore
- grpc.WithDialer is deprecated => use grpc.WithContextDialer instead

Go1.12 is now actually required by some of the dependencies.
2021-01-25 00:39:01 +01:00
Christian Schwarz
166e80bb96 lazy.sh: use 'go install' + import paths from build/tools.go 2021-01-25 00:16:01 +01:00
Mathias Fredriksson
6ac537210b daemon: avoid math/rand race by using global source
Unless we're using the global source for math/rand, (*rand.Rand).Read
should not be called concurrently. We seed the rng in daemon.Run to
avoid ambiguity or hiding global side effects inside packages.

closes #414
2021-01-25 00:16:01 +01:00
Christian Schwarz
596a39c0f5 bump golangci-lint to 1.35.2 and fix resulting lint errors
GO111MODULE=on golangci-lint run ./...
endpoint/endpoint.go:487:9: S1039: unnecessary use of fmt.Sprintf (gosimple)
                panic(fmt.Sprintf("ClientIdentityKey context value must be set"))
                      ^
platformtest/platformtest_ops.go:259:41: S1039: unnecessary use of fmt.Sprintf (gosimple)
                                return nil, &LineError{scan.Text(), fmt.Sprintf("unexpected tokens at EOL")}
                                                                    ^
platformtest/platformtest_ops.go:266:41: S1039: unnecessary use of fmt.Sprintf (gosimple)
                                return nil, &LineError{scan.Text(), fmt.Sprintf("unexpected tokens at EOL")}
                                                                    ^
util/optionaldeadline/optionaldeadline_test.go:97:50: SA1029: should not use built-in type string as key for value; define your own type to avoid collisions (staticcheck)
        pctx := context.WithValue(context.Background(), "key", "value")
                                                        ^
rpc/rpc_debug.go:8:5: var `debugEnabled` is unused (unused)
rpc/dataconn/dataconn_debug.go:8:5: var `debugEnabled` is unused (unused)
rpc/dataconn/frameconn/frameconn.go:42:9: S1039: unnecessary use of fmt.Sprintf (gosimple)
                panic(fmt.Sprintf("frame header is 8 bytes long"))
                      ^
platformtest/platformtest_ops.go:322:40: S1039: unnecessary use of fmt.Sprintf (gosimple)
                        return nil, &LineError{scan.Text(), fmt.Sprintf("unexpected tokens at EOL")}
2021-01-25 00:16:01 +01:00
Mathias Fredriksson
d118bcc717 rpc: fix data race in timeoutconn
- `timeoutconn` handles state, yet calls to Read/Write make a copy of
  that state (non-pointer receiver) so any outbound calls will not have
  the state updated

- Even without the copy issue, the renew methods can in edge cases set a
  new deadline _after_ DisableTimeouts have been called, consider the
  following racy behavior:
    1. `renewReadDeadline` is called, checks `renewDeadlinesDisabled`
       (not disabled)
    2. `DisableTimeouts` is called, sets `renewDeadlinesDisabled`
    3. `DisableTimeouts` invokes `c.SetDeadline`
    4. `renewReadDeadline` invokes `c.SetReadDeadline`

To fix the above, the `Conn` receiver was made to be a pointer
everywhere and access to renewDeadlinesDisabled is now guarded
by an RWMutex instead of using atomics.

closes #415
2021-01-25 00:16:01 +01:00
Mathias Fredriksson
48be4032a2 daemon: fix data race in snapjob pruner report
closes #416
2021-01-25 00:16:01 +01:00
Mathias Fredriksson
2d545c87cd rpc: fix read mutex unlock in dataconn stream
closes #413
2021-01-25 00:16:01 +01:00
Christian Schwarz
0b48ba54f8 replication/driver: simplify second-attempt step correlation code & fix statekeeping
Before this change, the step correlation code returned early in several cases:
- did not set f.planning.done in the cases where it was a no-op
- did not set f.planning.err in the cases where correlation did not
  succeed

Reported-by: InsanePrawn <insane.prawny@gmail.com>
2021-01-25 00:16:01 +01:00
Christian Schwarz
c3d87289bb [#388] util/semaphore: fix TestSemaphore test
fixes #388
2021-01-24 22:28:47 +01:00
Christian Schwarz
0d96627ffb [#388] trace: make WithTaskGroup actually concurrent 2021-01-24 22:28:47 +01:00
Christian Schwarz
61acc7494a [#388] endpoint: fix incorrect use of trace.WithTaskGroup in ListAbstractionsStreamed
Capturing behavior was broken.
2021-01-24 13:58:23 +01:00
Rafał Bugajewski
96d5288667
docs: fix typos 2020-12-17 12:00:29 +01:00
Christian Schwarz
b8cd3c59f1 docs: update supporters 2020-11-11 16:06:54 +01:00
Christian Schwarz
91632b52bb build: circleci: periodic full build: only build each of the matching branches once 2020-11-04 17:53:54 +01:00
Christian Schwarz
d39c0e3745 docs + readme: actually fix Patreon badge
see https://github.com/endel/shieldsio-patreon/issues/8#issuecomment-700144629
2020-11-01 16:10:48 +01:00
Christian Schwarz
180eaea195 docs: 0.3.1 changelog 2020-11-01 14:19:03 +01:00
Christian Schwarz
69ed2d7117 docs + readme: fix Patreon badge 2020-11-01 14:18:36 +01:00
Christian Schwarz
c420f3c909 [#381] zfs: ListFilesystemVersions: make list filesystems version invocation deterministic
fixes #381
ref #379
2020-11-01 13:59:21 +01:00
Christian Schwarz
0a2dea05a9 [#385] status + replication: warning if replication succeeeded without any filesystem being replicated
refs #385
refs #384
2020-11-01 13:51:28 +01:00
Christian Schwarz
293c89d392 [#385] replication: report AttemptDone if no filesystems are replicated
fixes #385
2020-11-01 13:51:28 +01:00
Jeremy Bryan Smith
bb5ef0c8b2 docs: fix link to template.sh sample hook file 2020-11-01 10:45:17 +01:00
Christian Schwarz
8f44cdc284 build: rpm: don't strip binaries
fixes build failure for aarch64

```
make rpm-docker GOARCH=arm64 GOOS=linux

+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip /usr/bin/strip
/usr/bin/strip: Unable to recognise the format of the input file `/build/src/artifacts/rpmbuild/BUILDROOT/zrepl-v0.3.0.38.g53028ed5-1.aarch64/usr/bin/zrepl'
error: Bad exit status from /var/tmp/rpm-tmp.g8Fdbs (%install)
    Bad exit status from /var/tmp/rpm-tmp.g8Fdbs (%install)
```
2020-11-01 10:23:50 +01:00
Christian Schwarz
53028ed50a docs: gen-sphinx-versioning-flags.py: implement the stable branch logic used for 0.3.0 release 2020-09-12 14:03:45 +02:00
Christian Schwarz
17e152c601 docs: build zrepl.github.io on circleci 2020-09-12 14:03:45 +02:00
Christian Schwarz
2ba6aabd4a build: circleci: quickcheck-docs: upload artifacts 2020-09-06 17:28:16 +02:00
Christian Schwarz
98207c904d docs + README: document new build & release process 2020-09-06 17:28:16 +02:00
Christian Schwarz
d2b825a8ae build: daily release builds of branches stable, master, problame/circleci-build 2020-09-06 17:28:16 +02:00
Christian Schwarz
913b8b37fe build: circleci: include pipeline number in minio url 2020-09-06 17:22:44 +02:00
Christian Schwarz
f494c5ba31 build: pin Go version in circleci 2020-09-06 17:15:32 +02:00
Christian Schwarz
b9503685f0 build: parametrized pipeline (no more approval for release build), embed exact go version info in artifacts
refs #378
refs #377
2020-09-06 15:39:34 +02:00
Christian Schwarz
0f4143c0e0 docs: update supporters 2020-09-05 17:49:35 +02:00
Christian Schwarz
8839ed1f95 docs: update multi-job & multi-host setup section 2020-09-05 17:45:18 +02:00
Christian Schwarz
41b4038ad5 docs: add example setup 'local disk backup' to jobs overview table 2020-09-05 17:44:46 +02:00
Christian Schwarz
88d21eb23a Makefile: docs: treat Sphinx warnings as errors 2020-09-05 16:41:22 +02:00
Christian Schwarz
e0be7e4d4f docs: installation/rpm-repos: remove ineffective literalinclude directive 2020-09-05 16:40:33 +02:00
Christian Schwarz
0c189265e8 platformtests: fix skipping encryption-only tests on systems that don't support encryption
(Or split the test into two tests  of which one is skipped depending on encryption support)
2020-09-05 16:04:34 +02:00
Christian Schwarz
b1f8cdf385 [#373] pruning: add optional regex field to last_n rule
fixes #373
2020-09-02 22:45:44 +02:00
Christian Schwarz
428a60870a pruning: cleanup retention grid impl + tests + correct docs
package is now at 95% code coverage and the additional tests codify
all behavior specified in the docs

There is a slight change in behavior:
Intervals are now [duration) instead of (duration].
If the leftmost interval is not keep=all, the most recently created
snapshot will be destroyed if there are other snapshots within
that first interval.
Since we recommend keep=all all over the docs, and zrepl 0.3
will put holds on that snapshot if it is being replicated,
I feel like this is an acceptable change in behavior.

refs #292
fixup of 0bbe2befce
2020-09-02 22:45:44 +02:00
Christian Schwarz
af2d6579c5 [#347] zfscmd: fix dangling trace Task on .Start() failure
fixes #347
2020-09-02 22:45:44 +02:00
Christian Schwarz
0f3da73ef1 [#347] zfscmd + zfs: define .Start() semantics, apply to call sites in pkg zfs
fixes #347
2020-09-02 22:45:44 +02:00
Christian Schwarz
fecc9416ab [#347] package trace: envconst-configurable debug mode 2020-09-02 22:45:44 +02:00
Christian Schwarz
a7915db4c3 [#347] package trace: printing debugString before instead of at panic (fixup e500d9e) 2020-09-02 22:45:44 +02:00