Commit Graph

947 Commits

Author SHA1 Message Date
InsanePrawn
393fc10a69 [#285] support setting zfs send / recv flags in the config (send: -wLcepbS, recv: -ox)
Co-authored-by: Christian Schwarz <me@cschwarz.com>
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>

closes #285
closes #276
closes #24
2021-02-20 17:20:45 +01:00
Christian Schwarz
1c937e58f7 zfs.NilBool: document its purpose and move it to its own package 'nodefault' 2021-02-20 17:04:57 +01:00
Christian Schwarz
70bbdfe760 zfs: ResumeToken: parse embedok, largeblockok, savedok if available
Developed for #285 but ultimately not used for it.
2021-02-20 17:04:57 +01:00
Christian Schwarz
efe7b17d21 Update to protobuf v1.25 and grpc 1.35; bump CI to go1.12
From:
github.com/golang/protobuf v1.3.2
google.golang.org/grpc v1.17.0

To:
github.com/golang/protobuf v1.4.3
google.golang.org/grpc v1.35.0
google.golang.org/protobuf v1.25.0

About the two protobuf packages:
https://developers.google.com/protocol-buffers/docs/reference/go/faq
> Version v1.4.0 and higher of github.com/golang/protobuf wrap the new
implementation and permit programs to adopt the new API incrementally. For
example, the well-known types defined in github.com/golang/protobuf/ptypes are
simply aliases of those defined in the newer module. Thus,
google.golang.org/protobuf/types/known/emptypb and
github.com/golang/protobuf/ptypes/empty may be used interchangeably.

Notable Code Changes in zrepl:
- generate protobufs now contain a mutex so we can't copy them by value
  anymore
- grpc.WithDialer is deprecated => use grpc.WithContextDialer instead

Go1.12 is now actually required by some of the dependencies.
2021-01-25 00:39:01 +01:00
Christian Schwarz
166e80bb96 lazy.sh: use 'go install' + import paths from build/tools.go 2021-01-25 00:16:01 +01:00
Mathias Fredriksson
6ac537210b daemon: avoid math/rand race by using global source
Unless we're using the global source for math/rand, (*rand.Rand).Read
should not be called concurrently. We seed the rng in daemon.Run to
avoid ambiguity or hiding global side effects inside packages.

closes #414
2021-01-25 00:16:01 +01:00
Christian Schwarz
596a39c0f5 bump golangci-lint to 1.35.2 and fix resulting lint errors
GO111MODULE=on golangci-lint run ./...
endpoint/endpoint.go:487:9: S1039: unnecessary use of fmt.Sprintf (gosimple)
                panic(fmt.Sprintf("ClientIdentityKey context value must be set"))
                      ^
platformtest/platformtest_ops.go:259:41: S1039: unnecessary use of fmt.Sprintf (gosimple)
                                return nil, &LineError{scan.Text(), fmt.Sprintf("unexpected tokens at EOL")}
                                                                    ^
platformtest/platformtest_ops.go:266:41: S1039: unnecessary use of fmt.Sprintf (gosimple)
                                return nil, &LineError{scan.Text(), fmt.Sprintf("unexpected tokens at EOL")}
                                                                    ^
util/optionaldeadline/optionaldeadline_test.go:97:50: SA1029: should not use built-in type string as key for value; define your own type to avoid collisions (staticcheck)
        pctx := context.WithValue(context.Background(), "key", "value")
                                                        ^
rpc/rpc_debug.go:8:5: var `debugEnabled` is unused (unused)
rpc/dataconn/dataconn_debug.go:8:5: var `debugEnabled` is unused (unused)
rpc/dataconn/frameconn/frameconn.go:42:9: S1039: unnecessary use of fmt.Sprintf (gosimple)
                panic(fmt.Sprintf("frame header is 8 bytes long"))
                      ^
platformtest/platformtest_ops.go:322:40: S1039: unnecessary use of fmt.Sprintf (gosimple)
                        return nil, &LineError{scan.Text(), fmt.Sprintf("unexpected tokens at EOL")}
2021-01-25 00:16:01 +01:00
Mathias Fredriksson
d118bcc717 rpc: fix data race in timeoutconn
- `timeoutconn` handles state, yet calls to Read/Write make a copy of
  that state (non-pointer receiver) so any outbound calls will not have
  the state updated

- Even without the copy issue, the renew methods can in edge cases set a
  new deadline _after_ DisableTimeouts have been called, consider the
  following racy behavior:
    1. `renewReadDeadline` is called, checks `renewDeadlinesDisabled`
       (not disabled)
    2. `DisableTimeouts` is called, sets `renewDeadlinesDisabled`
    3. `DisableTimeouts` invokes `c.SetDeadline`
    4. `renewReadDeadline` invokes `c.SetReadDeadline`

To fix the above, the `Conn` receiver was made to be a pointer
everywhere and access to renewDeadlinesDisabled is now guarded
by an RWMutex instead of using atomics.

closes #415
2021-01-25 00:16:01 +01:00
Mathias Fredriksson
48be4032a2 daemon: fix data race in snapjob pruner report
closes #416
2021-01-25 00:16:01 +01:00
Mathias Fredriksson
2d545c87cd rpc: fix read mutex unlock in dataconn stream
closes #413
2021-01-25 00:16:01 +01:00
Christian Schwarz
0b48ba54f8 replication/driver: simplify second-attempt step correlation code & fix statekeeping
Before this change, the step correlation code returned early in several cases:
- did not set f.planning.done in the cases where it was a no-op
- did not set f.planning.err in the cases where correlation did not
  succeed

Reported-by: InsanePrawn <insane.prawny@gmail.com>
2021-01-25 00:16:01 +01:00
Christian Schwarz
c3d87289bb [#388] util/semaphore: fix TestSemaphore test
fixes #388
2021-01-24 22:28:47 +01:00
Christian Schwarz
0d96627ffb [#388] trace: make WithTaskGroup actually concurrent 2021-01-24 22:28:47 +01:00
Christian Schwarz
61acc7494a [#388] endpoint: fix incorrect use of trace.WithTaskGroup in ListAbstractionsStreamed
Capturing behavior was broken.
2021-01-24 13:58:23 +01:00
Rafał Bugajewski
96d5288667
docs: fix typos 2020-12-17 12:00:29 +01:00
Christian Schwarz
b8cd3c59f1 docs: update supporters 2020-11-11 16:06:54 +01:00
Christian Schwarz
91632b52bb build: circleci: periodic full build: only build each of the matching branches once 2020-11-04 17:53:54 +01:00
Christian Schwarz
d39c0e3745 docs + readme: actually fix Patreon badge
see https://github.com/endel/shieldsio-patreon/issues/8#issuecomment-700144629
2020-11-01 16:10:48 +01:00
Christian Schwarz
180eaea195 docs: 0.3.1 changelog 2020-11-01 14:19:03 +01:00
Christian Schwarz
69ed2d7117 docs + readme: fix Patreon badge 2020-11-01 14:18:36 +01:00
Christian Schwarz
c420f3c909 [#381] zfs: ListFilesystemVersions: make list filesystems version invocation deterministic
fixes #381
ref #379
2020-11-01 13:59:21 +01:00
Christian Schwarz
0a2dea05a9 [#385] status + replication: warning if replication succeeeded without any filesystem being replicated
refs #385
refs #384
2020-11-01 13:51:28 +01:00
Christian Schwarz
293c89d392 [#385] replication: report AttemptDone if no filesystems are replicated
fixes #385
2020-11-01 13:51:28 +01:00
Jeremy Bryan Smith
bb5ef0c8b2 docs: fix link to template.sh sample hook file 2020-11-01 10:45:17 +01:00
Christian Schwarz
8f44cdc284 build: rpm: don't strip binaries
fixes build failure for aarch64

```
make rpm-docker GOARCH=arm64 GOOS=linux

+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip /usr/bin/strip
/usr/bin/strip: Unable to recognise the format of the input file `/build/src/artifacts/rpmbuild/BUILDROOT/zrepl-v0.3.0.38.g53028ed5-1.aarch64/usr/bin/zrepl'
error: Bad exit status from /var/tmp/rpm-tmp.g8Fdbs (%install)
    Bad exit status from /var/tmp/rpm-tmp.g8Fdbs (%install)
```
2020-11-01 10:23:50 +01:00
Christian Schwarz
53028ed50a docs: gen-sphinx-versioning-flags.py: implement the stable branch logic used for 0.3.0 release 2020-09-12 14:03:45 +02:00
Christian Schwarz
17e152c601 docs: build zrepl.github.io on circleci 2020-09-12 14:03:45 +02:00
Christian Schwarz
2ba6aabd4a build: circleci: quickcheck-docs: upload artifacts 2020-09-06 17:28:16 +02:00
Christian Schwarz
98207c904d docs + README: document new build & release process 2020-09-06 17:28:16 +02:00
Christian Schwarz
d2b825a8ae build: daily release builds of branches stable, master, problame/circleci-build 2020-09-06 17:28:16 +02:00
Christian Schwarz
913b8b37fe build: circleci: include pipeline number in minio url 2020-09-06 17:22:44 +02:00
Christian Schwarz
f494c5ba31 build: pin Go version in circleci 2020-09-06 17:15:32 +02:00
Christian Schwarz
b9503685f0 build: parametrized pipeline (no more approval for release build), embed exact go version info in artifacts
refs #378
refs #377
2020-09-06 15:39:34 +02:00
Christian Schwarz
0f4143c0e0 docs: update supporters 2020-09-05 17:49:35 +02:00
Christian Schwarz
8839ed1f95 docs: update multi-job & multi-host setup section 2020-09-05 17:45:18 +02:00
Christian Schwarz
41b4038ad5 docs: add example setup 'local disk backup' to jobs overview table 2020-09-05 17:44:46 +02:00
Christian Schwarz
88d21eb23a Makefile: docs: treat Sphinx warnings as errors 2020-09-05 16:41:22 +02:00
Christian Schwarz
e0be7e4d4f docs: installation/rpm-repos: remove ineffective literalinclude directive 2020-09-05 16:40:33 +02:00
Christian Schwarz
0c189265e8 platformtests: fix skipping encryption-only tests on systems that don't support encryption
(Or split the test into two tests  of which one is skipped depending on encryption support)
2020-09-05 16:04:34 +02:00
Christian Schwarz
b1f8cdf385 [#373] pruning: add optional regex field to last_n rule
fixes #373
2020-09-02 22:45:44 +02:00
Christian Schwarz
428a60870a pruning: cleanup retention grid impl + tests + correct docs
package is now at 95% code coverage and the additional tests codify
all behavior specified in the docs

There is a slight change in behavior:
Intervals are now [duration) instead of (duration].
If the leftmost interval is not keep=all, the most recently created
snapshot will be destroyed if there are other snapshots within
that first interval.
Since we recommend keep=all all over the docs, and zrepl 0.3
will put holds on that snapshot if it is being replicated,
I feel like this is an acceptable change in behavior.

refs #292
fixup of 0bbe2befce
2020-09-02 22:45:44 +02:00
Christian Schwarz
af2d6579c5 [#347] zfscmd: fix dangling trace Task on .Start() failure
fixes #347
2020-09-02 22:45:44 +02:00
Christian Schwarz
0f3da73ef1 [#347] zfscmd + zfs: define .Start() semantics, apply to call sites in pkg zfs
fixes #347
2020-09-02 22:45:44 +02:00
Christian Schwarz
fecc9416ab [#347] package trace: envconst-configurable debug mode 2020-09-02 22:45:44 +02:00
Christian Schwarz
a7915db4c3 [#347] package trace: printing debugString before instead of at panic (fixup e500d9e) 2020-09-02 22:45:44 +02:00
Christian Schwarz
3a4e841c73 [#292] pruning: grid: add all snapshots that do not match the regex to the rule's destroy list
Before this patch, multiple grids with disjoint regexes would result in
no snapshots being destroyed at all.

fixes #292
2020-09-02 22:45:44 +02:00
Christian Schwarz
bcf6ff1c08 [#292] pruning: add func MustNewKeepGrid 2020-09-02 22:45:44 +02:00
Christian Schwarz
ad7a104ab4 [#292] pruning: simplify table test driver 2020-09-02 22:45:44 +02:00
Christian Schwarz
639359f393 [#292] pruning: last_n: use snap name as fallback when creation is equal 2020-09-02 22:45:44 +02:00
Christian Schwarz
91e310b7e3 build: rpm + deb targets, build-in-docker targets, CircleCI pipeline rewrite
Co-authored-by: Armin Wehrfritz <dkxls23@gmail.com>
2020-09-02 21:34:52 +02:00