Commit Graph

1009 Commits

Author SHA1 Message Date
0c189265e8 platformtests: fix skipping encryption-only tests on systems that don't support encryption
(Or split the test into two tests  of which one is skipped depending on encryption support)
2020-09-05 16:04:34 +02:00
b1f8cdf385 [#373] pruning: add optional regex field to last_n rule
fixes #373
2020-09-02 22:45:44 +02:00
428a60870a pruning: cleanup retention grid impl + tests + correct docs
package is now at 95% code coverage and the additional tests codify
all behavior specified in the docs

There is a slight change in behavior:
Intervals are now [duration) instead of (duration].
If the leftmost interval is not keep=all, the most recently created
snapshot will be destroyed if there are other snapshots within
that first interval.
Since we recommend keep=all all over the docs, and zrepl 0.3
will put holds on that snapshot if it is being replicated,
I feel like this is an acceptable change in behavior.

refs #292
fixup of 0bbe2befce
2020-09-02 22:45:44 +02:00
af2d6579c5 [#347] zfscmd: fix dangling trace Task on .Start() failure
fixes #347
2020-09-02 22:45:44 +02:00
0f3da73ef1 [#347] zfscmd + zfs: define .Start() semantics, apply to call sites in pkg zfs
fixes #347
2020-09-02 22:45:44 +02:00
fecc9416ab [#347] package trace: envconst-configurable debug mode 2020-09-02 22:45:44 +02:00
a7915db4c3 [#347] package trace: printing debugString before instead of at panic (fixup e500d9e) 2020-09-02 22:45:44 +02:00
3a4e841c73 [#292] pruning: grid: add all snapshots that do not match the regex to the rule's destroy list
Before this patch, multiple grids with disjoint regexes would result in
no snapshots being destroyed at all.

fixes #292
2020-09-02 22:45:44 +02:00
bcf6ff1c08 [#292] pruning: add func MustNewKeepGrid 2020-09-02 22:45:44 +02:00
ad7a104ab4 [#292] pruning: simplify table test driver 2020-09-02 22:45:44 +02:00
639359f393 [#292] pruning: last_n: use snap name as fallback when creation is equal 2020-09-02 22:45:44 +02:00
91e310b7e3 build: rpm + deb targets, build-in-docker targets, CircleCI pipeline rewrite
Co-authored-by: Armin Wehrfritz <dkxls23@gmail.com>
2020-09-02 21:34:52 +02:00
5b30ad01ce make formatcheck: exit with non-zero status code if unformatted files are found
fixup of 7a5883d404
2020-09-02 21:33:36 +02:00
8f4f9338a9 build: remove travis config
as I did not find it got referenced in the PR builds
2020-09-01 19:21:59 +02:00
67bbce3c36 Merge pull request #375 from InsanePrawn/make_formatcheck
Add `make formatcheck` and reformat all files
2020-09-01 00:12:10 +02:00
180c3d9ae1 Reformat all files with make format.
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:57:45 +02:00
7a5883d404 circleci: Add make formatcheck
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:57:45 +02:00
c90acefacb Makefile: Add formatcheck target
- `make formatcheck` outputs the list of files that need formatting
  - `make format` applies the formatting and now also outputs the diffs.

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:53:27 +02:00
d6799e08d8 lazy.sh: various fixes (#372)
- Emit warning and return 1 when no cmd is specified
- get docdep requirements from lazy.sh parent dir, not $GOPATH/src/zrepl/zrepl
- Explicitly warn when gopath is not in PATH
- Fix `release` cmd
- replace `! -z` with `-n`

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 16:40:13 +02:00
c44dccc34b build: circleci: switch to 'machine' executor for test-build-in-docker job
refs #356
2020-08-26 11:47:17 +02:00
7f1695c457 docs: transport: fix easyrsa script (fixup of 6b4c6fc) 2020-08-23 20:36:43 +02:00
e500d9ee26 package trace: track activeChildTasks in a set if debugEnabled=true
refs #358
2020-08-23 20:13:58 +02:00
d5ce578929 [#358] platformtest: ReceiveForceIntoEncryptedErr: fix trace panic caused by dangling open zfs send process
```
panic: end task: 1 active child tasks: end task: task still has active child tasks

goroutine 1 [running]:
github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1.1(0xc000020680, 0xc000080f00)
	/home/cs/zrepl/zrepl/daemon/logging/trace/trace.go:221 +0x2f7
github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1()
	/home/cs/zrepl/zrepl/daemon/logging/trace/trace.go:237 +0x38
main.HarnessRun(0x7ffe0b49ad3a, 0x11, 0x7ffe0b49acf2, 0x1a, 0xc800000, 0x7ffe0b49ad19, 0x16, 0x1, 0x0, 0x0, ...)
	/home/cs/zrepl/zrepl/platformtest/harness/harness.go:168 +0xe54
main.main()
	/home/cs/zrepl/zrepl/platformtest/harness/harness.go:41 +0x2ef
```

fixes #358
2020-08-23 20:13:58 +02:00
3f8fe3a368 [#346] do not rely on creation property for filesystem version diff
fixes #346
fixes #358
2020-08-23 19:21:55 +02:00
6b4c6fc062 [#357] docs: update quickstart + tls transport to produce keypairs with subject alternative names
fixes #357
2020-08-22 03:05:30 +02:00
e239d6f633 build: make platformtest-* usable
* use plain `go build` instead of the `go test` + `func TestMain` hack
* add target cover-platform and use that to generate coverage reports
v0.3.0
2020-08-21 23:01:35 +02:00
0bbe2befce docs: prune: add prune interval visualisation
fixes #122

Co-Authored-By: Christian Schwarz <me@cschwarz.com>

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-21 22:05:05 +02:00
fa4e048169 readme: fix typo
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-21 22:05:05 +02:00
4f9f21f7f2 logger: fix go-1.15-discovered conversion from int to string
logger/datastructures.go:85:10: conversion from Level (int) to string yields a string of one rune
2020-08-12 21:44:02 +02:00
480176ba2d rpc/dataconn: fix go1.15-discovered recursive Error() method impl
rpc/dataconn/dataconn_client.go:69:9: Sprintf format %s with arg e causes recursive Error method call
2020-08-12 21:41:31 +02:00
1190c0f6d2 docs: supporters: update 2020-08-12 21:38:23 +02:00
720a284db5 dist/grafana: fix endpoint abstractions cache metric panel
fixup of 30cdc14
2020-08-04 01:36:31 +02:00
83fdffbcef replication: prometheus metric for number of failed replications in last attempt
- package replication: metric
- Grafana panel
- wiring
- changelog

Signed-off-by: Christian Schwarz <me@cschwarz.com>

closes #341
2020-08-04 01:19:44 +02:00
0ee7a49d31 [#289] zfs: workaround for OpenZFS 0.7 dry send info with zero estimated size
fixes #289
v0.3.0-rc2
2020-07-26 20:32:35 +02:00
02db5994fe [#345] fix broken identification of parent-fs for initial replication ordering
fixup of 02807279

fixes #345
2020-07-26 20:32:35 +02:00
1d7a84e8ae build: extract debian binary packaging workflow trigger into a reusable script 2020-07-26 20:32:35 +02:00
a0e3dc7040 [#348] replication: add platformtest to check behavior on recv fail while still sending
Regression test for #348
2020-07-26 20:32:35 +02:00
43495d70c7 [#348] replication: return errors if .Send rpc returns a nil SendRes
discovered while developing the next commit in this series
2020-07-26 20:32:35 +02:00
fa586b493c [#348] fix crash on early-recv error
* The SendStream.Close() was not called by dataconn.Server, which left the zfs send process dangling.
* When the source job's ctx interceptor closed the task, the dangling zfs send was detect by the trace package and panicked.

    020-07-25T19:54:41-04:00 [ERRO][latitude][rpc.data][cyZj$J3Ca$J3Ca.CJwB]: cannot write send stream err="frameconn: shutting down"
    panic: end task: 1 active child tasks: end task: task still has active child tasks

    goroutine 196966 [running]:
    github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1.1(0xc000320680, 0xcde000)
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace.go:221 +0x2f7
    github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1()
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace.go:237 +0x38
    github.com/zrepl/zrepl/daemon/logging/trace.WithTaskAndSpan.func1()
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace_convenience.go:41 +0x37
    github.com/zrepl/zrepl/daemon/job.(*PassiveSide).Run.func1(0xdcf780, 0xc0000a3560, 0xdc65a0, 0xc00035e620, 0xc0000a34d0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/job/passive.go:194 +0x2e7
    github.com/zrepl/zrepl/rpc.NewServer.func3(0xdcf780, 0xc0001ce4b0, 0xdc65e0, 0xc00035e600, 0xc0000a34d0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/rpc_server.go:82 +0xd5
    github.com/zrepl/zrepl/rpc/dataconn.(*Server).serveConn(0xc0000a2ba0, 0xc00018eca0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:149 +0x3be
    github.com/zrepl/zrepl/rpc/dataconn.(*Server).Serve.func3(0xc0000b8180, 0xc0000a2ba0, 0xc00018eca0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:108 +0x5d
    created by github.com/zrepl/zrepl/rpc/dataconn.(*Server).Serve
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:106 +0x24a
    2020-07-25T19:58:55-04:00 [ERRO][latitude][rpc.data][Pt4F$gCWT$gCWT.fzhc]: cannot write send stream err="frameconn: shutting down"
    panic: end task: 1 active child tasks: end task: task still has active child tasks

fixes #348
2020-07-26 20:32:35 +02:00
30cdc1430e replication + endpoint: replication guarantees: guarantee_{resumability,incremental,nothing}
This commit

- adds a configuration in which no step holds, replication cursors, etc. are created
- removes the send.step_holds.disable_incremental setting
- creates a new config option `replication` for active-side jobs
- adds the replication.protection.{initial,incremental} settings, each
  of which can have values
    - `guarantee_resumability`
    - `guarantee_incremental`
    - `guarantee_nothing`
  (refer to docs/configuration/replication.rst for semantics)

The `replication` config from an active side is sent to both endpoint.Sender and endpoint.Receiver
for each replication step. Sender and Receiver then act accordingly.

For `guarantee_incremental`, we add the new `tentative-replication-cursor` abstraction.
The necessity for that abstraction is outlined in https://github.com/zrepl/zrepl/issues/340.

fixes https://github.com/zrepl/zrepl/issues/340
2020-07-26 20:32:35 +02:00
27673a23e9 config: add test for fromdefaults behavior 2020-07-26 20:32:35 +02:00
95fc299733 daemon/job: test that sample configs are buildable 2020-07-26 20:32:35 +02:00
4e702eedc9 cmd: zfs-abstraction list --json: fix panic
(was panicking because `abstractions` is in fact a channel
2020-07-26 20:32:35 +02:00
8ff83f2f1a [#342] endpoint: always create unencrypted placeholder filesystems
This "breaks" the use case of receiving an unencrypted send into an encrypted receiver by setting the receiver's `root_fs`'s `encryption=on`.
"breaks" in air-quotes because we have not yet released a version of
zrepl with encrypted send support.

We will bring back the featured outlined above in a future release.
See https://github.com/zrepl/zrepl/issues/342#issuecomment-657231818 and following.
2020-07-26 20:32:35 +02:00
4b8f0ad112 docs: supporters: update 2020-06-22 13:36:00 +02:00
dbc8bbeb6a docs: config: prune: example: keep manual snapshots on receiver
Fixes #335
closes #336

Signed-off-by: Christian Schwarz <me@cschwarz.com>
2020-06-22 12:32:03 +02:00
b3e856f40d docs: changelog: 0.3: fix broken issue link 2020-06-22 12:30:42 +02:00
8e1937fe75 doc: fixup 0.3 changelog 05f1237a6d v0.3.0-rc1 2020-06-14 18:29:37 +02:00
073514fc21 docs/publish.sh: only render latest (patch+rc) version for each (major,minor) versio. 2020-06-14 18:24:20 +02:00
dab222d95f docs: GitHub Sponsors link 2020-06-14 15:26:05 +02:00