Commit Graph

917 Commits

Author SHA1 Message Date
Christian Schwarz
913b8b37fe build: circleci: include pipeline number in minio url 2020-09-06 17:22:44 +02:00
Christian Schwarz
f494c5ba31 build: pin Go version in circleci 2020-09-06 17:15:32 +02:00
Christian Schwarz
b9503685f0 build: parametrized pipeline (no more approval for release build), embed exact go version info in artifacts
refs #378
refs #377
2020-09-06 15:39:34 +02:00
Christian Schwarz
0f4143c0e0 docs: update supporters 2020-09-05 17:49:35 +02:00
Christian Schwarz
8839ed1f95 docs: update multi-job & multi-host setup section 2020-09-05 17:45:18 +02:00
Christian Schwarz
41b4038ad5 docs: add example setup 'local disk backup' to jobs overview table 2020-09-05 17:44:46 +02:00
Christian Schwarz
88d21eb23a Makefile: docs: treat Sphinx warnings as errors 2020-09-05 16:41:22 +02:00
Christian Schwarz
e0be7e4d4f docs: installation/rpm-repos: remove ineffective literalinclude directive 2020-09-05 16:40:33 +02:00
Christian Schwarz
0c189265e8 platformtests: fix skipping encryption-only tests on systems that don't support encryption
(Or split the test into two tests  of which one is skipped depending on encryption support)
2020-09-05 16:04:34 +02:00
Christian Schwarz
b1f8cdf385 [#373] pruning: add optional regex field to last_n rule
fixes #373
2020-09-02 22:45:44 +02:00
Christian Schwarz
428a60870a pruning: cleanup retention grid impl + tests + correct docs
package is now at 95% code coverage and the additional tests codify
all behavior specified in the docs

There is a slight change in behavior:
Intervals are now [duration) instead of (duration].
If the leftmost interval is not keep=all, the most recently created
snapshot will be destroyed if there are other snapshots within
that first interval.
Since we recommend keep=all all over the docs, and zrepl 0.3
will put holds on that snapshot if it is being replicated,
I feel like this is an acceptable change in behavior.

refs #292
fixup of 0bbe2befce
2020-09-02 22:45:44 +02:00
Christian Schwarz
af2d6579c5 [#347] zfscmd: fix dangling trace Task on .Start() failure
fixes #347
2020-09-02 22:45:44 +02:00
Christian Schwarz
0f3da73ef1 [#347] zfscmd + zfs: define .Start() semantics, apply to call sites in pkg zfs
fixes #347
2020-09-02 22:45:44 +02:00
Christian Schwarz
fecc9416ab [#347] package trace: envconst-configurable debug mode 2020-09-02 22:45:44 +02:00
Christian Schwarz
a7915db4c3 [#347] package trace: printing debugString before instead of at panic (fixup e500d9e) 2020-09-02 22:45:44 +02:00
Christian Schwarz
3a4e841c73 [#292] pruning: grid: add all snapshots that do not match the regex to the rule's destroy list
Before this patch, multiple grids with disjoint regexes would result in
no snapshots being destroyed at all.

fixes #292
2020-09-02 22:45:44 +02:00
Christian Schwarz
bcf6ff1c08 [#292] pruning: add func MustNewKeepGrid 2020-09-02 22:45:44 +02:00
Christian Schwarz
ad7a104ab4 [#292] pruning: simplify table test driver 2020-09-02 22:45:44 +02:00
Christian Schwarz
639359f393 [#292] pruning: last_n: use snap name as fallback when creation is equal 2020-09-02 22:45:44 +02:00
Christian Schwarz
91e310b7e3 build: rpm + deb targets, build-in-docker targets, CircleCI pipeline rewrite
Co-authored-by: Armin Wehrfritz <dkxls23@gmail.com>
2020-09-02 21:34:52 +02:00
Christian Schwarz
5b30ad01ce make formatcheck: exit with non-zero status code if unformatted files are found
fixup of 7a5883d404
2020-09-02 21:33:36 +02:00
chenrui
8f4f9338a9
build: remove travis config
as I did not find it got referenced in the PR builds
2020-09-01 19:21:59 +02:00
Christian Schwarz
67bbce3c36
Merge pull request #375 from InsanePrawn/make_formatcheck
Add `make formatcheck` and reformat all files
2020-09-01 00:12:10 +02:00
InsanePrawn
180c3d9ae1 Reformat all files with make format.
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:57:45 +02:00
InsanePrawn
7a5883d404 circleci: Add make formatcheck
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:57:45 +02:00
InsanePrawn
c90acefacb Makefile: Add formatcheck target
- `make formatcheck` outputs the list of files that need formatting
  - `make format` applies the formatting and now also outputs the diffs.

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:53:27 +02:00
InsanePrawn
d6799e08d8 lazy.sh: various fixes (#372)
- Emit warning and return 1 when no cmd is specified
- get docdep requirements from lazy.sh parent dir, not $GOPATH/src/zrepl/zrepl
- Explicitly warn when gopath is not in PATH
- Fix `release` cmd
- replace `! -z` with `-n`

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 16:40:13 +02:00
Christian Schwarz
c44dccc34b build: circleci: switch to 'machine' executor for test-build-in-docker job
refs #356
2020-08-26 11:47:17 +02:00
Christian Schwarz
7f1695c457 docs: transport: fix easyrsa script (fixup of 6b4c6fc) 2020-08-23 20:36:43 +02:00
Christian Schwarz
e500d9ee26 package trace: track activeChildTasks in a set if debugEnabled=true
refs #358
2020-08-23 20:13:58 +02:00
Christian Schwarz
d5ce578929 [#358] platformtest: ReceiveForceIntoEncryptedErr: fix trace panic caused by dangling open zfs send process
```
panic: end task: 1 active child tasks: end task: task still has active child tasks

goroutine 1 [running]:
github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1.1(0xc000020680, 0xc000080f00)
	/home/cs/zrepl/zrepl/daemon/logging/trace/trace.go:221 +0x2f7
github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1()
	/home/cs/zrepl/zrepl/daemon/logging/trace/trace.go:237 +0x38
main.HarnessRun(0x7ffe0b49ad3a, 0x11, 0x7ffe0b49acf2, 0x1a, 0xc800000, 0x7ffe0b49ad19, 0x16, 0x1, 0x0, 0x0, ...)
	/home/cs/zrepl/zrepl/platformtest/harness/harness.go:168 +0xe54
main.main()
	/home/cs/zrepl/zrepl/platformtest/harness/harness.go:41 +0x2ef
```

fixes #358
2020-08-23 20:13:58 +02:00
Christian Schwarz
3f8fe3a368 [#346] do not rely on creation property for filesystem version diff
fixes #346
fixes #358
2020-08-23 19:21:55 +02:00
Christian Schwarz
6b4c6fc062 [#357] docs: update quickstart + tls transport to produce keypairs with subject alternative names
fixes #357
2020-08-22 03:05:30 +02:00
Christian Schwarz
e239d6f633 build: make platformtest-* usable
* use plain `go build` instead of the `go test` + `func TestMain` hack
* add target cover-platform and use that to generate coverage reports
2020-08-21 23:01:35 +02:00
InsanePrawn
0bbe2befce docs: prune: add prune interval visualisation
fixes #122

Co-Authored-By: Christian Schwarz <me@cschwarz.com>

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-21 22:05:05 +02:00
InsanePrawn
fa4e048169 readme: fix typo
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-21 22:05:05 +02:00
Christian Schwarz
4f9f21f7f2 logger: fix go-1.15-discovered conversion from int to string
logger/datastructures.go:85:10: conversion from Level (int) to string yields a string of one rune
2020-08-12 21:44:02 +02:00
Christian Schwarz
480176ba2d rpc/dataconn: fix go1.15-discovered recursive Error() method impl
rpc/dataconn/dataconn_client.go:69:9: Sprintf format %s with arg e causes recursive Error method call
2020-08-12 21:41:31 +02:00
Christian Schwarz
1190c0f6d2 docs: supporters: update 2020-08-12 21:38:23 +02:00
Christian Schwarz
720a284db5 dist/grafana: fix endpoint abstractions cache metric panel
fixup of 30cdc14
2020-08-04 01:36:31 +02:00
Hans Schulz
83fdffbcef replication: prometheus metric for number of failed replications in last attempt
- package replication: metric
- Grafana panel
- wiring
- changelog

Signed-off-by: Christian Schwarz <me@cschwarz.com>

closes #341
2020-08-04 01:19:44 +02:00
Christian Schwarz
0ee7a49d31 [#289] zfs: workaround for OpenZFS 0.7 dry send info with zero estimated size
fixes #289
2020-07-26 20:32:35 +02:00
Christian Schwarz
02db5994fe [#345] fix broken identification of parent-fs for initial replication ordering
fixup of 02807279

fixes #345
2020-07-26 20:32:35 +02:00
Christian Schwarz
1d7a84e8ae build: extract debian binary packaging workflow trigger into a reusable script 2020-07-26 20:32:35 +02:00
Christian Schwarz
a0e3dc7040 [#348] replication: add platformtest to check behavior on recv fail while still sending
Regression test for #348
2020-07-26 20:32:35 +02:00
Christian Schwarz
43495d70c7 [#348] replication: return errors if .Send rpc returns a nil SendRes
discovered while developing the next commit in this series
2020-07-26 20:32:35 +02:00
Christian Schwarz
fa586b493c [#348] fix crash on early-recv error
* The SendStream.Close() was not called by dataconn.Server, which left the zfs send process dangling.
* When the source job's ctx interceptor closed the task, the dangling zfs send was detect by the trace package and panicked.

    020-07-25T19:54:41-04:00 [ERRO][latitude][rpc.data][cyZj$J3Ca$J3Ca.CJwB]: cannot write send stream err="frameconn: shutting down"
    panic: end task: 1 active child tasks: end task: task still has active child tasks

    goroutine 196966 [running]:
    github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1.1(0xc000320680, 0xcde000)
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace.go:221 +0x2f7
    github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1()
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace.go:237 +0x38
    github.com/zrepl/zrepl/daemon/logging/trace.WithTaskAndSpan.func1()
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace_convenience.go:41 +0x37
    github.com/zrepl/zrepl/daemon/job.(*PassiveSide).Run.func1(0xdcf780, 0xc0000a3560, 0xdc65a0, 0xc00035e620, 0xc0000a34d0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/job/passive.go:194 +0x2e7
    github.com/zrepl/zrepl/rpc.NewServer.func3(0xdcf780, 0xc0001ce4b0, 0xdc65e0, 0xc00035e600, 0xc0000a34d0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/rpc_server.go:82 +0xd5
    github.com/zrepl/zrepl/rpc/dataconn.(*Server).serveConn(0xc0000a2ba0, 0xc00018eca0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:149 +0x3be
    github.com/zrepl/zrepl/rpc/dataconn.(*Server).Serve.func3(0xc0000b8180, 0xc0000a2ba0, 0xc00018eca0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:108 +0x5d
    created by github.com/zrepl/zrepl/rpc/dataconn.(*Server).Serve
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:106 +0x24a
    2020-07-25T19:58:55-04:00 [ERRO][latitude][rpc.data][Pt4F$gCWT$gCWT.fzhc]: cannot write send stream err="frameconn: shutting down"
    panic: end task: 1 active child tasks: end task: task still has active child tasks

fixes #348
2020-07-26 20:32:35 +02:00
Christian Schwarz
30cdc1430e replication + endpoint: replication guarantees: guarantee_{resumability,incremental,nothing}
This commit

- adds a configuration in which no step holds, replication cursors, etc. are created
- removes the send.step_holds.disable_incremental setting
- creates a new config option `replication` for active-side jobs
- adds the replication.protection.{initial,incremental} settings, each
  of which can have values
    - `guarantee_resumability`
    - `guarantee_incremental`
    - `guarantee_nothing`
  (refer to docs/configuration/replication.rst for semantics)

The `replication` config from an active side is sent to both endpoint.Sender and endpoint.Receiver
for each replication step. Sender and Receiver then act accordingly.

For `guarantee_incremental`, we add the new `tentative-replication-cursor` abstraction.
The necessity for that abstraction is outlined in https://github.com/zrepl/zrepl/issues/340.

fixes https://github.com/zrepl/zrepl/issues/340
2020-07-26 20:32:35 +02:00
Christian Schwarz
27673a23e9 config: add test for fromdefaults behavior 2020-07-26 20:32:35 +02:00
Christian Schwarz
95fc299733 daemon/job: test that sample configs are buildable 2020-07-26 20:32:35 +02:00