Commit Graph

1102 Commits

Author SHA1 Message Date
Christian Schwarz
3a4e841c73 [#292] pruning: grid: add all snapshots that do not match the regex to the rule's destroy list
Before this patch, multiple grids with disjoint regexes would result in
no snapshots being destroyed at all.

fixes #292
2020-09-02 22:45:44 +02:00
Christian Schwarz
bcf6ff1c08 [#292] pruning: add func MustNewKeepGrid 2020-09-02 22:45:44 +02:00
Christian Schwarz
ad7a104ab4 [#292] pruning: simplify table test driver 2020-09-02 22:45:44 +02:00
Christian Schwarz
639359f393 [#292] pruning: last_n: use snap name as fallback when creation is equal 2020-09-02 22:45:44 +02:00
Christian Schwarz
91e310b7e3 build: rpm + deb targets, build-in-docker targets, CircleCI pipeline rewrite
Co-authored-by: Armin Wehrfritz <dkxls23@gmail.com>
2020-09-02 21:34:52 +02:00
Christian Schwarz
5b30ad01ce make formatcheck: exit with non-zero status code if unformatted files are found
fixup of 7a5883d404
2020-09-02 21:33:36 +02:00
chenrui
8f4f9338a9
build: remove travis config
as I did not find it got referenced in the PR builds
2020-09-01 19:21:59 +02:00
Christian Schwarz
67bbce3c36
Merge pull request #375 from InsanePrawn/make_formatcheck
Add `make formatcheck` and reformat all files
2020-09-01 00:12:10 +02:00
InsanePrawn
180c3d9ae1 Reformat all files with make format.
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:57:45 +02:00
InsanePrawn
7a5883d404 circleci: Add make formatcheck
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:57:45 +02:00
InsanePrawn
c90acefacb Makefile: Add formatcheck target
- `make formatcheck` outputs the list of files that need formatting
  - `make format` applies the formatting and now also outputs the diffs.

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 23:53:27 +02:00
InsanePrawn
d6799e08d8 lazy.sh: various fixes (#372)
- Emit warning and return 1 when no cmd is specified
- get docdep requirements from lazy.sh parent dir, not $GOPATH/src/zrepl/zrepl
- Explicitly warn when gopath is not in PATH
- Fix `release` cmd
- replace `! -z` with `-n`

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-31 16:40:13 +02:00
Christian Schwarz
c44dccc34b build: circleci: switch to 'machine' executor for test-build-in-docker job
refs #356
2020-08-26 11:47:17 +02:00
Christian Schwarz
7f1695c457 docs: transport: fix easyrsa script (fixup of 6b4c6fc) 2020-08-23 20:36:43 +02:00
Christian Schwarz
e500d9ee26 package trace: track activeChildTasks in a set if debugEnabled=true
refs #358
2020-08-23 20:13:58 +02:00
Christian Schwarz
d5ce578929 [#358] platformtest: ReceiveForceIntoEncryptedErr: fix trace panic caused by dangling open zfs send process
```
panic: end task: 1 active child tasks: end task: task still has active child tasks

goroutine 1 [running]:
github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1.1(0xc000020680, 0xc000080f00)
	/home/cs/zrepl/zrepl/daemon/logging/trace/trace.go:221 +0x2f7
github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1()
	/home/cs/zrepl/zrepl/daemon/logging/trace/trace.go:237 +0x38
main.HarnessRun(0x7ffe0b49ad3a, 0x11, 0x7ffe0b49acf2, 0x1a, 0xc800000, 0x7ffe0b49ad19, 0x16, 0x1, 0x0, 0x0, ...)
	/home/cs/zrepl/zrepl/platformtest/harness/harness.go:168 +0xe54
main.main()
	/home/cs/zrepl/zrepl/platformtest/harness/harness.go:41 +0x2ef
```

fixes #358
2020-08-23 20:13:58 +02:00
Christian Schwarz
3f8fe3a368 [#346] do not rely on creation property for filesystem version diff
fixes #346
fixes #358
2020-08-23 19:21:55 +02:00
Christian Schwarz
6b4c6fc062 [#357] docs: update quickstart + tls transport to produce keypairs with subject alternative names
fixes #357
2020-08-22 03:05:30 +02:00
Christian Schwarz
e239d6f633 build: make platformtest-* usable
* use plain `go build` instead of the `go test` + `func TestMain` hack
* add target cover-platform and use that to generate coverage reports
2020-08-21 23:01:35 +02:00
InsanePrawn
0bbe2befce docs: prune: add prune interval visualisation
fixes #122

Co-Authored-By: Christian Schwarz <me@cschwarz.com>

Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-21 22:05:05 +02:00
InsanePrawn
fa4e048169 readme: fix typo
Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
2020-08-21 22:05:05 +02:00
Christian Schwarz
4f9f21f7f2 logger: fix go-1.15-discovered conversion from int to string
logger/datastructures.go:85:10: conversion from Level (int) to string yields a string of one rune
2020-08-12 21:44:02 +02:00
Christian Schwarz
480176ba2d rpc/dataconn: fix go1.15-discovered recursive Error() method impl
rpc/dataconn/dataconn_client.go:69:9: Sprintf format %s with arg e causes recursive Error method call
2020-08-12 21:41:31 +02:00
Christian Schwarz
1190c0f6d2 docs: supporters: update 2020-08-12 21:38:23 +02:00
Christian Schwarz
720a284db5 dist/grafana: fix endpoint abstractions cache metric panel
fixup of 30cdc14
2020-08-04 01:36:31 +02:00
Hans Schulz
83fdffbcef replication: prometheus metric for number of failed replications in last attempt
- package replication: metric
- Grafana panel
- wiring
- changelog

Signed-off-by: Christian Schwarz <me@cschwarz.com>

closes #341
2020-08-04 01:19:44 +02:00
Christian Schwarz
0ee7a49d31 [#289] zfs: workaround for OpenZFS 0.7 dry send info with zero estimated size
fixes #289
2020-07-26 20:32:35 +02:00
Christian Schwarz
02db5994fe [#345] fix broken identification of parent-fs for initial replication ordering
fixup of 02807279

fixes #345
2020-07-26 20:32:35 +02:00
Christian Schwarz
1d7a84e8ae build: extract debian binary packaging workflow trigger into a reusable script 2020-07-26 20:32:35 +02:00
Christian Schwarz
a0e3dc7040 [#348] replication: add platformtest to check behavior on recv fail while still sending
Regression test for #348
2020-07-26 20:32:35 +02:00
Christian Schwarz
43495d70c7 [#348] replication: return errors if .Send rpc returns a nil SendRes
discovered while developing the next commit in this series
2020-07-26 20:32:35 +02:00
Christian Schwarz
fa586b493c [#348] fix crash on early-recv error
* The SendStream.Close() was not called by dataconn.Server, which left the zfs send process dangling.
* When the source job's ctx interceptor closed the task, the dangling zfs send was detect by the trace package and panicked.

    020-07-25T19:54:41-04:00 [ERRO][latitude][rpc.data][cyZj$J3Ca$J3Ca.CJwB]: cannot write send stream err="frameconn: shutting down"
    panic: end task: 1 active child tasks: end task: task still has active child tasks

    goroutine 196966 [running]:
    github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1.1(0xc000320680, 0xcde000)
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace.go:221 +0x2f7
    github.com/zrepl/zrepl/daemon/logging/trace.WithTask.func1()
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace.go:237 +0x38
    github.com/zrepl/zrepl/daemon/logging/trace.WithTaskAndSpan.func1()
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/logging/trace/trace_convenience.go:41 +0x37
    github.com/zrepl/zrepl/daemon/job.(*PassiveSide).Run.func1(0xdcf780, 0xc0000a3560, 0xdc65a0, 0xc00035e620, 0xc0000a34d0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/daemon/job/passive.go:194 +0x2e7
    github.com/zrepl/zrepl/rpc.NewServer.func3(0xdcf780, 0xc0001ce4b0, 0xdc65e0, 0xc00035e600, 0xc0000a34d0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/rpc_server.go:82 +0xd5
    github.com/zrepl/zrepl/rpc/dataconn.(*Server).serveConn(0xc0000a2ba0, 0xc00018eca0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:149 +0x3be
    github.com/zrepl/zrepl/rpc/dataconn.(*Server).Serve.func3(0xc0000b8180, 0xc0000a2ba0, 0xc00018eca0)
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:108 +0x5d
    created by github.com/zrepl/zrepl/rpc/dataconn.(*Server).Serve
            /home/jeremy/go/src/github.com/zrepl/zrepl/rpc/dataconn/dataconn_server.go:106 +0x24a
    2020-07-25T19:58:55-04:00 [ERRO][latitude][rpc.data][Pt4F$gCWT$gCWT.fzhc]: cannot write send stream err="frameconn: shutting down"
    panic: end task: 1 active child tasks: end task: task still has active child tasks

fixes #348
2020-07-26 20:32:35 +02:00
Christian Schwarz
30cdc1430e replication + endpoint: replication guarantees: guarantee_{resumability,incremental,nothing}
This commit

- adds a configuration in which no step holds, replication cursors, etc. are created
- removes the send.step_holds.disable_incremental setting
- creates a new config option `replication` for active-side jobs
- adds the replication.protection.{initial,incremental} settings, each
  of which can have values
    - `guarantee_resumability`
    - `guarantee_incremental`
    - `guarantee_nothing`
  (refer to docs/configuration/replication.rst for semantics)

The `replication` config from an active side is sent to both endpoint.Sender and endpoint.Receiver
for each replication step. Sender and Receiver then act accordingly.

For `guarantee_incremental`, we add the new `tentative-replication-cursor` abstraction.
The necessity for that abstraction is outlined in https://github.com/zrepl/zrepl/issues/340.

fixes https://github.com/zrepl/zrepl/issues/340
2020-07-26 20:32:35 +02:00
Christian Schwarz
27673a23e9 config: add test for fromdefaults behavior 2020-07-26 20:32:35 +02:00
Christian Schwarz
95fc299733 daemon/job: test that sample configs are buildable 2020-07-26 20:32:35 +02:00
Christian Schwarz
4e702eedc9 cmd: zfs-abstraction list --json: fix panic
(was panicking because `abstractions` is in fact a channel
2020-07-26 20:32:35 +02:00
Christian Schwarz
8ff83f2f1a [#342] endpoint: always create unencrypted placeholder filesystems
This "breaks" the use case of receiving an unencrypted send into an encrypted receiver by setting the receiver's `root_fs`'s `encryption=on`.
"breaks" in air-quotes because we have not yet released a version of
zrepl with encrypted send support.

We will bring back the featured outlined above in a future release.
See https://github.com/zrepl/zrepl/issues/342#issuecomment-657231818 and following.
2020-07-26 20:32:35 +02:00
Christian Schwarz
4b8f0ad112 docs: supporters: update 2020-06-22 13:36:00 +02:00
Brian Candler
dbc8bbeb6a docs: config: prune: example: keep manual snapshots on receiver
Fixes #335
closes #336

Signed-off-by: Christian Schwarz <me@cschwarz.com>
2020-06-22 12:32:03 +02:00
Christian Schwarz
b3e856f40d docs: changelog: 0.3: fix broken issue link 2020-06-22 12:30:42 +02:00
Christian Schwarz
8e1937fe75 doc: fixup 0.3 changelog 05f1237a6d 2020-06-14 18:29:37 +02:00
Christian Schwarz
073514fc21 docs/publish.sh: only render latest (patch+rc) version for each (major,minor) versio. 2020-06-14 18:24:20 +02:00
Christian Schwarz
dab222d95f docs: GitHub Sponsors link 2020-06-14 15:26:05 +02:00
Christian Schwarz
a827894274 docs: add backup-to-external-disk quick-start guide and convert existing tutorial to quick-start guide
refs #219
fixes #329
2020-06-14 15:26:05 +02:00
Christian Schwarz
9a8d813d14 docs: fix typo in cli help for zfs-abstraction subcommand 2020-06-14 15:26:05 +02:00
Christian Schwarz
e391fa94f9 dist/grafana: update grafana dashboard
- uses version metric for 'instances up'
- displays active task count
- displays send abstractions cache entry count
- in general, graphs have a shorter y axis for better overview

fixes #332
2020-06-14 15:26:05 +02:00
Christian Schwarz
509185dfbe prometheus: expose zrepl version as const metric 2020-06-14 15:26:05 +02:00
Christian Schwarz
4b1b7a8561 envconst: queryable report of resolved variables + integration inot zrepl status --raw
fixes #299
refs #186
2020-06-14 15:26:05 +02:00
Christian Schwarz
b330ccca5d transport/ssh: bump go-netssh version to fix ssh client process leaks
fixes #322
2020-06-14 15:26:05 +02:00
Christian Schwarz
05f1237a6d docs: 0.3 changelog 2020-06-14 15:26:05 +02:00