Commit Graph

29 Commits

Author SHA1 Message Date
Christian Schwarz
74445a0017 fixup 2018-08-08 13:12:50 +02:00
Christian Schwarz
a0b320bfeb streamrpc now requires net.Conn => use it instead of rwc everywhere 2018-08-08 13:09:51 +02:00
Christian Schwarz
1826535e6f WIP 2018-07-15 17:36:53 +02:00
Christian Schwarz
8cca0a8547 Initial working version
Summary:
* Logging is still bad
* test output in a lot of placed
* FIXMEs every where

Test Plan: None, just review

Differential Revision: https://phabricator.cschwarz.com/D2
2018-06-24 10:44:00 +02:00
Christian Schwarz
aa3865d0a3 daemon: Job types as dedicated type
refs #67
2018-04-05 22:22:55 +02:00
Christian Schwarz
0895e02844 daemon: Task: track relation to parent job
refs #67
2018-04-05 22:18:22 +02:00
Christian Schwarz
aa92261ea7 bookmarking: prune policy for bookmarks
refs #34
2018-02-17 20:48:31 +01:00
Christian Schwarz
94967b596c docs: document changes to ssh+stdinserver transport implementation: ccd062e 2018-02-17 15:16:29 +01:00
Christian Schwarz
6b5bd0a43c job pull + source: fix broken connection teardown
Issue #56 shows zombie SSH processes.
We fix this by actually Close()ing the RWC in job pull.
If this fixes #56 it also fixes #6 --- it's the same issue.

Additionally, debugging around this revealed another issue: just
Close()ing the sshbytestream in job source will apparently outpace the
normal data stream of stdin and stdout (URG or PUSH flags?).  leading
to ugly errors in the logs.
With proper TCP connections, we would simply set the connection to
linger and close it, letting the kernel handle the final timeout. Meh.

refs #56
refs #6
2018-02-16 20:57:27 +01:00
Christian Schwarz
921bccb960 job source: use task logger 2018-02-15 23:51:57 +01:00
Christian Schwarz
63fa7a67e9 job source: refactor + use Task API
refs #10
2017-12-27 18:34:24 +01:00
Christian Schwarz
58ee796394 adopt Task API: infect datastructures
refs #10
2017-12-27 14:39:46 +01:00
Christian Schwarz
8c7e373049 daemon: DaemonStatus + JobStatus + dummy implementation
refs #10
2017-12-27 14:39:46 +01:00
Christian Schwarz
51af880701 refactor: parametrize PrefixFilter VersionType check
refs #34
2017-11-13 10:59:22 +01:00
Christian Schwarz
40919d06c2 source job: fix errnous log message when accept() on closed listener 2017-10-05 21:19:42 +02:00
Christian Schwarz
83d450b1f2 config: support days (d) and weeks (w) in durations
fixes #18
2017-10-05 15:17:37 +02:00
Christian Schwarz
3e647c14c0 config: source job: rename field 'datasets' to 'filesystems'
While filesystems is also not the right term (since it excludes ZVOLs),
we want to stay consistent with comments & terminology used in docs.

BREAK CONFIG

fixes #17
2017-10-05 13:39:05 +02:00
Christian Schwarz
c31ec8c646 convert more code to structured logging 2017-09-23 17:52:29 +02:00
Christian Schwarz
9465b593f9 cmd: configurable logrus formatters
We lost the nice context-stack [jobname][taskname][...] at the beginning
of each log line when switching to logrus.

Define some field names that define these contexts.
Write a human-friendly formatter that presents these field names like
the solution we had before logrus.

Write some other formatters for logfmt and json output along the way.

Limit ourselves to stdout logging for now.
2017-09-23 11:24:36 +02:00
Christian Schwarz
bfcba7b281 cmd: logging using logrus 2017-09-22 17:01:54 +02:00
Christian Schwarz
9cd83399d3 cmd: remove global state in main.go
* refactoring
* Now supporting default config locations
2017-09-17 18:32:00 +02:00
Christian Schwarz
6a05e101cf WIP daemon:
Implement
* pruning on source side
* local job
* test subcommand for doing a dry-run of a prune policy

* use a non-blocking callback from autosnap to trigger the depending
jobs -> avoids races, looks saner in the debug log
2017-09-16 21:13:19 +02:00
Christian Schwarz
e3ec093d53 cmd: handler: check FilesystemVersionFilter as part of ACL 2017-09-16 20:24:46 +02:00
Christian Schwarz
e70b6f3071 WIP: recurring jobs
Done:

* implement autosnapper that asserts interval between snapshots
* implement pruner

* job pull: pulling + pruning
* job source: autosnapping + serving

TODO

* job source: pruning
* job local: everything
* fatal errors such as serve that cannot bind socket must be more
visible
* couldn't things that need a snapshotprefix just use a interface
Prefixer() instead? then we could have prefixsnapshotfilter and not
duplicate it every time...
* either go full context.Context or not at all...? just wait because
community climate around it isn't that great and we only need it for
cancellation? roll our own?
2017-09-15 19:35:19 +02:00
Christian Schwarz
0acb2e9ec0 cmd: fix missing error message 2017-09-15 19:32:09 +02:00
Christian Schwarz
1deaa459c8 config: unify job debugging options 2017-09-11 15:45:10 +02:00
Christian Schwarz
d76d3db0b3 handler: remove unused SinkMappingFunc 2017-09-11 13:51:19 +02:00
Christian Schwarz
0a53b2415f signal handling for source job 2017-09-11 13:50:35 +02:00
Christian Schwarz
73c9033583 WIP: Switch to new config format.
Don't use jobrun for daemon, just call JobDo() once, the job must
organize stuff itself.

Sacrifice all the oneshot commands, they will be reintroduced as
client-calls to the daemon.
2017-09-10 17:53:54 +02:00