Commit Graph

818 Commits

Author SHA1 Message Date
6b5bd0a43c job pull + source: fix broken connection teardown
Issue #56 shows zombie SSH processes.
We fix this by actually Close()ing the RWC in job pull.
If this fixes #56 it also fixes #6 --- it's the same issue.

Additionally, debugging around this revealed another issue: just
Close()ing the sshbytestream in job source will apparently outpace the
normal data stream of stdin and stdout (URG or PUSH flags?).  leading
to ugly errors in the logs.
With proper TCP connections, we would simply set the connection to
linger and close it, letting the kernel handle the final timeout. Meh.

refs #56
refs #6
2018-02-16 20:57:27 +01:00
921bccb960 job source: use task logger 2018-02-15 23:51:57 +01:00
24b29a0865 Gopkg: remove unused dependencies + cleanup Gopkg.toml 2018-02-15 22:18:32 +01:00
5f2c14adab zfs: use custom datatype to pass ZFS properties in ZFSSet
refs #55
2018-01-05 18:42:10 +01:00
787675aee8 control status command: only show verbose logs on user request 2017-12-30 13:53:19 +01:00
6f68c98c16 logger.Levle: implement flag.Value 2017-12-30 13:52:51 +01:00
01e0519b7b control status subcommand: fix typo in usage 2017-12-30 13:44:55 +01:00
8742b7f763 handler: fix typo in log message 2017-12-30 13:29:04 +01:00
710bf79f7e logger.Logger: fix WithFields() dropping all but last field 2017-12-30 13:00:23 +01:00
a622ef1487 docs: promote test subcommand 2017-12-29 22:53:33 +01:00
56f13741f9 test pattern subcommand: better example command 2017-12-29 22:45:38 +01:00
746fb4ff88 build: include generate step in release build + warn of dirty git working copy 2017-12-29 22:34:14 +01:00
8473462adf build: adjust wrong path of zrepl source dir in build.Dockerfile
was symlinking /zrepl to /go/src/github.com/zrepl/zrepl earlier, forgot
to change that apparently

see 47726ad877

refs #38
2017-12-29 22:25:48 +01:00
61842988b9 Task & TaskStatus: DeepCopy(): actually copy lastUpdate field
otherwise, only changes to activity level would udpate TaskStatus
LastUpdate field

refs #10
2017-12-29 21:43:12 +01:00
be7176bee7 Puller: fix wrong filesystem log field usage
was introduced in 9465b593
2017-12-29 21:25:42 +01:00
c403e56835 fixup: broken test case for logger
refs #26
2017-12-29 21:14:49 +01:00
839eccf513 logger.Outlet: WriteEntry must not block
- make TCPOutlet fully asynchronous, dropping messages if connection is
  not fast enough
- syslog is just fine for now, local anyways
- stdout same thing

refs #26
2017-12-29 17:21:58 +01:00
9a19615fd4 docs: document bookmarking + remove warning about replication lag
refs #34
2017-12-28 13:24:25 +01:00
03ba2bb7c8 docs: move config files + runtime dir doc to new configuration/preface 2017-12-27 18:34:24 +01:00
7ac2821147 docs: small usage section mentioning CLI 2017-12-27 18:34:24 +01:00
e6554b77c0 docs: mention control status command in tutorial
refs #10
2017-12-27 18:34:24 +01:00
acd9aedb98 cmd control status: unify job logs, option to show only one job & always show logs
refs #10
2017-12-27 18:34:24 +01:00
835cf6b12f cmd control status: warn about inactive tasks
refs #10
2017-12-27 18:34:24 +01:00
4b3d83ec1f TaskStatus: add LastUpdate field
refs #10
2017-12-27 18:34:24 +01:00
d13c6e3fc3 job local: refactor + use Task API
refs #10
2017-12-27 18:34:24 +01:00
63fa7a67e9 job source: refactor + use Task API
refs #10
2017-12-27 18:34:24 +01:00
7d89d1fb00 job pull: refactor + use Task API
refs #10
2017-12-27 18:34:24 +01:00
b69089a527 Puller: refactor + use Task API
* drop rx byte count functionality
* will be re-added to Task as necessary

refs #10
2017-12-27 14:39:47 +01:00
59e34942d1 Puller: make main interface public
refs #10
2017-12-27 14:39:46 +01:00
91c4a97f72 Pruner: refactor + use Task API
refs #10
2017-12-27 14:39:46 +01:00
13562b48ed IntervalAutosnap: refactor + use Task API
refs #10
2017-12-27 14:39:46 +01:00
58ee796394 adopt Task API: infect datastructures
refs #10
2017-12-27 14:39:46 +01:00
ce351146cf job control: implement JobStatus 2017-12-27 14:39:46 +01:00
14b8d69a63 cmd control status + expose DaemonStatus via control API
refs #10
2017-12-27 14:39:46 +01:00
8c7e373049 daemon: DaemonStatus + JobStatus + dummy implementation
refs #10
2017-12-27 14:39:46 +01:00
2c87b15e83 daemon: Task abstraction + TaskStatus
An instance of Task tracks a single thread of activity that is part of a Job.

While the docs already use this terminology of tasks being composed of jobs,
the code did not have an object to represent these semantics.
Now it does:

* A task t is initialized with a root activity, which is its name
* t can t.Enter() and t.Finish() an activity, building
  a stack of activities
* t's code can get a logger t.Log() whose logTaskField is set to the
  concatenated stack of activities
* t's code can update IO progress it made since leaving idle state
* t's code's log output vie t.Log() is captured since leaving idle
  state
  * FIXME: find a way to bound that buffer

refs #10
refs #48
2017-12-27 14:39:46 +01:00
d7f3fb93ae bash completions: hidden subcommand + integrate into Makefile 2017-12-27 14:39:46 +01:00
ebf209427a logging: support ignoring fields in HumanFormatter
should be refactored to logger one day so the implementation of ignoring
is not duplicated to each outlet.

refs #10
2017-12-27 14:39:46 +01:00
51377a8ffb logger: support replacing fields
do no delete() from array since this could lead to resizing

refs #10
2017-12-27 14:39:43 +01:00
f14dc3107f logger: implement json.Unmarshaler
refs #10
2017-12-27 13:50:07 +01:00
261d095108 logger: support forking of outlets
refs #10
2017-12-27 13:50:07 +01:00
583a63a68f refactor: encapsulate pulling in a struct
refs #10
2017-12-24 15:23:28 +01:00
2716c75ad5 build: target for go library dependencies
Didn't notice it because vendor/ was already populated on my dev
machine, but did notice it in Docker build.

Docker build now consumes devsetup like regular user, so this should
catch future problems.

Remove remaining curl|shit functionality from lazy.sh (no checkout logic
needed anymore).

refs #35
2017-11-19 12:34:01 +01:00
e8facfe9fa docs: sphinx-versioning would not build master
sphinx-versioning only build branches / commits with a 'docs/conf.py',
otherwise:

    => Gathering info about the remote git repository...
    => Getting list of all remote branches/tags...
    => Found: docs_theme master resumable_send_recv 0.0.1 0.0.2
    => With docs: 0.0.2
    => Root ref master not found in: 0.0.2

refs #35
2017-11-18 21:28:10 +01:00
d424e800c8 docs: publish.sh check if sphinx-versioning is installed
refs #35
2017-11-18 21:16:54 +01:00
896f31bbf3 'zrepl version' and 'zrepl control version' subcommand + maintainer README
Version is autodetected on build using git
If it cannot be detected with git, an override must be provided.

For tracability of distros, the distroy packagers should override as
well, which is why I added a README entry for package mainatiners.

refs #35
2017-11-18 21:12:48 +01:00
d59426a8cc document new build infra in README.md
refs #35
2017-11-18 19:17:59 +01:00
903fbff710 Add Docker build image, modularize lazy.sh and adjust build from source instructions
refs #35
2017-11-18 19:11:14 +01:00
bc4b129536 lazy.sh: support non-terminal outputs 2017-11-18 17:02:43 +01:00
b4b1bebb5c rename clone_and_build.sh to lazy.sh
refs #35
2017-11-18 17:02:11 +01:00