docs: reflect changes in replication_rewrite branch

docs/changelog.rst

@@ -1,81 +1,103 @@
-.. |break_config| replace:: **[BREAK]**
+.. |break_config| replace:: **[CONFIG]**
 .. |break| replace:: **[BREAK]**
 .. |bugfix| replace:: [BUG]
 .. |docs| replace:: [DOCS]
 .. |feature| replace:: [FEATURE]
 
+.. _changelog:
+
 Changelog
 =========
 
-The changelog summarized bugfixes that are deemed relevant for users.
+The changelog summarizes bugfixes that are deemed relevant for users and package maintainers.
 Developers should consult the git commit log or GitHub issue tracker.
 
-0.0.4 (unreleased)
+e use the following annotations for classifying changes:
+
+* |break_config| Change that breaks the config.
+  As a package maintainer, make sure to warn your users about config breakage somehow.
+* |break| Change that breaks interoperability or persistent state representation with previous releases.
+  As a package maintainer, make sure to warn your users about config breakage somehow.
+  Note that even updating the package on both sides might not be sufficient, e.g. if persistent state needs to be migrated to a new format.
+* |feature| Change that introduces new functionality.
+* |bugfix| Change that fixes a bug, no regressions or incompatibilities expected.
+* |docs| Change to the documentation.
+
+0.1 (unreleased)
 ------------------
 
+This release is a milestone for zrepl and required significant refactoring if not rewrites of substantial parts of the application.
+It breaks both configuration and transport format, and thus requires manual intervention and updates on both sides of a replication setup.
+
+Notes to Package Maintainer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* If the daemon crashes, the stack trace produced by the Go runtime and possibly diagnostic output of zrepl will be written to stderr.
+  This behavior is independent from the ``stdout`` outlet type.
+  Please make sure the stderr output of the daemon is captured to a file.
+  Rotation should not be necessary because stderr is not written to under normal circumstances.
+  To conserve precious stack traces, make sure that multiple service restarts do not directly discard previous stderr output.
+* Make it obvious for users how to set the ``GOTRACEBACK`` environment variable to ``GOTRACEBACK=crash``.
+  This functionality will cause SIGABRT on panics and can be used to capture a coredump of the panicking process.
+  To that extend, make sure that your package build system, your OS's coredump collection and the Go delve debugger work together.
+  Use your build system to package the Go program in `this tutorial on Go coredumps and the delve debugger <https://rakyll.org/coredumps/>`_ , and make sure the symbol resolution etc. work on coredumps captured from the binary produced by your build system. (Special focus on symbol stripping, etc.)
+
+Changes
+~~~~~~~
+
+* |feature| :issue:`55` : Push replication (see :ref:`push job <job-push>` and :ref:`sink job <job-sink>`)
+* |feature| :ref:`TCP Transport <transport-tcp>`
+* |feature| :ref:`TCP + TLS client authentication transport <transport-tcp+tlsclientauth>`
+* |feature| :issue:`78` TODO MERGE COMMIT Replication protocol rewrite
+
+  * Uses ``github.com/problame/go-streamrpc`` for RPC layer
+  * |break| zrepl 0.1 and restart on both sides of a replication setup is required
+  * |feature| :issue:`83`:  Improved error handling of network-level errors (zrepl retries instead of failing the entire job)
+  * |bugfix| :issue:`75` :issue:`81`: use connection timeouts and protocol-level heartbeats
+  * |break| |break_config|: mappings are no longer supported
+
+    * Receiving sides (``pull`` and ``sink`` job) specify a single ``root_fs``.
+      Received filesystems are then stored *per client* in ``${root_fs}/${client_identity}``.
+
+* |feature| |break| |break_config| Manual snapshotting + triggering of replication
+
+  * |feature| :issue:`69`: include manually created snapshots in replication
+  * |break_config| ``manual`` and ``periodic`` :ref:`snapshotting types <job-snapshotting-spec>`
+  * |feature| ``zrepl wakeup JOB`` subcommand to trigger *just* replication
+
+* |feature| |break| |break_config| New pruning system
+
+  * The active side of a replication (pull or push) decides what to prune for both sender and receiver.
+    The RPC protocol is used to execute the destroy operations on the remote side.
+  * New pruning policies (see :ref:`configuration documentation <prune>` )
+
+    * The decision what snapshots shall be pruned is now made based on *keep rules*
+    * |feature| :issue:`68`: keep rule ``not_replicated`` prevents divergence of sender and receiver
+
+  * |feature| |break| Bookmark pruning is no longer necessary
+
+    * Per filesystem, zrepl creates a single bookmark (``#zrepl_replication_cursor``) and moves it forward with the most recent successfully replicated snapshot on the receiving side.
+    * Old bookmarks created prior to zrepl 0.1 (named like their corresponding snapshot) must be deleted manually.
+    * |break_config| ``keep_bookmarks`` parameter of the ``grid`` keep rule has been removed
+
+* |feature| ``zrepl status`` for live-updating replication progress (it's really cool!)
 * |feature| :issue:`67`: Expose `Prometheus <https://prometheus.io>`_ metrics via HTTP (:ref:`config docs <monitoring-prometheus>`)
 
-0.0.3
+* |break_config| Logging outlet types must be specified using the ``type`` instead of ``outlet`` key
------
+* |break| :issue:`53`: CLI: ``zrepl control *`` subcommands have been made direct subcommands of ``zrepl *``
 
-* |break_config| |feature| :issue:`34`: automatic bookmarking of snapshots
+* |bugfix| :issue:`81` :issue:`77` : handle failed accepts correctly (``source`` job)
 
-  * Snapshots are automatically bookmarked and pruning of bookmarks **must** be configured.
+.. |lastrelease| replace:: 0.0.3
-  * This breaks existing configuration: ``grid`` :ref:`prune policy <prune-retention-grid>`  specifications require the new ``keep_bookmarks`` parameter.
-  * Make sure to understand the meaning bookmarks have for :ref:`maximum replication downtime <replication-downtime>`.
-  * Example: :sampleconf:`pullbackup/productionhost.yml`
 
-* |break| :commit:`ccd062e`: ``ssh+stdinserver`` transport: changed protocol requires daemon restart on both sides
+Previous Releases
+-----------------
 
-  * The delicate procedure of talking to the serving-side zrepl daemon via the stdinserver proxy command now has better error handling.
+.. NOTE::
-  * This includes handshakes between client+proxy and client + remote daemo, which is not implemented in previous versions of zrepl.
+    Due to limitations in our documentation system, we only show the changelog since the last release and the time this documentation is built.
-  * The connecting side will therefore time out, with the message ``dial_timeout of 10s exceeded``.
+    For the changelog of previous releases, use the version selection in the hosted version of these docs at `zrepl.github.io <https://zrepl.github.io>`_.
-  * Both sides of a replication setup must be updated and restarted. Otherwise the connecting side will hang and not time out.
-
-* |break_config| :commit:`2bfcfa5`: first outlet in ``global.logging`` is now used for logging meta-errors, for example problems encountered when writing to other outlets.
-* |feature| :issue:`10`: ``zrepl control status`` subcommand
-
-  * Allows inspection of job activity per task and their log output at runtime.
-  * Supports ``--format raw`` option for JSON output, usable for monitoring from scripts.
-
-* |feature| :commit:`d7f3fb9`: subcommand bash completions
-
-  * Package maintainers should install this as appropriate.
-
-* |bugfix| :issue:`61`: fix excessive memory usage
-* |bugfix| :issue:`8` and :issue:`56`: ``ssh+stdinserver`` transport properly reaps SSH child processes
-* |bugfix| :commit:`cef63ac`: ``human`` format now prints non-string values correctly
-* |bugfix| :issue:`26`: slow TCP outlets no longer block the daemon
-* |docs| :issue:`64`: tutorial: document ``known_host`` file entry
-
-0.0.2
------
-
-* |break_config| :commit:`b95260f`: ``global.logging`` is no longer a dictionary but a list
-
-* |break_config| :commit:`3e647c1`: ``source`` job field ``datasets`` renamed to ``filesystems``
-
-  * **NOTE**: zrepl will parse missing ``filesystems`` field as an empty filter,
-    i.e. no filesystems are presented to the other side.
-
-* |bugfix| :commit:`72d2885` fix aliasing bug with root `<` subtree wildcard
-
-  * Filesystems paths with final match at blank `s` subtree wildcard are now appended to the target path
-  * Non-root subtree wildcards, e.g. `zroot/foo/bar<` still map directrly onto the target path
-
-* Support days (``d``) and weeks (``w``) in durations
-
-* Docs
-
-  * Ditch Hugo, move to Python Sphinx
-  * Improve & simplify tutorial (single SSH key per installation)
-  * Document pruning policies
-  * Document job types
-  * Document logging
-  * Start updating implementation overview
     
     
-0.0.1
------
 
-* Initial release
+
+W

docs/configuration.rst

@@ -10,7 +10,7 @@ Configuration
     configuration/preface
     configuration/jobs
     configuration/transports
-    configuration/map_filter_syntax
+    configuration/filter_syntax
     configuration/prune
     configuration/logging
     configuration/monitoring

docs/configuration/filter_syntax.rst

@@ -0,0 +1,67 @@
+.. include:: ../global.rst.inc
+
+.. _pattern-filter:
+
+Filter Syntax
+=============
+
+For :ref:`source jobs <job-source>` and :ref:`push jobs <job-push>`, a filesystem filter must be defined (field ``filesystems``).
+A filter takes a filesystem path (in the ZFS filesystem hierarchy) as parameter and returns ``true`` (pass) or ``false`` (block).
+
+A filter is specified as a **YAML dictionary** with patterns as keys and booleans as values.
+The following rules determine which result is chosen for a given filesystem path:
+
+* More specific path patterns win over less specific ones
+* Non-wildcard patterns (full path patterns) win over *subtree wildcards* (`<` at end of pattern)
+* If the path in question does not match any pattern, the result is ``false``.
+
+The **subtree wildcard** ``<`` means "the dataset left of ``<`` and all its children".
+   
+.. TIP::
+  You can try out patterns for a configured job using the ``zrepl test`` subcommand.
+
+Examples
+--------
+
+Full Access
+~~~~~~~~~~~
+
+The following configuration will allow access to all filesystems.
+
+::
+
+   jobs:
+   - type: source
+     filesystems: {
+       "<": true,
+     }
+     ...
+
+
+Fine-grained
+~~~~~~~~~~~~
+
+The following configuration demonstrates all rules presented above.
+
+::
+
+    jobs:
+    - type: source
+      filesystems: {
+        "tank<": true,          # rule 1
+        "tank/foo<": false,     # rule 2
+        "tank/foo/bar": true,  # rule 3
+      }
+      ...
+
+
+Which rule applies to given path, and what is the result?
+
+::
+
+    tank/foo/bar/loo => 2    false
+    tank/bar         => 1    true
+    tank/foo/bar     => 3    true
+    zroot            => NONE false
+    tank/var/log     => 1    true
+

docs/configuration/jobs.rst

@@ -1,20 +1,106 @@
 .. include:: ../global.rst.inc
 
-.. |patient| replace:: :ref:`patient <job-term-patient>`
+.. |serve-transport| replace:: :ref:`serve specification<transport>`
-.. |serve-transport| replace:: :ref:`serve transport<transport-ssh+stdinserver-serve>`
+.. |connect-transport| replace:: :ref:`connect specification<transport>`
-.. |connect-transport| replace:: :ref:`connect transport<transport-ssh+stdinserver-connect>`
+.. |snapshotting-spec| replace:: :ref:`snapshotting specification <job-snapshotting-spec>`
-.. |mapping| replace:: :ref:`mapping <pattern-mapping>`
+.. |pruning-spec| replace:: :ref:`pruning specification <prune>`
-.. |filter| replace:: :ref:`filter <pattern-filter>`
+.. |filter-spec| replace:: :ref:`filter specification<pattern-filter>`
-.. |prune| replace:: :ref:`prune <prune>`
 
 .. _job:
 
-Job Types
+Job Types & Replication
-=========
+=======================
+
+Overview & Terminology
+----------------------
 
 A *job* is the unit of activity tracked by the zrepl daemon and configured in the |mainconfig|.
 Every job has a unique ``name``, a ``type`` and type-dependent fields which are documented on this page.
-Check out the :ref:`tutorial` and :sampleconf:`/` for examples on how job types are actually used.
+
+Replication always happens between a pair of jobs: one is the **active side**, and one the **passive side**.
+The active side executes the replication logic whereas the passive side responds to requests after checking the active side's permissions.
+For communication, the active side connects to the passive side using a :ref:`transport <transport>` and starts issuing remote procedure calls (RPCs).
+
+The following table shows how different job types can be combined to achieve both push and pull mode setups:
+
++-----------------------+--------------+----------------------------------+-----------------------------------------------+
+| Setup name            | active side  | passive side                     | use case                                      |
++=======================+==============+==================================+===============================================+
+| Push mode             | ``push``     | ``sink``                         | * Laptop backup                               |
+|                       |              |                                  | * NAS behind NAT to offsite                   |
++-----------------------+--------------+----------------------------------+-----------------------------------------------+
+| Pull mode             | ``pull``     | ``source``                       | * Central backup-server for many nodes        |
+|                       |              |                                  | * Remote server to NAS behind NAT             |
++-----------------------+--------------+----------------------------------+-----------------------------------------------+
+| Local replication     | | ``push`` + ``sink`` in one config             | * Backup FreeBSD boot pool                    |
+|                       | | with :ref:`local transport <transport-local>` |                                               |
++-----------------------+--------------+----------------------------------+-----------------------------------------------+
+
+How the Active Side Works
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The active side (:ref:`push <job-push>` and :ref:`pull <job-pull>` job) executes the replication and pruning logic:
+
+* Wakeup because of finished snapshotting (``push`` job) or pull interval ticker (``pull`` job).
+* Connect to the corresponding passive side using a :ref:`transport <transport>` and instantiate an RPC client.
+* Replicate data from the sending to the receiving side.
+* Prune on sender & receiver.
+
+.. TIP::
+  The progress of the active side can be watched live using the ``zrepl status`` subcommand.
+
+How the Passive Side Works
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The passive side (:ref:`sink <job-sink>` and :ref:`source <job-source>`) waits for connections from the corresponding active side,
+using the transport listener type specified in the ``serve`` field of the job configuration.
+Each transport listener provides a client's identity to the passive side job.
+It uses the client identity for access control:
+
+* The ``sink`` job only allows pushes to those ZFS filesystems to the active side that are located below ``root_fs/${client_identity}``.
+* The ``source`` job has a whitelist of client identities that are allowed pull access.
+
+.. TIP::
+   The implementation of the ``sink`` job requires that the connecting client identities be a valid ZFS filesystem name components.
+
+How Replication Works
+~~~~~~~~~~~~~~~~~~~~~
+
+One of the major design goals of the replication module is to avoid any duplication of the nontrivial logic.
+As such, the code works on abstract senders and receiver **endpoints**, where typically one will be implemented by a local program object and the other is an RPC client instance.
+Regardless of push- or pull-style setup, the logic executes on the active side, i.e. in the ``push`` or ``pull`` job.
+
+The following steps take place during replication and can be monitored using the ``zrepl status`` subcommand:
+
+* Plan the replication:
+
+  * Compare sender and receiver filesystem snapshots
+  * Build the **replication plan**
+
+    * Per filesystem, compute a diff between sender and receiver snapshots
+    * Build a list of replication steps
+
+      * If possible, use incremental sends (``zfs send -i``)
+      * Otherwise, use full send of most recent snapshot on sender
+      * Give up on filesystems that cannot be replicated without data loss
+
+  * Retry on errors that are likely temporary (i.e. network failures).
+  * Give up on filesystems where a permanent error was received over RPC.
+
+* Execute the plan
+
+  * Perform replication steps in the following order:
+    Among all filesystems with pending replication steps, pick the filesystem whose next replication step's snapshot is the oldest.
+  * After a successful replication step, update the replication cursor bookmark (see below)
+   
+The idea behind the execution order of replication steps is that if the sender snapshots all filesystems simultaneously at fixed intervals, the receiver will have all filesystems snapshotted at time ``T1`` before the first snapshot at ``T2 = T1 + $interval`` is replicated.
+
+.. _replication-cursor-bookmark:
+
+The **replication cursor bookmark** ``#zrepl_replication_cursor`` is kept per filesystem on the sending side of a replication setup:
+It is a bookmark of the most recent successfully replicated snapshot to the receiving side.
+It is is used by the :ref:`not_replicated <prune-keep-not-replicated>` keep rule to identify all snapshots that have not yet been replicated to the receiving side.
+Regardless of whether that keep rule is used, the bookmark ensures that replication can always continue incrementally.
 
 .. ATTENTION::
 
@@ -22,12 +108,133 @@ Check out the :ref:`tutorial` and :sampleconf:`/` for examples on how job types
     Whe receiving a filesystem, it is never mounted (`-u` flag)  and `mountpoint=none` is set.
     This is temporary and being worked on :issue:`24`.
 
+
+.. _job-snapshotting-spec:
+
+Taking Snaphots
+---------------
+
+The ``push`` and ``source`` jobs can automatically take periodic snapshots of the filesystems matched by the ``filesystems`` filter field.
+The snapshot names are composed of a user-defined prefix followed by a UTC date formatted like ``20060102_150405_000``.
+We use UTC because it will avoid name conflicts when switching time zones or between summer and winter time.
+
+For ``push`` jobs, replication is automatically triggered after all filesystems have been snapshotted.
+
+::
+
+    jobs:
+    - type: push
+      filesystems: {
+        "<": true,
+        "tmp": false
+      }
+      snapshotting:
+        type: periodic
+        prefix: zrepl_
+        interval: 10m
+      ...
+
+There is also a ``manual`` snapshotting type, which covers the following use cases:
+
+* Existing infrastructure for automatic snapshots: you only want to use zrepl for replication.
+* Run scripts before and after taking snapshots (like locking database tables).
+  We are working on better integration for this use case: see :issue:`74`.
+
+Note that you will have to trigger replication manually using the ``zrepl wakeup JOB`` subcommand in that case.
+
+::
+
+   jobs:
+   - type: push
+     filesystems: {
+       "<": true,
+       "tmp": false
+     }
+     snapshotting:
+       type: manual
+     ...
+
+
+.. _job-push:
+
+Job Type ``push``
+-----------------
+
+.. list-table::
+    :widths: 20 80
+    :header-rows: 1
+
+    * - Parameter
+      - Comment
+    * - ``type``
+      - = ``push``
+    * - ``name``
+      - unique name of the job
+    * - ``connect``
+      - |connect-transport|
+    * - ``filesystems``
+      - |filter-spec| for filesystems to be snapshotted and pushed to the sink
+    * - ``snapshotting``
+      - |snapshotting-spec|
+    * - ``pruning``
+      - |pruning-spec|
+
+Example config: :sampleconf:`/push.yml`
+
+.. _job-sink:
+
+Job Type ``sink``
+-----------------
+
+.. list-table::
+    :widths: 20 80
+    :header-rows: 1
+
+    * - Parameter
+      - Comment
+    * - ``type``
+      - = ``sink``
+    * - ``name``
+      - unique name of the job
+    * - ``serve``
+      - |serve-transport|
+    * - ``root_fs``
+      - ZFS dataset path are received to
+        ``$root_fs/$client_identity``
+
+Example config: :sampleconf:`/sink.yml`
+
+.. _job-pull:
+
+Job Type ``pull``
+-----------------
+
+.. list-table::
+    :widths: 20 80
+    :header-rows: 1
+
+    * - Parameter
+      - Comment
+    * - ``type``
+      - = ``pull``
+    * - ``name``
+      - unique name of the job
+    * - ``connect``
+      - |connect-transport|
+    * - ``root_fs``
+      - ZFS dataset path are received to
+        ``$root_fs/$client_identity``
+    * - ``interval``
+      - Interval at which to pull from the source job
+    * - ``pruning``
+      - |pruning-spec|
+
+Example config: :sampleconf:`/pull.yml`
+
 .. _job-source:
 
-Source Job
+Job Type ``source``
-----------
+-------------------
-
-Example: :sampleconf:`pullbackup/productionhost.yml`.
 
 .. list-table::
     :widths: 20 80
@@ -42,142 +249,19 @@ Example: :sampleconf:`pullbackup/productionhost.yml`.
     * - ``serve``
       - |serve-transport|
     * - ``filesystems``
-      - |filter| for filesystems to expose to client
+      - |filter-spec| for filesystems to be snapshotted and exposed to connecting clients
-    * - ``snapshot_prefix``
+    * - ``snapshotting``
-      - prefix for ZFS snapshots taken by this job
+      - |snapshotting-spec|
-    * - ``interval``
+
-      - snapshotting interval
+Example config: :sampleconf:`/source.yml`
-    * - ``prune``
+
-      - |prune| for versions of filesytems in ``filesystems``, versions prefixed with ``snapshot_prefix``
+.. _replication-local:
+
+Local replication
+-----------------
+
+If you have the need for local replication (most likely between two local storage pools), you can use the :ref:`local transport type <transport-local>` to connect a local push job to a local sink job.
+
+Example config: :sampleconf:`/local.yml`.
 
 
-- Snapshotting Task (every ``interval``, |patient|)
-
-  - A snapshot of filesystems matched by ``filesystems`` is taken every ``interval`` with prefix ``snapshot_prefix``.
-  - A bookmark of that snapshot is created with the same name.
-  - The ``prune`` policy is evaluated for versions of filesystems matched by ``filesystems``, versions prefixed with ``snapshot_prefix``.
-
-- Serve Task
-
-  - Wait for connections from pull job using ``serve``.
-
-A source job is the counterpart to a :ref:`job-pull`.
-
-Make sure you read the |prune| policy documentation.
-
-.. _job-pull:
-
-Pull Job
---------
-
-Example: :sampleconf:`pullbackup/backuphost.yml`
-
-.. list-table::
-    :widths: 20 80
-    :header-rows: 1
-
-    * - Parameter
-      - Comment
-    * - ``type``
-      - = ``pull``
-    * - ``name``
-      - unqiue name of the job
-    * - ``connect``
-      - |connect-transport|
-    * - ``interval``
-      - Interval between pull attempts
-    * - ``mapping``
-      - |mapping| for remote to local filesystems
-    * - ``snapshot_prefix``
-      - prefix snapshots must match to be considered for replication & pruning
-    * - ``prune``
-      - |prune| policy for versions of filesystems of local filesystems reachable by ``mapping``, versions prefixed with ``snapshot_prefix``
-
-* Main Task (every ``interval``, |patient|)
- 
-  #. A connection to the remote source job is established using the strategy in ``connect``
-  #. ``mapping`` maps filesystems presented by the remote side to local *target filesystems*
-  #. Those remote filesystems with a local *target filesystem* are replicated
- 
-     #. Only snapshots with prefix ``snapshot_prefix`` are replicated.
-     #. If possible, incremental replication takes place.
-     #. If the local target filesystem does not exist, the most recent snapshot is sent fully (non-incremental).
-     #. On conflicts, an error is logged but replication of other filesystems with mapping continues.
-  
-  #. The ``prune`` policy is evaluated for all *target filesystems*
-
-A pull job is the counterpart to a :ref:`job-source`.
-
-Make sure you read the |prune| policy documentation.
-
-.. _job-local:
-
-Local Job
----------
-
-Example: :sampleconf:`localbackup/host1.yml`
-
-.. list-table::
-    :widths: 20 80
-    :header-rows: 1
-
-    * - Parameter
-      - Comment
-    * - ``type``
-      - = ``local``
-    * - ``name``
-      - unqiue name of the job
-    * - ``mapping``
-      - |mapping| from source to target filesystem (both local)
-    * - ``snapshot_prefix``
-      - prefix for ZFS snapshots taken by this job
-    * - ``interval``
-      - snapshotting & replication interval
-    * - ``prune_lhs``
-      -  pruning policy on left-hand-side (source)
-    * - ``prune_rhs``
-      - pruning policy on right-hand-side (target)
-
-* Main Task (every ``interval``, |patient|)
-
-  #. Evaluate ``mapping`` for local filesystems, those with a *target filesystem* are called *mapped filesystems*.
-  #. Snapshot *mapped filesystems* with ``snapshot_prefix``.
-  #. Bookmark the snapshot created above.
-  #. Replicate *mapped filesystems* to their respective *target filesystems*:
-
-     #. Only snapshots with prefix ``snapshot_prefix`` are replicated.
-     #. If possible, incremental replication takes place.
-     #. If the *target filesystem* does not exist, the most recent snapshot is sent fully (non-incremental).
-     #. On conflicts, an error is logged but replication of other *mapped filesystems* continues.
-
-  #. The ``prune_lhs`` policy is triggered for all *mapped filesystems*
-  #. The ``prune_rhs`` policy is triggered for all *target filesystems*
-
-A local job is combination of source & pull job executed on the same machine.
-
-Terminology
------------
-
-task
-
-    A job consists of one or more tasks and a task consists of one or more steps.
-    Some tasks may be periodic while others wait for an event to occur.
-
-
-patient task
-
-    .. _job-term-patient:
-
-    A patient task is supposed to execute some task every `interval`.
-    We call the start of the task an *invocation*.
-
-    * If the task completes in less than `interval`, the task is restarted at `last_invocation + interval`.
-    * Otherwise, a patient job
-        * logs a warning as soon as a task exceeds its configured `interval`
-        * waits for the last invocation to finish
-        * logs a warning with the effective task duration
-        * immediately starts a new invocation of the task
-
-filesystem version
-
-    A snapshot or a bookmark.

docs/configuration/logging.rst

@@ -8,18 +8,17 @@ Logging
 zrepl uses structured logging to provide users with easily processable log messages.
 
 Logging outlets are configured in the ``global`` section of the |mainconfig|.
-Check out :sampleconf:`random/logging_and_monitoring.yml` for an example on how to configure multiple outlets:
 
 ::
 
     global:
       logging:
     
-        - outlet: OUTLET_TYPE
+        - type: OUTLET_TYPE
           level: MINIMUM_LEVEL
           format: FORMAT
     
-        - outlet: OUTLET_TYPE
+        - type: OUTLET_TYPE
           level: MINIMUM_LEVEL
           format: FORMAT
     
@@ -45,7 +44,7 @@ By default, the following logging configuration is used
     global:
       logging:
     
-        - outlet: "stdout"
+        - type: "stdout"
           level:  "warn"
           format: "human"
 
@@ -93,8 +92,8 @@ Formats
     * - Format
       - Description
     * - ``human``
-      - emphasizes context by putting job, task, step and other context variables into brackets
+      - prints job and subsystem into brackets before the actual message,
-        before the actual message, followed by remaining fields in logfmt style|
+        followed by remaining fields in logfmt style
     * - ``logfmt``
       - `logfmt <https://brandur.org/logfmt>`_ output. zrepl uses `this Go package <https://github.com/go-logfmt/logfmt>`_.
     * - ``json``
@@ -118,7 +117,7 @@ Outlets are the destination for log entries.
 
     * - Parameter
       - Comment
-    * - ``outlet``
+    * - ``type``
       - ``stdout``
     * - ``level``
       -  minimum  :ref:`log level <logging-levels>`
@@ -126,9 +125,11 @@ Outlets are the destination for log entries.
       - output :ref:`format <logging-formats>`
     * - ``time``
       - always include time in output (``true`` or ``false``)
+    * - ``color``
+      - colorize output according to log level (``true`` or ``false``)
 
 Writes all log entries with minimum level ``level`` formatted by ``format`` to stdout.
-If stdout is a tty, interactive usage is assumed and the current time is included in the output.
+If stdout is a tty, interactive usage is assumed and both ``time`` and ``color`` are set to ``true``.
 
 Can only be specified once.
 
@@ -140,7 +141,7 @@ Can only be specified once.
 
     * - Parameter
       - Comment
-    * - ``outlet``
+    * - ``type``
       - ``syslog``
     * - ``level``
       -  minimum  :ref:`log level <logging-levels>`
@@ -163,7 +164,7 @@ Can only be specified once.
 
     * - Parameter
       - Comment
-    * - ``outlet``
+    * - ``type``
       - ``tcp``
     * - ``level``
       -  minimum  :ref:`log level <logging-levels>`
@@ -179,11 +180,9 @@ Can only be specified once.
       - TLS config (see below)
 
 Establishes a TCP connection to ``address`` and sends log messages with minimum level ``level`` formatted by ``format``.
-
 If ``tls`` is not specified, an unencrypted connection is established.
-
 If ``tls`` is specified, the TCP connection is secured with TLS + Client Authentication.
-This is particularly useful in combination with log aggregation services that run on an other machine.
+The latter is particularly useful in combination with log aggregation services.
 
 .. list-table::
     :widths: 10 90

docs/configuration/map_filter_syntax.rst

@@ -1,108 +0,0 @@
-.. include:: ../global.rst.inc
-
-Mapping & Filter Syntax
-=======================
-
-For various job types, a filesystem ``mapping`` or ``filter`` needs to be
-specified.
-
-Both have in common that they take a filesystem path (in the ZFS filesystem hierarchy)as parameters and return something.
-Mappings return a *target filesystem* and filters return a *filter result*.
-
-The pattern syntax is the same for mappings and filters and is documented in the following section.
-
-Common Pattern Syntax
----------------------
-
-A mapping / filter is specified as a **YAML dictionary** with patterns as keys and
-results as values.
-The following rules determine which result is chosen for a given filesystem path:
-
-* More specific path patterns win over less specific ones
-* Non-wildcard patterns (full path patterns) win over *subtree wildcards* (`<` at end of pattern)
-
-The **subtree wildcard** ``<`` means "the dataset left of ``<`` and all its children".
-
-Example
-~~~~~~~
-
-::
-
-    # Rule number and its pattern
-    1: tank<            # tank and all its children
-    2: tank/foo/bar     # full path pattern (no wildcard)
-    3: tank/foo<        # tank/foo and all its children
-    
-    # Which rule applies to given path?
-    tank/foo/bar/loo => 3
-    tank/bar         => 1
-    tank/foo/bar     => 2
-    zroot            => NO MATCH
-    tank/var/log     => 1
-
-.. _pattern-mapping:
-
-Mappings
---------
-
-Mappings map a *source filesystem path* to a *target filesystem path*.
-Per pattern, either a target filesystem path or ``"!"`` is specified as a result.
-
-* If no pattern matches, there exists no target filesystem (``NO MATCH``).
-* If the result is a ``"!"``, there exists no target filesystem (``NO MATCH``).
-* If the pattern is a non-wildcard pattern, the source path is mapped to the target path on the right.
-* If the pattern ends with a *subtree wildcard* (``<``), the source path is **prefix-trimmed** with the path specified left of ``<``.
-
-  * Note: this means that only for *wildcard-only* patterns (pattern= ``<`` ) is the source path simply appended to the target path.
-
-The example is from the :sampleconf:`localbackup/host1.yml` example config.
-
-::
-
-    jobs:
-    - name: mirror_local
-      type: local
-      mapping: {
-        "zroot/var/db<":    "storage/backups/local/zroot/var/db",
-        "zroot/usr/home<":  "storage/backups/local/zroot/usr/home",
-        "zroot/usr/home/paranoid":  "!", #don't backup paranoid user
-        "zroot/poudriere/ports<": "!", #don't backup the ports trees
-      }
-      ...
-
-
-::
-
-    zroot/var/db                        => storage/backups/local/zroot/var/db
-    zroot/var/db/a/child                => storage/backups/local/zroot/var/db/a/child
-    zroot/usr/home                      => storage/backups/local/zroot/usr/home
-    zroot/usr/home/paranoid             => NOT MAPPED
-    zroot/usr/home/bob                  => storage/backups/local/zroot/usr/home/bob
-    zroot/usr/src                       => NOT MAPPED
-    zroot/poudriere/ports/2017Q3        => NOT MAPPED
-    zroot/poudriere/ports/HEAD          => NOT MAPPED
-
-.. TIP::
-
-   You can try out patterns for a configured job using the ``zrepl test`` subcommand.
-
-.. _pattern-filter:
-
-Filters
--------
-
-Valid filter results: ``ok`` or ``!``.
-
-The example below show the source job from the :ref:`tutorial <tutorial-configure-app-srv>`:
-The corresponding pull job is allowed access to ``zroot/var/db``, ``zroot/usr/home`` + children except ``zroot/usr/home/paranoid``::
-
-    jobs:
-    - name: pull_backup
-      type: source
-      ...
-      filesystems: {
-        "zroot/var/db": "ok",
-        "zroot/usr/home<": "ok",
-        "zroot/usr/home/paranoid": "!",
-      }
-      ...

docs/configuration/monitoring.rst

@@ -6,7 +6,6 @@ Monitoring
 ==========
 
 Monitoring endpoints are configured in the ``global.monitoring`` section of the |mainconfig|.
-Check out :sampleconf:`random/logging_and_monitoring.yml` for examples.
 
 .. _monitoring-prometheus:
 
@@ -17,7 +16,7 @@ zrepl can expose `Prometheus metrics <https://prometheus.io/docs/instrumenting/e
 The ``listen`` attribute is a `net.Listen <https://golang.org/pkg/net/#Listen>`_  string for tcp, e.g. ``:9091`` or ``127.0.0.1:9091``.
 
 The Prometheues monitoring job appears in the ``zrepl control`` job list and may be specified **at most once**.
-There is no stability guarantee on the exported metrics.
+At the time of writing, there is no stability guarantee on the exported metrics.
 
 ::
 

docs/configuration/preface.rst

@@ -15,7 +15,7 @@ zrepl searches for its main configuration file in the following locations (in th
 * ``/etc/zrepl/zrepl.yml``
 * ``/usr/local/etc/zrepl/zrepl.yml``
 
-The examples in the :ref:`tutorial` or the ``cmd/sampleconf`` directory should provide a good starting point.
+The examples in the :ref:`tutorial` or the :sampleconf:`/` directory should provide a good starting point.
 
 -------------------
 Runtime Directories

docs/configuration/prune.rst

@@ -3,46 +3,96 @@
 Pruning Policies
 ================
 
-In zrepl, *pruning* means *destroying filesystem versions by some policy* where filesystem versions are bookmarks and snapshots.
+In zrepl, *pruning* means *destroying snapshots*.
+Pruning must happen on both sides of a replication or the systems would inevitable run out of disk space at some point.
 
-A *pruning policy* takes a list of filesystem versions and decides for each whether it should be kept or destroyed.
+Typically, the requirements to temporal resolution and maximum retention time differ per side.
+For example, when using zrepl to back up a busy database server, you will want high temporal resolution (snapshots every 10 min) for the last 24h in case of administrative disasters, but cannot afford to store them for much longer because you might have high turnover volume in the database.
+On the receiving side, you may have more disk space available, or need to comply with other backup retention policies.
 
-The job context defines which snapshots are even considered for pruning, for example through the ``snapshot_prefix`` variable.
+zrepl uses a set of  **keep rules** to determine which snapshots shall be kept per filesystem.
-Check the respective :ref:`job definition <job>` for details.
+**A snapshot that is not kept by any rule is destroyed.**
+The keep rules are **evaluated on the active side** (:ref:`push <job-push>` or :ref:`pull job <job-pull>`) of the replication setup, for both active and passive side, after replication completed or was determined to have failed permanently.
 
-Currently, the :ref:`prune-retention-grid` is the only supported pruning policy.
+Example Configuration:
+
+::
+
+   jobs:
+     - type: push
+       name: ...
+       connect: ...
+       filesystems: {
+         "<": true,
+         "tmp": false
+       }
+       snapshotting:
+         type: periodic
+         prefix: zrepl_
+         interval: 10m
+       pruning:
+         keep_sender:
+           - type: not_replicated
+           # make sure manually created snapshots by the administrator are kept
+           - type: regex
+             regex: "^manual_.*"
+           - type: grid
+             grid: 1x1h(keep=all) | 24x1h | 14x1d
+             regex: "^zrepl_.*"
+         keep_receiver:
+           - type: grid
+             grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
+             regex: "^zrepl_.*"
+           # manually created snapshots will be kept forever on receiver
 
 .. TIP::
 
     You can perform a dry-run of a job's pruning policy using the ``zrepl test`` subcommand.
 
-.. _prune-retention-grid:
+.. ATTENTION::
 
-Retention Grid
+    It is currently not possible to define pruning on a source job.
---------------
+    The source job creates snapshots, which means that extended replication downtime will fill up the source's zpool with snapshots, since pruning is directed by the corresponding active side (pull job).
+    If this is a potential risk for you, consider using :ref:`push mode <job-push>`.
+
+
+.. _prune-keep-not-replicated:
+
+Policy ``not_replicated``
+-------------------------
 
 ::
 
    jobs:
-    - name: pull_app-srv
+   - type: push
-      type: pull
+     pruning:
+       keep_sender:
+       - type: not_replicated
      ...
-      prune:
+
-        policy: grid
+``not_replicated`` keeps all snapshots that have not been replicated to the receiving side.
+It only makes sense to specify this rule on a sender (source or push job).
+The state required to evaluate this rule is stored in the :ref:`replication cursor bookmark <replication-cursor-bookmark>` on the sending side.
+
+.. _prune-keep-retention-grid:
+
+Policy ``grid``
+---------------
+
+::
+
+    jobs:
+    - type: pull
+      pruning:
+        keep_receiver:
+        - type: grid
+          regex: "^zrepl_.*"
           grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
                 │                │
                 └─ one hour interval
                                  │
                                  └─ 24 adjacent one-hour intervals
-
+      ...
-    - name: pull_backup
-      type: source
-      interval: 10m
-      prune:
-        policy: grid
-        grid: 1x1d(keep=all)
-        keep_bookmarks: 144
-
 
 The retention grid can be thought of as a time-based sieve:
 The ``grid`` field specifies a list of adjacent time intervals:
@@ -51,16 +101,13 @@ All intervals to its right describe time intervals further in the past.
 
 Each interval carries a maximum number of snapshots to keep.
 It is secified via ``(keep=N)``, where ``N`` is either ``all`` (all snapshots are kept) or a positive integer.
-The default value is **1**.
+The default value is **keep=1**.
-
-Bookmarks are not affected by the above.
-Instead, the ``keep_bookmarks`` field specifies the number of bookmarks to be kept per filesystem.
-You only need to specify ``keep_bookmarks`` at the source-side of a replication setup since the destination side does not receive bookmarks.
-You can specify ``all`` as a value to keep all bookmarks, but be warned that you should install some other way to prune unneeded ones then (see below).
 
 The following procedure happens during pruning:
 
-#. The list of snapshots eligible for pruning is sorted by ``creation``
+#. The list of snapshots is filtered by the regular expression in ``regex``.
+   Only snapshots names that match the regex are considered for this rule, all others are not affected.
+#. The filtered list of snapshots is sorted by ``creation``
 #. The left edge of the first interval is aligned to the ``creation`` date of the youngest snapshot
 #. A list of buckets is created, one for each interval
 #. The list of snapshots is split up into the buckets.
@@ -69,16 +116,42 @@ The following procedure happens during pruning:
    #. the contained snapshot list is sorted by creation.
    #. snapshots from the list, oldest first, are destroyed until the specified ``keep`` count is reached.
    #. all remaining snapshots on the list are kept.
-#. The list of bookmarks eligible for pruning is sorted by ``createtxg`` and the most recent ``keep_bookmarks`` bookmarks are kept.
 
-.. _replication-downtime:
 
-.. ATTENTION::
+.. _prune-keep-last-n:
+
+Policy ``last_n``
+-----------------
+
+::
+
+   jobs:
+     - type: push
+       pruning:
+         keep_receiver:
+         - type: last_n
+           count: 10
+     ...
+
+``last_n`` keeps the last ``count`` snapshots (last = youngest = most recent creation date).
+
+.. _prune-keep-regex:
+
+Policy ``regex``
+----------------
+
+::
+
+   jobs:
+     - type: push
+       pruning:
+         keep_receiver:
+         - type: regex
+           regex: "^(zrepl|manual)_.*"
+    ...
+
+``regex`` keeps all snapshots whose names are matched by the regular expressionin ``regex``.
+Like all other regular expression fields in prune policies, zrepl uses Go's `regexp.Regexp <https://golang.org/pkg/regexp/#Compile>`_ Perl-compatible regular expressions (`Syntax <https://golang.org/pkg/regexp/syntax>`_).
 
-    Be aware that ``keep_bookmarks x interval`` (interval of the job level) controls the **maximum allowable replication downtime** between source and destination.
-    If replication does not work for whatever reason, source and destination will eventually run out of sync because the source will continue pruning snapshots.
-    The only recovery in that case is full replication, which may not always be viable due to disk space or traffic constraints.
 
-    Further note that while bookmarks consume a constant amount of disk space, listing them requires temporary dynamic **kernel memory** proportional to the number of bookmarks.
-    Thus, do not use ``all`` or an inappropriately high value without good reason.
 

docs/configuration/transports.rst

@@ -5,42 +5,156 @@
 Transports
 ==========
 
-A transport provides an authenticated `io.ReadWriteCloser <https://golang.org/pkg/io/#ReadWriteCloser>`_ to the RPC layer.
+The zrepl RPC layer uses **transports** to establish a single, bidirectional data stream between an active and passive job.
-(An ``io.ReadWriteCloser`` is essentially a bidirectional reliable communication channel.)
+On the passive (serving) side, the transport also provides the **client identity** to the upper layers:
+this string is used for access control and separation of filesystem sub-trees in :ref:`sink jobs <job-sink>`.
+Transports are specified in the ``connect`` or ``serve`` section of a job definition.
 
-Currently, only the ``ssh+stdinserver`` transport is supported.
+.. ATTENTION::
+
+    The **client identities must be valid ZFS dataset path components**
+    because the :ref:`sink job <job-sink>` uses ``${root_fs}/${client_identity}`` to determine the client's subtree.
+
+.. _transport-tcp:
+
+``tcp`` Transport
+-----------------
+
+The ``tcp`` transport uses plain TCP, which means that the data is **not encrypted** on the wire.
+Clients are identified by their IPv4 or IPv6 addresses, and the client identity is established through a mapping on the server.
+
+This transport may also be used in conjunction with network-layer encryption and/or VPN tunnels to provide encryption on the wire.
+To make the IP-based client authentication effective, such solutions should provide authenticated IP addresses.
+Some options to consider:
+
+* `WireGuard <https://www.wireguard.com/>`_: Linux-focussed, in-kernel TLS
+* `OpenVPN <https://openvpn.net/>`_: Cross-platform VPN, uses tun on \*nix
+* `IPSec <https://en.wikipedia.org/wiki/IPsec>`_: Properly standardized, in-kernel network-layer VPN
+* `spiped <http://www.tarsnap.com/spiped.html>`_: think of it as an encrypted pipe between two servers
+* SSH
+
+  * `sshuttle <https://sshuttle.readthedocs.io/en/stable/overview.html>`_: VPN-like solution, but using SSH
+  * `SSH port forwarding <https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding>`_: Systemd user unit & make it start before the zrepl service.
+
+Serve
+~~~~~
+
+::
+
+    jobs:
+    - type: sink
+      serve:
+        type: tcp
+        listen: ":8888"
+        clients: {
+          "192.168.122.123" : "mysql01"
+          "192.168.122.123" : "mx01"
+        }
+      ...
+
+Connect
+~~~~~~~
+
+::
+
+    jobs:
+     - type: push
+       connect:
+         type: tcp
+         address: "10.23.42.23:8888"
+         dial_timeout: # optional, default 10s
+       ...
+
+.. _transport-tcp+tlsclientauth:
+
+``tls`` Transport
+-----------------
+
+The ``tls`` transport uses TCP + TLS with client authentication using client certificates.
+The client identity is the common name (CN) presented in the client certificate.
+It is recommended to set up a dedicated CA infrastructure for this transport, e.g. using OpenVPN's `EasyRSA <https://github.com/OpenVPN/easy-rsa>`_.
+
+The implementation uses `Go's TLS library <https://golang.org/pkg/crypto/tls/>`_.
+Since Go binaries are statically linked, you or your distribution need to recompile zrepl when vulnerabilities in that library are disclosed.
+
+All file paths are resolved relative to the zrepl daemon's working directory.
+Specify absolute paths if you are unsure what directory that is (or find out from your init system).
+
+Serve
+~~~~~
+
+::
+
+    jobs:
+      - type: sink
+        root_fs: "pool2/backup_laptops"
+        serve:
+          type: tls
+          listen: ":8888"
+          ca: /etc/zrepl/ca.crt
+          cert: /etc/zrepl/prod.crt
+          key: /etc/zrepl/prod.key
+          client_cns:
+            - "laptop1"
+            - "homeserver"
+
+The ``ca`` field specified the certificate authority used to validate client certificates.
+The ``client_cns`` list specifies a list of accepted client common names (which are also the client identities for this transport).
+
+Connect
+~~~~~~~
+
+::
+
+    jobs:
+    - type: pull
+      connect:
+        type: tls
+        address: "server1.foo.bar:8888"
+        ca: /etc/zrepl/ca.crt
+        cert: /etc/zrepl/backupserver.crt
+        key:  /etc/zrepl/backupserver.key
+        server_cn: "server1"
+        dial_timeout: # optional, default 10s
+
+The ``ca`` field specifies the CA which signed the server's certificate (``serve.cert``).
+The ``server_cn`` specifies the expected common name (CN) of the server's certificate.
+It overrides the hostname specified in ``address``.
+The connection fails if either do not match.
 
 .. _transport-ssh+stdinserver:
 
 ``ssh+stdinserver`` Transport
 -----------------------------
 
-The way the ``ssh+stdinserver`` transport works is inspired by `git shell <https://git-scm.com/docs/git-shell>`_ and `Borg Backup <https://borgbackup.readthedocs.io/en/stable/deployment.html>`_.
+``ssh+stdinserver`` is inspired by `git shell <https://git-scm.com/docs/git-shell>`_ and `Borg Backup <https://borgbackup.readthedocs.io/en/stable/deployment.html>`_.
 It is provided by the Go package ``github.com/problame/go-netssh``.
-The config excerpts are taken from the :ref:`tutorial` which you should complete before reading further.
 
 .. _transport-ssh+stdinserver-serve:
 
-Serve Mode
+Serve
-~~~~~~~~~~
+~~~~~
 
 ::
 
     jobs:
-    - name: pull_backup
+    - type: source
-      type: source
       serve:
         type: stdinserver
-        client_identity: backup-srv.example.com
+        client_identities:
+        - "client1"
+        - "client2"
       ...
 
-The serving job opens a UNIX socket named after ``client_identity`` in the runtime directory, e.g. ``/var/run/zrepl/stdinserver/backup-srv.example.com``.
+First of all, note that ``type=stdinserver`` in this case:
+Currently, only ``connect.type=ssh+stdinserver`` can connect to a ``serve.type=stdinserver``, but we want to keep that option open for future extensions.
 
-On the same machine, the ``zrepl stdinserver $client_identity`` command connects to that socket.
+The serving job opens a UNIX socket named after ``client_identity`` in the runtime directory.
-For example, ``zrepl stdinserver backup-srv.example.com`` connects to the UNIX socket ``/var/run/zrepl/stdinserver/backup-srv.example.com``.
+In our example above, that is ``/var/run/zrepl/stdinserver/client1`` and ``/var/run/zrepl/stdinserver/client2``.
 
+On the same machine, the ``zrepl stdinserver $client_identity`` command connects to ``/var/run/zrepl/stdinserver/$client_identity``.
 It then passes its stdin and stdout file descriptors to the zrepl daemon via *cmsg(3)*.
-zrepl daemon in turn combines them into an ``io.ReadWriteCloser``:
+zrepl daemon in turn combines them into an object implementing ``net.Conn``:
 a ``Write()`` turns into a write to stdout, a ``Read()`` turns into a read from stdin.
 
 Interactive use of the ``stdinserver`` subcommand does not make much sense.
@@ -54,8 +168,8 @@ This can be achieved with an entry in the ``authorized_keys`` file of the servin
     # for older OpenSSH versions
     command="zrepl stdinserver CLIENT_IDENTITY",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc CLIENT_SSH_KEY
 
-* CLIENT_IDENTITY is substituted with ``backup-srv.example.com`` in our example
+* CLIENT_IDENTITY is substituted with an entry from ``client_identities`` in our example
-* CLIENT_SSH_KEY is substituted with the public part of the SSH keypair specified in the ``connect`` directive on the connecting host.
+* CLIENT_SSH_KEY is substituted with the public part of the SSH keypair specified in the ``connect.identity_file`` directive on the connecting host.
 
 .. NOTE::
 
@@ -64,24 +178,24 @@ This can be achieved with an entry in the ``authorized_keys`` file of the servin
 
 To recap, this is of how client authentication works with the ``ssh+stdinserver`` transport:
 
-* Connections to the ``client_identity`` UNIX socket are blindly trusted by zrepl daemon.
+* Connections to the ``/var/run/zrepl/stdinserver/${client_identity}`` UNIX socket are blindly trusted by zrepl daemon.
-* Thus, the runtime directory must be private to the zrepl user (checked by zrepl daemon)
+  The connection client identity is the name of the socket, i.e. ``${client_identity}``.
+* Thus, the runtime directory must be private to the zrepl user (this is checked by zrepl daemon)
 * The admin of the host with the serving zrepl daemon controls the ``authorized_keys`` file.
 * Thus, the administrator controls the mapping ``PUBKEY -> CLIENT_IDENTITY``.
 
 .. _transport-ssh+stdinserver-connect:
 
-Connect Mode
+Connect
-~~~~~~~~~~~~
+~~~~~~~
 
 ::
 
     jobs:
-    - name: pull_app-srv
+    - type: pull
-      type: pull
       connect:
         type: ssh+stdinserver
-        host: app-srv.example.com
+        host: prod.example.com
         user: root
         port: 22
         identity_file: /etc/zrepl/ssh/identity
@@ -102,15 +216,46 @@ The connecting zrepl daemon
       #. The remote user, host and port correspond to those configured.
       #. Further options can be specified using the ``options`` field, which appends each entry in the list to the command line using ``-o $entry``.
 
-#. Wraps the pipe ends in an ``io.ReadWriteCloser`` and uses it for RPC.
+#. Wraps the pipe ends in a ``net.Conn`` and returns it to the RPC layer.
-
 
 As discussed in the section above, the connecting zrepl daemon expects that ``zrepl stdinserver $client_identity`` is  executed automatically via an ``authorized_keys`` file entry.
 
-The ``known_hosts`` file used by the ssh command must contain an entry for the serving host, e.g., ``app-srv.example.com`` in the example above.
+The ``known_hosts`` file used by the ssh command must contain an entry for ``connect.host`` prior to starting zrepl.
+Thus, run the following on the pulling host's command line (substituting ``connect.host``):
+
+::
+
+    ssh -i /etc/zrepl/ssh/identity root@prod.example.com
 
 .. NOTE::
 
     The environment variables of the underlying SSH process are cleared. ``$SSH_AUTH_SOCK`` will not be available.
     It is suggested to create a separate, unencrypted SSH key solely for that purpose.
 
+
+.. _transport-local:
+
+``local`` Transport
+-------------------
+
+The local transport can be used to implement :ref:`local replication <replication-local>`, i.e., push replication between a push and sink job defined in the same configuration file.
+
+The ``listener_name`` is analogous to a hostname and must match between ``serve`` and ``connect``.
+The ``client_identity`` is used by the sink as documented above.
+
+::
+
+    jobs:
+    - type: sink
+      serve:
+        type: local
+        listener_name: localsink
+      ...
+
+    - type: push
+      connect:
+        type: local
+        listener_name: localsink
+        client_identity: local_backup
+      ...
+

docs/implementation.rst

@@ -5,9 +5,12 @@ Implementation Overview
 
 .. WARNING::
 
-    Incomplete / under construction
+   Incomplete and possibly outdated.
+   Check out the :ref:`talks about zrepl <pr-talks>` at various conferences for up-to-date material.
+   Alternatively, have a `look at the source code <http://github.com/zrepl/zrepl>`_ ;)
 
 The following design aspects may convince you that ``zrepl`` is superior to a hacked-together shell script solution.
+Also check out the :ref:`talks about zrepl <pr-talks>` at various conferences.
 
 Testability & Performance
 -------------------------
@@ -28,7 +31,7 @@ While it is tempting to just issue a few ``ssh remote 'zfs send ...' | zfs recv`
 * The snapshot streams need to be compatible.
 * Communication is still unidirectional. Thus, you will most likely
 
-  * either not take advantage of features such as *compressed send & recv*
+  * either not take advantage of advanced replication features such as *compressed send & recv*
   * or issue additional ``ssh`` commands in advance to figure out what features are supported on the other side.
 
 * Advanced logic in shell scripts is ugly to read, poorly testable and a pain to maintain.
@@ -36,25 +39,21 @@ While it is tempting to just issue a few ``ssh remote 'zfs send ...' | zfs recv`
 zrepl takes a different approach:
 
 * Define an RPC protocol.
-* Establish an encrypted, authenticated, bidirectional communication channel...
+* Establish an encrypted, authenticated, bidirectional communication channel.
-* ... with zrepl running at both ends of it.
+* Run daemons on both sides of the setup and let them talk to each other.
 
 This has several obvious benefits:
 
 * No blank root shell access is given to the other side.
-* Instead, an *authenticated* peer can *request* filesystem lists, snapshot streams, etc.
+* An *authenticated* peer *requests* filesystem lists, snapshot streams, etc.
-* Requests are then checked against job-specific ACLs, limiting a client to the filesystems it is actually allowed to replicate.
+* The server decides which filesystems it exposes to which peers.
-* The :ref:`transport mechanism <transport>` is decoupled from the remaining logic, keeping it extensible.
+* The :ref:`transport mechanism <transport>` is decoupled from the remaining logic, which allows us to painlessly offer multiple transport mechanisms.
 
 Protocol Implementation
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-zrepl implements its own RPC protocol.
+zrepl uses a custom RPC protol because, at the time of writing, existing solutions like gRPC do not provide efficient means to transport large amounts of data, whose size is unknown at send time (= zfs send streams).
-This is mostly due to the fact that existing solutions do not provide efficient means to transport large amounts of data.
+The package used is `github.com/problame/go-streamrpc <https://github.com/problame/go-streamrpc/tree/master>`_.
-
-Package `github.com/zrepl/zrepl/rpc <https://github.com/zrepl/zrepl/tree/master/rpc>`_ builds a special-case handling around returning an ``io.Reader`` as part of a unary RPC call.
-
-Measurements show only a single memory-to-memory copy of a snapshot stream is made using ``github.com/zrepl/zrepl/rpc``, and there is still potential for further optimizations.
 
 Logging & Transparency
 ----------------------

docs/index.rst

@@ -8,10 +8,15 @@
 zrepl - ZFS replication
 -----------------------
 
-.. ATTENTION::
+**zrepl** is a one-stop, integrated solution for ZFS replication.
-    zrepl as well as this documentation is still under active development.
+
-    It is neither feature complete nor is there a stability guarantee on the configuration format.
+.. raw:: html
-    Use & test at your own risk ;)
+
+   <div style="margin-bottom: 1em; background: #2e3436; min-height: 6em; max-width: 100%">
+     <a href="https://raw.githubusercontent.com/wiki/zrepl/zrepl/zrepl_0.1_status.mp4" target="_new" >
+       <video title="zrepl status subcommand" loop autoplay style="width: 100%; display: block;" src="https://raw.githubusercontent.com/wiki/zrepl/zrepl/zrepl_0.1_status.mp4"></video>
+     </a>
+   </div>
 
 Getting started
 ~~~~~~~~~~~~~~~
@@ -21,42 +26,47 @@ The :ref:`10 minutes tutorial setup <tutorial>` gives you a first impression.
 Main Features
 ~~~~~~~~~~~~~
 
-* Filesystem Replication
+* **Filesystem replication**
 
-  * [x] Local & Remote
+  * [x] Pull & Push mode
-  * [x] Pull mode
+  * [x] Multiple transport :ref:`transports <transport>`: TCP, TCP + TLS client auth, SSH
-  * [ ] Push mode
-  * [x] Access control checks when pulling datasets
-  * [x] :ref:`Flexible mapping <pattern-mapping>` rules
-  * [x] Bookmarks support
-  * [ ] Feature-negotiation for
 
-    * Resumable `send & receive`
+  * Advanced replication features
-    * Compressed `send & receive`
-    * Raw encrypted `send & receive` (as soon as it is available)
 
-* Automatic snapshot creation
+    * [ ] Resumable send & receive
+    * [ ] Compressed send & receive
+    * [ ] Raw encrypted send & receive
 
-  * [x] Ensure fixed time interval between snapshots
+* **Automatic snapshot management**
 
-* Automatic snapshot :ref:`pruning <prune>`
+  * [x] Periodic filesystem snapshots
+  * [x] Flexible :ref:`pruning rule system <prune>`
 
     * [x] Age-based fading (grandfathering scheme)
+    * [x] Bookmarks to avoid divergence between sender and receiver
 
-* Logging \& Monitoring
+* **Sophisticated Monitoring & Logging**
 
-  * Detailed & structured :ref:`logging <logging>`
+  * [x] Live progress reporting via `zrepl status` :ref:`subcommand <usage>`
+  * [x] Comprehensive, structured :ref:`logging <logging>`
 
     * ``human``, ``logfmt`` and ``json`` formatting
     * stdout, syslog and TCP (+TLS client auth) outlets
 
-  * Prometheus :ref:`monitoring <monitoring>` endpoint
+  * [x] Prometheus :ref:`monitoring <monitoring>` endpoint
 
-* Maintainable implementation in Go
+* **Maintainable implementation in Go**
 
   * [x] Cross platform
   * [x] Type safe & testable code
 
+
+.. ATTENTION::
+    zrepl as well as this documentation is still under active development.
+    There is no stability guarantee on the RPC protocol or configuration format,
+    but we do our best to document breaking changes in the :ref:`changelog`.
+
+
 Contributing
 ~~~~~~~~~~~~
 
@@ -88,6 +98,7 @@ Table of Contents
    configuration
    usage
    implementation
+   pr
    changelog
    GitHub Repository & Issue Tracker <https://github.com/zrepl/zrepl>
-   pr
+

docs/pr.rst

@@ -1,8 +1,17 @@
+
+.. _pr-talks:
+
 Talks & Presentations
 =====================
 
-* Talk at EuroBSDCon2017 FreeBSD DevSummit (
+* Talk at OpenZFS Developer Summit 2018 of pre-release 0.1 (
-  `Slides <https://docs.google.com/presentation/d/1EmmeEvOXAWJHCVnOS9-TTsxswbcGKmeLWdY_6BH4w0Q/edit?usp=sharing>`_,
+  `25min Recording <https://www.youtube.com/watch?v=U4TUPQzZzPk&index=4&list=PLaUVvul17xSe0pC6sCirlZXYqICP09Y8z&t=0s>`__ ,
-  `Event <https://wiki.freebsd.org/DevSummit/201709>`_
+  `Slides <https://docs.google.com/presentation/d/1HXJ_9Q0kiHffeoQ7PlFn1qAfUcWyzedRgFlvL3nUlMU/edit?usp=sharing>`__ ,
+  `Event <http://www.open-zfs.org/wiki/OpenZFS_Developer_Summit_2018>`__
+  )
+* Talk at EuroBSDCon2017 FreeBSD DevSummit with live demo of zrepl 0.0.3 (
+  `55min Recording <https://www.youtube.com/watch?v=c1LKeyP1mos&t=2316s>`__,
+  `Slides <https://docs.google.com/presentation/d/1EmmeEvOXAWJHCVnOS9-TTsxswbcGKmeLWdY_6BH4w0Q/edit?usp=sharing>`__,
+  `Event <https://wiki.freebsd.org/DevSummit/201709>`__
   )
 

docs/sphinxconf/conf.py

@@ -101,6 +101,17 @@ html_static_path = ['../_static']
 
 html_logo = '../_static/zrepl.svg'
 
+html_context = {
+  # https://github.com/rtfd/sphinx_rtd_theme/issues/205
+  # Add 'Edit on Github' link instead of 'View page source'
+  "display_github": True, 
+  "github_user": "zrepl",
+  "github_repo": "zrepl",
+  "github_version": "master",
+  "conf_py_path": "/docs/",
+  "source_suffix": source_suffix,
+}
+
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
@@ -162,7 +173,7 @@ texinfo_documents = [
 # http://www.sphinx-doc.org/en/stable/ext/extlinks.html
 extlinks = {
     'issue':('https://github.com/zrepl/zrepl/issues/%s', 'issue #'),
-    'sampleconf':('https://github.com/zrepl/zrepl/blob/master/cmd/sampleconf/%s', 'cmd/sampleconf/'),
+    'sampleconf':('https://github.com/zrepl/zrepl/blob/master/config/samples%s', 'config/samples'),
     'commit':('https://github.com/zrepl/zrepl/commit/%s', 'commit '),
         }
 

docs/tutorial.rst

@@ -9,164 +9,120 @@ Tutorial
 This tutorial shows how zrepl can be used to implement a ZFS-based pull backup.
 We assume the following scenario:
 
-* Production server ``app-srv`` with filesystems to back up:
+* Production server ``prod`` with filesystems to back up:
 
   * ``zroot/var/db``
   * ``zroot/usr/home`` and all its child filesystems
   * **except** ``zroot/usr/home/paranoid`` belonging to a user doing backups themselves
 
-* Backup server ``backup-srv`` with
+* Backup server ``backups`` with
 
-  * Filesystem ``storage/zrepl/pull/app-srv`` + children dedicated to backups of ``app-srv``
+  * Filesystem ``storage/zrepl/pull/prod`` + children dedicated to backups of ``prod``
 
 Our backup solution should fulfill the following requirements:
 
-* Periodically snapshot the filesystems on ``app-srv`` *every 10 minutes*
+* Periodically snapshot the filesystems on ``prod`` *every 10 minutes*
-* Incrementally replicate these snapshots to ``storage/zrepl/pull/app-srv/*`` on ``backup-srv``
+* Incrementally replicate these snapshots to ``storage/zrepl/pull/prod/*`` on ``backups``
-* Keep only very few snapshots on ``app-srv`` to save disk space
+* Keep only very few snapshots on ``prod`` to save disk space
-* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on ``backup-srv``
+* Keep a fading history (24 hourly, 30 daily, 6 monthly) of snapshots on ``backups``
 
 Analysis
 --------
 
 We can model this situation as two jobs:
 
-* A **source job** on ``app-srv``
+* A **source job** on ``prod``
 
   * Creates the snapshots
-  * Keeps a short history of snapshots to enable incremental replication to ``backup-srv``
+  * Keeps a short history of snapshots to enable incremental replication to ``backups``
-  * Accepts connections from ``backup-srv``
+  * Accepts connections from ``backups``
 
-* A **pull job** on ``backup-srv``
+* A **pull job** on ``backups``
 
-  * Connects to the ``zrepl daemon`` process on ``app-srv``
+  * Connects to the ``zrepl daemon`` process on ``prod``
-  * Pulls the snapshots to ``storage/zrepl/pull/app-srv/*``
+  * Pulls the snapshots to ``storage/zrepl/pull/prod/*``
-  * Fades out snapshots in ``storage/zrepl/pull/app-srv/*`` as they age
+  * Fades out snapshots in ``storage/zrepl/pull/prod/*`` as they age
 
 
 Why doesn't the **pull job** create the snapshots before pulling?
 
-As is the case with all distributed systems, the link between ``app-srv`` and ``backup-srv`` might be down for an hour or two.
+As is the case with all distributed systems, the link between ``prod`` and ``backups`` might be down for an hour or two.
 We do not want to sacrifice our required backup resolution of 10 minute intervals for a temporary connection outage.
 
-When the link comes up again, ``backup-srv`` will happily catch up the 12 snapshots taken by ``app-srv`` in the meantime, without
+When the link comes up again, ``backups`` will catch up with the snapshots taken by ``prod`` in the meantime, without a gap in our backup history.
-a gap in our backup history.
 
 Install zrepl
 -------------
 
 Follow the :ref:`OS-specific installation instructions <installation>` and come back here.
 
-Configure ``backup-srv``
+Configure server ``backups``
-------------------------
+----------------------------
 
-We define a **pull job** named ``pull_app-srv`` in the |mainconfig| on host ``backup-srv``: ::
+We define a **pull job** named ``pull_prod`` in ``/etc/zrepl/zrepl.yml`` or ``/usr/local/etc/zrepl/zrepl.yml`` on host ``backups`` : ::
 
     jobs:
-    - name: pull_app-srv
+    - name: pull_prod
       type: pull
       connect:
-          type: ssh+stdinserver
+        type: tcp
-          host: app-srv.example.com
+        address: "192.168.2.20:2342"
-          user: root
+      root_fs: "storage/zrepl/pull/prod"
-          port: 22
+      interval: 10m
-          identity_file: /etc/zrepl/ssh/identity
+      pruning:
+        keep_sender:
+        - type: not_replicated
+        - type: last_n
+          count: 10
+        keep_receiver:
+        - type: grid
+          grid: 1x1h(keep=all) | 24x1h | 30x1d | 6x30d
+          regex: "^zrepl_"
           interval: 10m
-      mapping: {
-          "<":"storage/zrepl/pull/app-srv"
-      }
-      snapshot_prefix: zrepl_pull_backup_
-      prune:
-          policy: grid
-          grid: 1x1h(keep=all) | 24x1h | 35x1d | 6x30d
 
-The ``connect`` section instructs the zrepl daemon to use the ``stdinserver`` transport:
+The ``connect`` section instructs the zrepl daemon to use plain TCP transport.
-``backup-srv`` will connect to the specified SSH server and expect ``zrepl stdinserver CLIENT_IDENTITY`` instead of the shell on the other side.
+Check out the :ref:`transports <transport>` section for alternatives that support encryption.
 
-It uses the private key specified at ``connect.identity_file`` which we still need to create: ::
+.. _tutorial-configure-prod:
 
-    cd /etc/zrepl
+Configure server ``prod``
-    mkdir -p ssh
+-------------------------
-    chmod 0700 ssh
-    ssh-keygen -t ed25519 -N '' -f /etc/zrepl/ssh/identity
 
-Note that most use cases do not benefit from separate keypairs per remote endpoint.
+We define a corresponding **source job** named ``source_backups`` in ``/etc/zrepl/zrepl.yml`` or ``/usr/local/etc/zrepl/zrepl.yml`` on host ``prod`` : ::
-Thus, it is sufficient to create one keypair and use it for all ``connect`` directives on one host.
-
-zrepl uses ssh's default ``known_hosts`` file, which must contain a host identification entry for ``app-srv.example.com``.
-If that entry does not already exist, we need to generate it.
-Run the following command, compare the host fingerprints, and confirm with yes if they match.
-You will not be able to get a shell with the identity file we just generated, which is fine. ::
-
-    ssh -i /etc/zrepl/ssh/identity root@app-srv.example.com
-
-Learn more about :ref:`transport-ssh+stdinserver` transport and the :ref:`pull job <job-pull>` format.
-
-.. _tutorial-configure-app-srv:
-
-Configure ``app-srv``
----------------------
-
-We define a corresponding **source job** named ``pull_backup`` in the |mainconfig| on host ``app-srv``:  ::
 
     jobs:
-    - name: pull_backup
+    - name: source_backups
       type: source
       serve:
-          type: stdinserver
+        type: tcp
-          client_identity: backup-srv.example.com
+        listen: ":2342"
-      filesystems: {
+        clients: {
-          "zroot/var/db": "ok",
+          "192.168.2.10" : "backups"
-          "zroot/usr/home<": "ok",
-          "zroot/usr/home/paranoid": "!",
         }
-      snapshot_prefix: zrepl_pull_backup_
+      filesystems: {
+        "zroot/var/db:": true,
+        "zroot/usr/home<": true,
+        "zroot/usr/home/paranoid": false
+      }
+      snapshotting:
+        type: periodic
+        prefix: zrepl_
         interval: 10m
-      prune:
-          policy: grid
-          grid: 1x1d(keep=all)
-          keep_bookmarks: 144
 
 
-The ``serve`` section corresponds to the ``connect`` section in the configuration of ``backup-srv``.
+The ``serve`` section whitelists ``backups``'s IP address ``192.168.2.10`` and assigns it the client identity ``backups`` which will show up in the logs.
-
+Again, check the :ref:`docs for encrypted transports <transport>`.
-We now want to authenticate ``backup-srv`` before allowing it to pull data.
-This is done by limiting SSH connections from ``backup-srv`` to execute the ``stdinserver`` subcommand.
-
-Open ``/root/.ssh/authorized_keys`` and add either of the the following lines.::
-
-    # for OpenSSH >= 7.2
-    command="zrepl stdinserver backup-srv.example.com",restrict CLIENT_SSH_KEY
-    # for older OpenSSH versions
-    command="zrepl stdinserver backup-srv.example.com",no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc  CLIENT_SSH_KEY
-
-.. ATTENTION::
-
-    Replace CLIENT_SSH_KEY with the contents of ``/etc/zrepl/ssh/identity.pub`` from ``app-srv``.
-    Mind the trailing ``.pub`` in the filename.
-    The entries **must** be on a single line, including the replaced CLIENT_SSH_KEY.
-
-
-.. HINT::
-
-    You may need to adjust the ``PermitRootLogin`` option in ``/etc/ssh/sshd_config`` to ``forced-commands-only`` or higher for this to work.
-    Refer to sshd_config(5) for details.
-
-The argument ``backup-srv.example.com`` is the client identity of ``backup-srv`` as defined in ``jobs.serve.client_identity``.
-
-Again, both :ref:`transport-ssh+stdinserver` transport and the :ref:`job-source` format are documented.
 
 Apply Configuration Changes
 ---------------------------
 
-We need to restart the zrepl daemon on **both** ``app-srv`` and ``backup-srv``.
+We need to restart the zrepl daemon on **both** ``prod`` and ``backups``.
 This is :ref:`OS-specific <usage-zrepl-daemon-restarting>`.
 
 Watch it Work
 -------------
 
-Run ``zrepl control status`` to view the current activity of the configured jobs.
+Run ``zrepl status`` on ``prod`` to monitor the replication and pruning activity.
-If a job encountered problems since it last left idle state, the output contains useful debug log.
 
 Additionally, you can check the detailed structured logs of the `zrepl daemon` process and use GNU *watch* to view the snapshots present on both machines.
 
@@ -175,7 +131,7 @@ If you like tmux, here is a handy script that works on FreeBSD: ::
     pkg install gnu-watch tmux
     tmux new-window
     tmux split-window "tail -f /var/log/zrepl.log"
-    tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
+    tmux split-window "gnu-watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_'"
     tmux select-layout tiled
 
 The Linux equivalent might look like this: ::
@@ -183,7 +139,7 @@ The Linux equivalent might look like this: ::
     # make sure tmux is installed & let's assume you use systemd + journald
     tmux new-window
     tmux split-window "journalctl -f -u zrepl.service"
-    tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_pull_backup_'"
+    tmux split-window "watch 'zfs list -t snapshot -o name,creation -s creation | grep zrepl_'"
     tmux select-layout tiled
 
 Summary

docs/usage.rst

@@ -10,8 +10,8 @@ CLI Overview
 
 .. NOTE::
 
-    To avoid duplication, the zrepl binary is self-documenting:
+    The zrepl binary is self-documenting:
-    invoke any subcommand at any level with the ``--help`` flag to get information on the subcommand, available flags, etc.
+    run ``zrepl help`` for an overview of the available subcommands or ``zrepl SUBCOMMAND --help`` for information on available flags, etc.
 
 .. list-table::
     :widths: 30 70
@@ -19,16 +19,18 @@ CLI Overview
 
     * - Subcommand
       - Description
+    * - ``zrepl help``
+      - show subcommand overview
     * - ``zrepl daemon``
       - run the daemon, required for all zrepl functionality
-    * - ``zrepl control``
+    * - ``zrepl status``
-      - control / query the daemon
+      - show job activity, or with ``--raw`` for JSON output
-    * - ``zrepl control status``
-      - show job activity / monitoring (``--format raw``)
-    * - ``zrepl test``
-      - test configuration, try pattern syntax, dry run pruning policy, etc.
     * - ``zrepl stdinserver``
       - see :ref:`transport-ssh+stdinserver`
+    * - ``zrepl wakeup JOB``
+      - manually trigger replication + pruning
+    * - ``zrepl configcheck``
+      - check if config can be parsed without errors
 
 .. _usage-zrepl-daemon: