From 98207c904dde65bda970bed948f577625be34cad Mon Sep 17 00:00:00 2001 From: Christian Schwarz Date: Sun, 6 Sep 2020 16:40:12 +0200 Subject: [PATCH] docs + README: document new build & release process --- README.md | 137 +++++++++------------- docs/installation/compile-from-source.rst | 44 ++++--- 2 files changed, 78 insertions(+), 103 deletions(-) diff --git a/README.md b/README.md index 6ad3c87..9529005 100644 --- a/README.md +++ b/README.md @@ -30,20 +30,12 @@ zrepl is a one-stop ZFS backup & replication solution. The above does not apply if you already implemented everything. Check out the *Coding Workflow* section below for details. -## Package Maintainer Information +## Building, Releasing, Downstream-Packaging -* Follow the steps in `docs/installation.rst -> Compiling from Source` and read the Makefile / shell scripts used in this process. -* Make sure your distro is compatible with the paths in `docs/installation.rst`. -* Ship a default config that adheres to your distro's `hier` and logging system. -* Ship a service manager file and _please_ try to upstream it to this repository. - * `dist/systemd` contains a Systemd unit template. -* Ship other material provided in `./dist`, e.g. in `/usr/share/zrepl/`. -* Use `make release ZREPL_VERSION='mydistro-1.2.3_1'` - * Your distro's name and any versioning supplemental to zrepl's (e.g. package revision) should be in this string -* Use `sudo make test-platform` **on a test system** to validate that zrepl's abstractions on top of ZFS work with the system ZFS. -* Make sure you are informed about new zrepl versions, e.g. by subscribing to GitHub's release RSS feed. +This section provides an overview of the zrepl build & release process. +Check out `docs/installation/compile-from-source.rst` for build-from-source instructions. -## Developer Documentation +### Overview zrepl is written in [Go](https://golang.org) and uses [Go modules](https://github.com/golang/go/wiki/Modules) to manage dependencies. The documentation is written in [ReStructured Text](http://docutils.sourceforge.net/rst.html) using the [Sphinx](https://www.sphinx-doc.org) framework. @@ -60,60 +52,61 @@ An HTML report can be generated using `make cover-html`. **Code generation** is triggered by `make generate`. Generated code is committed to the source tree. -### Project Structure +### Build & Release Process -``` -├── artifacts # build artifcats generate by make -├── cli # wrapper around CLI package cobra -├── client # all subcommands that are not `daemon` -├── config # config data types (=> package yaml-config) -│   └── samples -├── daemon # the implementation of `zrepl daemon` subcommand -│   ├── filters -│   ├── hooks # snapshot hooks -│   ├── job # job implementations -│   ├── logging # logging outlets + formatters -│   ├── nethelpers -│   ├── prometheus -│   ├── pruner # pruner implementation -│   ├── snapper # snapshotter implementation -├── dist # supplemental material for users & package maintainers -├── docs # sphinx-based documentation -│   ├── **/*.rst # documentation in reStructuredText -│   ├── sphinxconf -│   │   └── conf.py # sphinx config (see commit 445a280 why its not in docs/) -│   ├── requirements.txt # pip3 requirements to build documentation -│   ├── publish.sh # shell script for automated rendering & deploy to zrepl.github.io repo -│   └── public_git # checkout of zrepl.github.io managed by above shell script -├── endpoint # implementation of replication endpoints (=> package replication) -├── logger # our own logger package -├── platformtest # test suite for our zfs abstractions (error classification, etc) -├── pruning # pruning rules (the logic, not the actual execution) -│   └── retentiongrid -├── replication -│ ├── driver # the driver of the replication logic (status reporting, error handling) -│ ├── logic # planning & executing replication steps via rpc -| |   └── pdu # the generated gRPC & protobuf code used in replication (and endpoints) -│ └── report # the JSON-serializable report datastructures exposed to the client -├── rpc # the hybrid gRPC + ./dataconn RPC client: connects to a remote replication.Endpoint -│ ├── dataconn # Bulk data-transfer RPC protocol -│ ├── grpcclientidentity # adaptor to inject package transport's 'client identity' concept into gRPC contexts -│ ├── netadaptor # adaptor to convert a package transport's Connecter and Listener into net.* primitives -│ ├── transportmux # TCP connecter and listener used to split control & data traffic -│ └── versionhandshake # replication protocol version handshake perfomed on newly established connections -├── tlsconf # abstraction for Go TLS server + client config -├── transport # transport implementations -│ ├── fromconfig -│ ├── local -│ ├── ssh -│ ├── tcp -│ └── tls -├── util -├── version # abstraction for versions (filled during build by Makefile) -└── zfs # zfs(8) wrappers -``` +**The `Makefile` is catering to the needs of developers & CI, not distro packagers**. +It provides phony targets for +* local development (building, running tests, etc) +* building a release in Docker (used by the CI & release management) +* building .deb and .rpm packages out of the release artifacts. -### Coding Workflow +**Build tooling & dependencies** are documented as code in `lazy.sh`. +Go dependencies are then fetched by the go command and pip dependencies are pinned through a `requirements.txt`. + +**We use CircleCI for continuous integration**. +There are two workflows: + +* `ci` runs for every commit / branch / tag pushed to GitHub. + It is supposed to run very fast (<5min and provides quick feedback to developers). + It runs formatting checks, lints and tests on the most important OSes / architectures. + Artifacts are published to minio.cschwarz.com (see GitHub Commit Status). + +* `release` runs + * on manual triggers through the CircleCI API (in order to produce a release) + * periodically on `master` + Artifacts are published to minio.cschwarz.com (see GitHub Commit Status). + +**Releases** are issued via Git tags + GitHub Releases feature. +The procedure to issue a release is as follows: +* Issue the source release: + * Git tag the release on the `master` branch. + * Push the tag. + * Run `./docs/publish.sh` to re-build & push zrepl.github.io. +* Issue the official binary release: + * Run the `release` pipeline (triggered via CircleCI API) + * Download the artifacts to the release manager's machine. + * Create a GitHub release, edit the changelog, upload all the release artifacts, including .rpm and .deb files. + * Issue the GitHub release. + * Add the .rpm and .deb files to the official zrepl repos, publish those. + +**Official binary releases are not re-built when Go receives an update. If the Go update is critical to zrepl (e.g. a Go security update that affects zrepl), we'd issue a new source release**. +The rationale for this is that whereas distros provide a mechanism for this (`$zrepl_source_release-$distro_package_revision`), GitHub Releases doesn't which means we'd need to update the existing GitHub release's assets, which nobody would notice (no RSS feed updates, etc.). +Downstream packagers can read the changelog to determine whether they want to push that minor release into their distro or simply skip it. + +### Additional Notes to Distro Package Maintainers + +* Use `sudo make test-platform-bin && sudo make test-platform` **on a test system** to validate that zrepl's abstractions on top of ZFS work with the system ZFS. +* Ship a default config that adheres to your distro's `hier` and logging system. +* Ship a service manager file and _please_ try to upstream it to this repository. + * `dist/systemd` contains a Systemd unit template. +* Ship other material provided in `./dist`, e.g. in `/usr/share/zrepl/`. +* Have a look at the `Makefile`'s `ZREPL_VERSION` variable and how it passed to Go's `ldFlags`. + This is how `zrepl version` knows what version number to show. + Your build system should set the `ldFlags` flags appropriately and add a prefix or suffix that indicates that the given zrepl binary is a distro build, not an official one. +* Make sure you are informed about new zrepl versions, e.g. by subscribing to GitHub's release RSS feed. + + +## Contributing Code * Open an issue when starting to hack on a new feature * Commits should reference the issue they are related to @@ -123,9 +116,6 @@ An HTML report can be generated using `make cover-html`. Backward-incompatible changes must be documented in the git commit message and are listed in `docs/changelog.rst`. -* Config-breaking changes must contain a line `BREAK CONFIG` in the commit message -* Other breaking changes must contain a line `BREAK` in the commit message - ### Glossary & Naming Inconsistencies In ZFS, *dataset* refers to the objects *filesystem*, *ZVOL* and *snapshot*.
@@ -142,16 +132,3 @@ variables and types are often named *dataset* when they in fact refer to a *file There will not be a big refactoring (an attempt was made, but it's destroying too much history without much gain). However, new contributions & patches should fix naming without further notice in the commit message. - -### RPC debugging - -Optionally, there are various RPC-related environment variables, that if set to something != `""` will produce additional debug output on stderr: - -https://github.com/zrepl/zrepl/blob/master/rpc/rpc_debug.go#L11 - -https://github.com/zrepl/zrepl/blob/master/rpc/dataconn/dataconn_debug.go#L11 - -https://github.com/zrepl/zrepl/blob/master/rpc/dataconn/stream/stream_debug.go#L11 - -https://github.com/zrepl/zrepl/blob/master/rpc/dataconn/heartbeatconn/heartbeatconn_debug.go#L11 - diff --git a/docs/installation/compile-from-source.rst b/docs/installation/compile-from-source.rst index 63a422f..77dbc11 100644 --- a/docs/installation/compile-from-source.rst +++ b/docs/installation/compile-from-source.rst @@ -9,37 +9,35 @@ Producing a release requires **Go 1.11** or newer and **Python 3** + **pip3** + A tutorial to install Go is available over at `golang.org `_. Python and pip3 should probably be installed via your distro's package manager. +:: + cd to/your/zrepl/checkout + python3 -m venv3 + source venv3/bin/activate + ./lazy.sh devsetup + make release + # build artifacts are available in ./artifacts/release + +The Python venv is used for the documentation build dependencies. +If you just want to build the zrepl binary, leave it out and use `./lazy.sh godep` instead. + Alternatively, you can use the Docker build process: it is used to produce the official zrepl `binary releases`_ and serves as a reference for build dependencies and procedure: :: - git clone https://github.com/zrepl/zrepl.git && \ - cd zrepl && \ - sudo docker build -t zrepl_build -f build.Dockerfile . && \ - sudo docker run -it --rm \ - -v "${PWD}:/src" \ - --user "$(id -u):$(id -g)" \ - zrepl_build make release + cd to/your/zrepl/checkout + # make sure your user has access to the docker socket + make release-docker + # if you want .deb or .rpm packages, invoke the follwoing + # targets _after_ you invoked release-docker + make deb-docker + make rpm-docker + # build artifacts are available in ./artifacts/release + # packages are available in ./artifacts -Alternatively, you can install build dependencies on your local system and then build in your ``$GOPATH``: - -:: - - mkdir -p "${GOPATH}/src/github.com/zrepl/zrepl" - git clone https://github.com/zrepl/zrepl.git "${GOPATH}/src/github.com/zrepl/zrepl" - cd "${GOPATH}/src/github.com/zrepl/zrepl" - python3 -m venv3 - source venv3/bin/activate - ./lazy.sh devsetup - make release - -The Python venv is used for the documentation build dependencies. -If you just want to build the zrepl binary, leave it out and use `./lazy.sh godep` instead. -Either way, all build results are located in the ``artifacts/`` directory. .. NOTE:: - It is your job to install the appropriate binary in the zrepl users's ``$PATH``, e.g. ``/usr/local/bin/zrepl``. + It is your job to install the built binary in the zrepl users's ``$PATH``, e.g. ``/usr/local/bin/zrepl``. Otherwise, the examples in the :ref:`quick-start guides ` may need to be adjusted.