extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-22 00:13:52 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs + README: document new build & release process

Browse Source
This commit is contained in:
Christian Schwarz 2020-09-06 16:40:12 +02:00
parent d2b825a8ae
commit 98207c904d
2 changed files with 78 additions and 103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

137
README.md
View File

@ -30,20 +30,12 @@ zrepl is a one-stop ZFS backup & replication solution.
The above does not apply if you already implemented everything. The above does not apply if you already implemented everything.
Check out the *Coding Workflow* section below for details. Check out the *Coding Workflow* section below for details.
## Package Maintainer Information ## Building, Releasing, Downstream-Packaging
* Follow the steps in `docs/installation.rst -> Compiling from Source` and read the Makefile / shell scripts used in this process. This section provides an overview of the zrepl build & release process.
* Make sure your distro is compatible with the paths in `docs/installation.rst`. Check out `docs/installation/compile-from-source.rst` for build-from-source instructions.
* Ship a default config that adheres to your distro's `hier` and logging system.
* Ship a service manager file and _please_ try to upstream it to this repository.
* `dist/systemd` contains a Systemd unit template.
* Ship other material provided in `./dist`, e.g. in `/usr/share/zrepl/`.
* Use `make release ZREPL_VERSION='mydistro-1.2.3_1'`
* Your distro's name and any versioning supplemental to zrepl's (e.g. package revision) should be in this string
* Use `sudo make test-platform` **on a test system** to validate that zrepl's abstractions on top of ZFS work with the system ZFS.
* Make sure you are informed about new zrepl versions, e.g. by subscribing to GitHub's release RSS feed.
## Developer Documentation ### Overview
zrepl is written in [Go](https://golang.org) and uses [Go modules](https://github.com/golang/go/wiki/Modules) to manage dependencies. zrepl is written in [Go](https://golang.org) and uses [Go modules](https://github.com/golang/go/wiki/Modules) to manage dependencies.
The documentation is written in [ReStructured Text](http://docutils.sourceforge.net/rst.html) using the [Sphinx](https://www.sphinx-doc.org) framework. The documentation is written in [ReStructured Text](http://docutils.sourceforge.net/rst.html) using the [Sphinx](https://www.sphinx-doc.org) framework.
@ -60,60 +52,61 @@ An HTML report can be generated using `make cover-html`.
**Code generation** is triggered by `make generate`. Generated code is committed to the source tree. **Code generation** is triggered by `make generate`. Generated code is committed to the source tree.
### Project Structure ### Build & Release Process
``` **The `Makefile` is catering to the needs of developers & CI, not distro packagers**.
├── artifacts # build artifcats generate by make It provides phony targets for
├── cli # wrapper around CLI package cobra * local development (building, running tests, etc)
├── client # all subcommands that are not `daemon` * building a release in Docker (used by the CI & release management)
├── config # config data types (=> package yaml-config) * building .deb and .rpm packages out of the release artifacts.
│   └── samples
├── daemon # the implementation of `zrepl daemon` subcommand
│   ├── filters
│   ├── hooks # snapshot hooks
│   ├── job # job implementations
│   ├── logging # logging outlets + formatters
│   ├── nethelpers
│   ├── prometheus
│   ├── pruner # pruner implementation
│   ├── snapper # snapshotter implementation
├── dist # supplemental material for users & package maintainers
├── docs # sphinx-based documentation
│   ├── **/*.rst # documentation in reStructuredText
│   ├── sphinxconf
│   │   └── conf.py # sphinx config (see commit 445a280 why its not in docs/)
│   ├── requirements.txt # pip3 requirements to build documentation
│   ├── publish.sh # shell script for automated rendering & deploy to zrepl.github.io repo
│   └── public_git # checkout of zrepl.github.io managed by above shell script
├── endpoint # implementation of replication endpoints (=> package replication)
├── logger # our own logger package
├── platformtest # test suite for our zfs abstractions (error classification, etc)
├── pruning # pruning rules (the logic, not the actual execution)
│   └── retentiongrid
├── replication
│ ├── driver # the driver of the replication logic (status reporting, error handling)
│ ├── logic # planning & executing replication steps via rpc
| |   └── pdu # the generated gRPC & protobuf code used in replication (and endpoints)
│ └── report # the JSON-serializable report datastructures exposed to the client
├── rpc # the hybrid gRPC + ./dataconn RPC client: connects to a remote replication.Endpoint
│ ├── dataconn # Bulk data-transfer RPC protocol
│ ├── grpcclientidentity # adaptor to inject package transport's 'client identity' concept into gRPC contexts
│ ├── netadaptor # adaptor to convert a package transport's Connecter and Listener into net.* primitives
│ ├── transportmux # TCP connecter and listener used to split control & data traffic
│ └── versionhandshake # replication protocol version handshake perfomed on newly established connections
├── tlsconf # abstraction for Go TLS server + client config
├── transport # transport implementations
│ ├── fromconfig
│ ├── local
│ ├── ssh
│ ├── tcp
│ └── tls
├── util
├── version # abstraction for versions (filled during build by Makefile)
└── zfs # zfs(8) wrappers
```
### Coding Workflow **Build tooling & dependencies** are documented as code in `lazy.sh`.
Go dependencies are then fetched by the go command and pip dependencies are pinned through a `requirements.txt`.
**We use CircleCI for continuous integration**.
There are two workflows:
* `ci` runs for every commit / branch / tag pushed to GitHub.
It is supposed to run very fast (<5min and provides quick feedback to developers).
It runs formatting checks, lints and tests on the most important OSes / architectures.
Artifacts are published to minio.cschwarz.com (see GitHub Commit Status).
* `release` runs
* on manual triggers through the CircleCI API (in order to produce a release)
* periodically on `master`
Artifacts are published to minio.cschwarz.com (see GitHub Commit Status).
**Releases** are issued via Git tags + GitHub Releases feature.
The procedure to issue a release is as follows:
* Issue the source release:
* Git tag the release on the `master` branch.
* Push the tag.
* Run `./docs/publish.sh` to re-build & push zrepl.github.io.
* Issue the official binary release:
* Run the `release` pipeline (triggered via CircleCI API)
* Download the artifacts to the release manager's machine.
* Create a GitHub release, edit the changelog, upload all the release artifacts, including .rpm and .deb files.
* Issue the GitHub release.
* Add the .rpm and .deb files to the official zrepl repos, publish those.
**Official binary releases are not re-built when Go receives an update. If the Go update is critical to zrepl (e.g. a Go security update that affects zrepl), we'd issue a new source release**.
The rationale for this is that whereas distros provide a mechanism for this (`$zrepl_source_release-$distro_package_revision`), GitHub Releases doesn't which means we'd need to update the existing GitHub release's assets, which nobody would notice (no RSS feed updates, etc.).
Downstream packagers can read the changelog to determine whether they want to push that minor release into their distro or simply skip it.
### Additional Notes to Distro Package Maintainers
* Use `sudo make test-platform-bin && sudo make test-platform` **on a test system** to validate that zrepl's abstractions on top of ZFS work with the system ZFS.
* Ship a default config that adheres to your distro's `hier` and logging system.
* Ship a service manager file and _please_ try to upstream it to this repository.
* `dist/systemd` contains a Systemd unit template.
* Ship other material provided in `./dist`, e.g. in `/usr/share/zrepl/`.
* Have a look at the `Makefile`'s `ZREPL_VERSION` variable and how it passed to Go's `ldFlags`.
This is how `zrepl version` knows what version number to show.
Your build system should set the `ldFlags` flags appropriately and add a prefix or suffix that indicates that the given zrepl binary is a distro build, not an official one.
* Make sure you are informed about new zrepl versions, e.g. by subscribing to GitHub's release RSS feed.
## Contributing Code
* Open an issue when starting to hack on a new feature * Open an issue when starting to hack on a new feature
* Commits should reference the issue they are related to * Commits should reference the issue they are related to
@ -123,9 +116,6 @@ An HTML report can be generated using `make cover-html`.
Backward-incompatible changes must be documented in the git commit message and are listed in `docs/changelog.rst`. Backward-incompatible changes must be documented in the git commit message and are listed in `docs/changelog.rst`.
* Config-breaking changes must contain a line `BREAK CONFIG` in the commit message
* Other breaking changes must contain a line `BREAK` in the commit message
### Glossary & Naming Inconsistencies ### Glossary & Naming Inconsistencies
In ZFS, *dataset* refers to the objects *filesystem*, *ZVOL* and *snapshot*. <br /> In ZFS, *dataset* refers to the objects *filesystem*, *ZVOL* and *snapshot*. <br />
@ -142,16 +132,3 @@ variables and types are often named *dataset* when they in fact refer to a *file
There will not be a big refactoring (an attempt was made, but it's destroying too much history without much gain). There will not be a big refactoring (an attempt was made, but it's destroying too much history without much gain).
However, new contributions & patches should fix naming without further notice in the commit message. However, new contributions & patches should fix naming without further notice in the commit message.
### RPC debugging
Optionally, there are various RPC-related environment variables, that if set to something != `""` will produce additional debug output on stderr:
https://github.com/zrepl/zrepl/blob/master/rpc/rpc_debug.go#L11
https://github.com/zrepl/zrepl/blob/master/rpc/dataconn/dataconn_debug.go#L11
https://github.com/zrepl/zrepl/blob/master/rpc/dataconn/stream/stream_debug.go#L11
https://github.com/zrepl/zrepl/blob/master/rpc/dataconn/heartbeatconn/heartbeatconn_debug.go#L11

44
docs/installation/compile-from-source.rst
View File

@ -9,37 +9,35 @@ Producing a release requires **Go 1.11** or newer and **Python 3** + **pip3** +
A tutorial to install Go is available over at `golang.org <https://golang.org/doc/install>`_. A tutorial to install Go is available over at `golang.org <https://golang.org/doc/install>`_.
Python and pip3 should probably be installed via your distro's package manager. Python and pip3 should probably be installed via your distro's package manager.
::
cd to/your/zrepl/checkout
python3 -m venv3
source venv3/bin/activate
./lazy.sh devsetup
make release
# build artifacts are available in ./artifacts/release
The Python venv is used for the documentation build dependencies.
If you just want to build the zrepl binary, leave it out and use `./lazy.sh godep` instead.
Alternatively, you can use the Docker build process: Alternatively, you can use the Docker build process:
it is used to produce the official zrepl `binary releases`_ it is used to produce the official zrepl `binary releases`_
and serves as a reference for build dependencies and procedure: and serves as a reference for build dependencies and procedure:
:: ::
git clone https://github.com/zrepl/zrepl.git && \ cd to/your/zrepl/checkout
cd zrepl && \ # make sure your user has access to the docker socket
sudo docker build -t zrepl_build -f build.Dockerfile . && \ make release-docker
sudo docker run -it --rm \ # if you want .deb or .rpm packages, invoke the follwoing
-v "${PWD}:/src" \ # targets _after_ you invoked release-docker
--user "$(id -u):$(id -g)" \ make deb-docker
zrepl_build make release make rpm-docker
# build artifacts are available in ./artifacts/release
# packages are available in ./artifacts
Alternatively, you can install build dependencies on your local system and then build in your ``$GOPATH``:
::
mkdir -p "${GOPATH}/src/github.com/zrepl/zrepl"
git clone https://github.com/zrepl/zrepl.git "${GOPATH}/src/github.com/zrepl/zrepl"
cd "${GOPATH}/src/github.com/zrepl/zrepl"
python3 -m venv3
source venv3/bin/activate
./lazy.sh devsetup
make release
The Python venv is used for the documentation build dependencies.
If you just want to build the zrepl binary, leave it out and use `./lazy.sh godep` instead.
Either way, all build results are located in the ``artifacts/`` directory.
.. NOTE:: .. NOTE::
It is your job to install the appropriate binary in the zrepl users's ``$PATH``, e.g. ``/usr/local/bin/zrepl``. It is your job to install the built binary in the zrepl users's ``$PATH``, e.g. ``/usr/local/bin/zrepl``.
Otherwise, the examples in the :ref:`quick-start guides <quickstart-toc>` may need to be adjusted. Otherwise, the examples in the :ref:`quick-start guides <quickstart-toc>` may need to be adjusted.