From 3d77080e71c839d087272abc915d3fab58fd0c27 Mon Sep 17 00:00:00 2001 From: kev <31861128+miooochi@users.noreply.github.com> Date: Sun, 7 Apr 2024 21:32:48 +0800 Subject: [PATCH] docs: add host-custom-binary-cache-with-s3.md (#140) * fix: correct a typo * docs: add host-custom-binary-cache-with-s3.md * docs(zh): add host-custom-binary-cache-with-s3.md * feat(config.ts): enable routing * patch: reordering * fix: fix inconsistent typos * fix: use nix run nixpkgs#pwgen as per suggestion * fix: use MinIO as per suggestion * feat: add background * patch: rework nix-cache-info creation * patch: clarify nix-cache-write.json usage * fix: comment console output line as per suggestion * fix: apply minior fixes * fix: improve clarify as per suggestion * feat: add prerequisites * refactor: rework zh version * fix: fix inconsistent quotations * feat: add step to create aws-credentials * style: add spaces between English and Chinese Characters --------- Co-authored-by: kev <31861128+yqlbu@users.noreply.github.com> Co-authored-by: Ryan Yin --- docs/.vitepress/config.ts | 8 + .../host-custom-binary-cache-with-s3.md | 267 ++++++++++++++++++ .../host-custom-binary-cache-with-s3.md | 243 ++++++++++++++++ 3 files changed, 518 insertions(+) create mode 100644 docs/best-practices/host-custom-binary-cache-with-s3.md create mode 100644 docs/zh/best-practices/host-custom-binary-cache-with-s3.md diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index b9240bb..d91c424 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -261,6 +261,10 @@ function themeConfigEnglish() { text: "Debugging Derivations and Nix Expressions", link: "/best-practices/debugging.md", }, + { + text: "Host Custom Binary Cache with S3", + link: "/best-practices/host-custom-binary-cache-with-s3.md", + }, ], }, @@ -463,6 +467,10 @@ function themeConfigChinese() { text: "调试 Nix 软件包与 Nix 表达式", link: "/zh/best-practices/debugging.md", }, + { + text: "使用 S3 托管自定义二进制缓存", + link: "/zh/best-practices/host-custom-binary-cache-with-s3.md", + }, ], }, { diff --git a/docs/best-practices/host-custom-binary-cache-with-s3.md b/docs/best-practices/host-custom-binary-cache-with-s3.md new file mode 100644 index 0000000..6fa3f74 --- /dev/null +++ b/docs/best-practices/host-custom-binary-cache-with-s3.md @@ -0,0 +1,267 @@ +# Host Custom Binary Cache with S3 {#host-custom-binary-cache-with-s3} + +## TL;DR + +A guide on how to set up your own S3 nix binary cache using MinIO S3 server. + +## How Software Stored in Nix? {#how-software-stored-in-nix} + +Multiple versions of the same software package can be installed on a system, making it +possible to satisfy various dependency chains at a time. This enables the installation of +multiple packages that depend on the same third package, but on different versions of it. +To achieve this, all packages are installed in the global nix store under `/nix/store/` +and are then symlinked to the respective locations. To verify a package's uniqueness, its +whole directory is hashed, and the hash put into the name of the package's main folder. +Every software built from the same Nix expression that uses the same dependency software +versions results in the same hash, no matter what system it was built on. If any of the +dependency software versions were changed, this will result in a new hash for the final +package. + +Using symlinks to install a package and link all the right dependencies to it also enables +atomic updating. To make this clearer, let's think of an example where software X is +installed in an older version and should be updated. Software X is installed in its very +own directory in the global nix store and symlinked to the right directory, let's say +`/usr/local/bin/`. When the update is triggered, the new version of X is installed into +the global nix store without interfering with its older version. Once the installation +with all its dependencies in the nix store is completed, the final step is to change the +symlink to `/usr/local/bin/`. Since creating a new symlink that overwrites the old one is +an atomic operation in Unix, it is impossible for this operation to fail and leave the +package in a corrupted state. The only possible problem would be that it fails before or +after the symlink creation. Either way, the result would be that we either have the old +version of X or the newly installed version, but nothing in between. + +Quoted from the original work of +https://medium.com/earlybyte/the-s3-nix-cache-manual-e320da6b1a9b + +## Nix Binary Caches {#nix-binary-caches} + +No matter how great every aspect of Nix sounds, its design has also a major drawback, +which is that every package build triggers the build process for the whole dependency +chain from scratch. This can take quite a while, since even compilers such as gcc or ghc +must be built in advance. Such build processes can eat up a remarkable amount of memory, +introducing an additional hurdle if one wants to use it on restricted platforms such as +the Raspberry Pi. + +To overcome the drawback of always building everything from scratch and the chance of +losing access to prebuilt versions of packages, it is also possible to build your Nix +binary cache using an S3 server such as MinIO (https://min.io/). + +The process of setting up your cache, populating it with binaries and use the binaries +from your newly built cache will be described step by step. For this manual, I assume that +you already have nix in place, and that you already have a running MinIO server somewhere +in your environment. If not, you may check out the +[official deployment guide](https://min.io/docs/minio/linux/operations/installation.html) +from MinIO. You'll need to ensure that MinIO is accessible via `HTTPS` using a trusted +certificate. Let's Encrypt will be helpful here. + +In this post, let's explore how we can self-host an S3-compatible server, MinIO, as a +binary cache store. + +Quoted from the original work of +https://medium.com/earlybyte/the-s3-nix-cache-manual-e320da6b1a9b + +## How To Use S3 as a Binary Cache Server {#how-to-use-s3-as-a-binary-cache-server} + +### Prerequisites {#prerequisites} + +- Set up MinIO somewhere in your environment. +- Hold a valid SSL certificates either public or private. For demonstration purpose, we + will use `minio.homelab.local` (private certificate) in the steps mentioned in this + tutorial. If you plan to use a private certificate, you MUST resolve the DNS challenges + on your own. Hence, it is recommended to use a public certificate. +- Install `minio-client` in your environment. + +### Generate Password {#generate-password} + +```bash +nix run nixpkgs#pwgen -- -c -n -y -s -B 32 1 +# oenu1Yuch3rohz2ahveid0koo4giecho +``` + +### Set Up MinIO Client {#set-up-minio-client} + +Install the MinIO command-line client `mc`. + +```nix +{ pkgs, ... }: + +{ + environment.systemPackages = with pkgs; [ + minio-client # A replacement for ls, cp, mkdir, diff, and rsync commands for filesystems and object storage + ]; +} +``` + +Create or edit `~/.mc/config.json`. + +```json +{ + "version": "10", + "aliases": { + "s3": { + "url": "https://s3.homelab.local", + "accessKey": "minio", + "secretKey": "oenu1Yuch3rohz2ahveid0koo4giecho", + "api": "s3v4", + "path": "auto" + } + } +} +``` + +On the machine we use to build the nix packages, we need the S3 credentials in places. Set +up a file `~/.aws/credentials` and populate it with the credentials of our nixbuilder +user. Replace `` with the password generated by the `pwgen` command. + +```bash +[default] +aws_access_key_id=nixbuilder +aws_secret_access_key= +``` + +### Setup S3 Bucket as Binary Cache {#setup-s3-bucket-as-binary-cache} + +Create the `nix-cache` bucket. + +```bash +mc mb s3/nix-cache +``` + +Create the `nixbuilder` MinIO user and assign a password. + +```bash +mc admin user add s3 nixbuilder +``` + +Create a file called `nix-cache-write.json` in your current working directory with the +following contents: + +```json +{ + "Id": "AuthenticatedWrite", + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AuthenticatedWrite", + "Action": [ + "s3:AbortMultipartUpload", + "s3:GetBucketLocation", + "s3:GetObject", + "s3:ListBucket", + "s3:ListBucketMultipartUploads", + "s3:ListMultipartUploadParts", + "s3:PutObject" + ], + "Effect": "Allow", + "Resource": ["arn:aws:s3:::nix-cache", "arn:aws:s3:::nix-cache/*"], + "Principal": "nixbuilder" + } + ] +} +``` + +Create a policy with `nix-cache-write.json` that allows `nixbuilder` to upload files to +the cache. + +```bash +mc admin policy add s3 nix-cache-write nix-cache-write.json +``` + +Associate the policy that we created above with the `nixbuilder` user. + +```bash +mc admin policy set s3 nix-cache-write user=nixbuilder +``` + +Allow anonymous users to download files without authenticating. + +```bash +mc anonymous set download s3/nix-cache +``` + +Create a file called `nix-cache-info` in your working directory. This file tells Nix that +the bucket is indeed a binary cache. + +```bash +cat > nix-cache-info < ~/.config/nix/secret.key +nix key convert-secret-to-public < ~/.config/nix/secret.key > ~/.config/nix/public.key +cat ~/.config/nix/public.key +# s3.homelab.local-1:m0J/oDlLEuG6ezc6MzmpLCN2MYjssO3NMIlr9JdxkTs= +``` + +### Activate Binary Cache with Flake {#activate-binary-cache-with-flake} + +Put the following lines in `configuration.nix` or any of your custom NixOS module: + +```nix +{ + nix = { + settings = { + # Substituters will be appended to the default substituters when fetching packages. + extra-substituters = [ + "https://s3.homelab.local/nix-cache/" + ]; + extra-trusted-public-keys = [ + "s3.homelab.local-1:m0J/oDlLEuG6ezc6MzmpLCN2MYjssO3NMIlr9JdxkTs=" + ]; + }; + }; +} +``` + +Rebuild the system. + +```bash +sudo nixos-rebuild switch --upgrade --flake .# +``` + +### Push Paths to the Store {#push-paths-to-the-store} + +Sign some paths in the local store. + +```bash +nix store sign --recursive --key-file ~/.config/nix/secret.key /run/current-system +``` + +Copy those paths to the cache. + +```bash +nix copy --to 's3://nix-cache?profile=nixbuilder&endpoint=s3.homelab.local' /run/current-system +``` + +### Add Automatic Object Expiration Policy {#add-automatic-object-expiration-policy} + +```bash +mc ilm rule add s3/nix-cache --expire-days "DAYS" +# Example: mc ilm rule add s3/nix-cache --expire-days "7" +``` + +### References {#references} + +Here are some of the sources that I used in making this document: + +- [Blog post by Jeff on Nix binary caches](https://jcollie.github.io/nixos/2022/04/27/nixos-binary-cache-2022.html) +- [Binary cache in the NixOS wiki](https://nixos.wiki/wiki/Binary_Cache) +- [Serving a Nox store via S3 in the NixOS manual](https://nixos.org/manual/nix/stable/package-management/s3-substituter.html) +- [Serving a Nix store via HTTP in the NixOS manual](https://nixos.org/manual/nix/stable/package-management/binary-cache-substituter.html) diff --git a/docs/zh/best-practices/host-custom-binary-cache-with-s3.md b/docs/zh/best-practices/host-custom-binary-cache-with-s3.md new file mode 100644 index 0000000..ec91b43 --- /dev/null +++ b/docs/zh/best-practices/host-custom-binary-cache-with-s3.md @@ -0,0 +1,243 @@ +# 使用 S3 自定义二进制缓存托管 + +## 简介 + +一个关于如何使用 MinIO S3 服务器设置自己的 S3 Nix 二进制缓存的指南。 + +## Nix 中的软件如何存储？ + +可以在系统上安装同一软件包的多个版本，从而能够同时满足各种依赖链。这使得可以安装多个依赖于 +同一个第三方软件包的软件包，但使用不同版本的它。为了实现这一点，所有软件包都安装在全局的 +Nix 存储中，路径为 `/nix/store/`，然后通过符号链接到相应的位置。为了验证软件包的唯一性，其 +整个目录被哈希，并将哈希放入软件包的主文件夹的名称中。从使用相同依赖软件版本构建的相同 Nix +表达式构建的每个软件包都会产生相同的哈希，无论它是在什么系统上构建的。如果任何依赖软件的版 +本发生更改，这将导致最终软件包的新哈希。 + +使用符号链接来“安装”软件包并将所有正确的依赖项链接到它还使得原子更新成为可能。为了使这一点 +更清楚，让我们来考虑一个例子，其中软件 X 安装在旧版本中并且应该进行更新。软件 X 安装在全局 +Nix 存储中的自己的目录中，并符号链接到正确的目录，比如 `/usr/local/bin/`。当触发更新时，X +的新版本会安装到全局 Nix 存储中，而不会干扰其旧版本。一旦安装完成，包括其所有依赖项在内的 +最终软件包在 Nix 存储中，最后一步是将符号链接更改为 `/usr/local/bin/`。由于在 Unix 中创建 +新的符号链接来覆盖旧的符号链接是一个原子操作，因此这个操作不可能失败并使软件包处于损坏状 +态。唯一可能的问题是在符号链接创建之前或之后失败。无论哪种方式，结果都是我们要么有旧版本的 +X，要么有新安装的版本，但中间没有任何东西。 + +引用自原作：https://medium.com/earlybyte/the-s3-nix-cache-manual-e320da6b1a9b + +## Nix 二进制缓存 + +无论 Nix 的每个方面听起来多么棒，它的设计也有一个主要缺点，那就是每次构建软件包都会触发整 +个依赖链的构建过程。这可能需要相当长的时间，因为甚至像 gcc 或 ghc 这样的编译器都必须提前构 +建。这样的构建过程可能会消耗大量内存，在受限平台（如树莓派）上使用它时会引入额外的障碍。 + +为了克服总是从头开始构建一切以及丢失对软件包预构建版本的访问权限的缺点，还可以使用 S3 服务 +器（如 MinIO）构建自己的 Nix 二进制缓存。 + +设置您的缓存，填充它的二进制文件并使用您新构建的缓存中的二进制文件的过程将被逐步描述。对于 +本手册，我假设您已经有了 Nix，并且在您的环境中已经运行了 MinIO 服务器。如果没有，您可以查 +看 MinIO 的[官方部署指南](https://min.io/docs/minio/linux/operations/installation.html)。 +您需要确保通过信任的证书以 `HTTPS` 方式访问 MinIO。在这里，Let's Encrypt 将非常有用。 + +在本文中，让我们探讨如何自托管一个 S3 兼容服务器 MinIO 作为二进制缓存存储。 + +引用自原作：https://medium.com/earlybyte/the-s3-nix-cache-manual-e320da6b1a9b + +## 如何将 S3 用作二进制缓存服务器 + +### 先决条件 + +- 在您的环境中设置 MinIO。 +- 拥有有效的 SSL 证书，可以是公共证书也可以是私有证书。在本教程中，我们将使用 + `minio.homelab.local`（私有证书）的示例步骤。如果您计划使用私有证书，您必须自己解决 DNS + 挑战。因此，建议使用公共证书。 +- 在您的环境中安装 `minio-client`。 + +### 生成密码 + +```bash +nix run nixpkgs#pwgen -- -c -n -y -s -B 32 1 +# oenu1Yuch3rohz2ahveid0koo4giecho +``` + +### 设置 MinIO 客户端 + +安装 MinIO 命令行客户端 `mc`。 + +```nix +{ pkgs, ... }: + +{ + environment.systemPackages = with pkgs; [ + minio-client # 用于文件系统和对象存储的 ls、cp、mkdir、diff 和 rsync 命令的替代品 + ]; +} +``` + +创建或编辑 `~/.mc/config.json`。 + +```json +{ + "version": "10", + "aliases": { + "s3": { + "url": "https://s3.homelab.local", + "accessKey": "minio", + "secretKey": "oenu1Yuch3rohz2ahveid0koo4giecho", + "api": "s3v4", + "path": "auto" + } + } +} +``` + +在我们用来构建 Nix 包的机器上，我们需要在一些地方使用 S3 凭据。设置一个文件 +`~/.aws/credentials` 并用我们的 nixbuilder 用户的凭据填充它。用 `pwgen` 命令生成的密码替换 +``。 + +```plaintext +[default] +aws_access_key_id=nixbuilder +aws_secret_access_key= +``` + +### 设置 S3 存储桶作为二进制缓存 + +创建 `nix-cache` 存储桶。 + +```bash +mc mb s3/nix-cache +``` + +创建 `nixbuilder` MinIO 用户并分配密码。 + +```bash +mc admin user add s3 nixbuilder +``` + +在当前工作目录中创建名为 `nix-cache-write.json` 的文件，并具有以下内容： + +```json +{ + "Id": "AuthenticatedWrite", + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AuthenticatedWrite", + "Action": [ + "s3:AbortMultipartUpload", + "s3:GetBucketLocation", + "s3:GetObject", + "s3:ListBucket", + "s3:ListBucketMultipartUploads", + "s3:ListMultipartUploadParts", + "s3:PutObject" + ], + "Effect": "Allow", + "Resource": ["arn:aws:s3:::nix-cache", "arn:aws:s3:::nix-cache/*"], + "Principal": "nixbuilder" + } + ] +} +``` + +使用 `nix-cache-write.json` 创建允许 `nixbuilder` 上载文件到缓存的策略。 + +```bash +mc admin policy add s3 nix-cache-write nix-cache-write.json +``` + +将我们上面创建的策略与 `nixbuilder` 用户关联。 + +```bash +mc admin policy set s3 nix-cache-write user=nixbuilder +``` + +允许匿名用户在不进行身份验证的情况下下载文件。 + +```bash +mc anonymous set download s3/nix-cache +``` + +在工作目录中创建名为 `nix-cache-info` 的文件。此文件告诉 Nix 桶确实是一个二进制缓存。 + +```bash +cat > nix-cache-info < ~/.config/nix/secret.key +nix key convert-secret-to-public < ~/.config/nix/secret.key > ~/.config/nix/public.key +cat ~/.config/nix/public.key +# s3.homelab.local-1:m0J/oDlLEuG6ezc6MzmpLCN2MYjssO3NMIlr9JdxkTs= +``` + +### 使用 Flake 激活二进制缓存 + +将以下内容放入 `configuration.nix` 或您的任何自定义 NixOS 模块中： + +```nix +{ + nix = { + settings = { + # 在获取软件包时，替代器将被附加到默认的替代器。 + extra-substituters = [ + "https://s3.homelab.local/nix-cache/" + ]; + extra-trusted-public-keys = [ + "s3.homelab.local-1:m0J/oDlLEuG6ezc6MzmpLCN2MYjssO3NMIlr9JdxkTs=" + ]; + }; + }; +} +``` + +重新构建系统。 + +```bash +sudo nixos-rebuild switch --upgrade --flake .# +``` + +### 推送路径到存储 + +对本地存储中的一些路径进行签名。 + +```bash +nix store sign --recursive --key-file ~/.config/nix/secret.key /run/current-system +``` + +将这些路径复制到缓存。 + +```bash +nix copy --to 's3://nix-cache?profile=nixbuilder&endpoint=s3.homelab.local' /run/current-system +``` + +### 添加自动对象到期策略 + +```bash +mc ilm rule add s3/nix-cache --expire-days "DAYS" +# 例如：mc ilm rule add s3/nix-cache --expire-days "7" +``` + +### 参考资料 + +以下是我在编写本文档时使用的一些来源： + +- [Jeff 的博客文章：Nix 二进制缓存](https://jcollie.github.io/nixos/2022/04/27/nixos-binary-cache-2022.html) +- [NixOS wiki 上的二进制缓存](https://nixos.wiki/wiki/Binary_Cache) +- [NixOS 手册中关于通过 S3 提供 Nix 存储](https://nixos.org/manual/nix/stable/package-management/s3-substituter.html) +- [NixOS 手册中关于通过 HTTP 提供 Nix 存储](https://nixos.org/manual/nix/stable/package-management/binary-cache-substituter.html)