feat: multiple nixpkgs instances

This commit is contained in:
Ryan Yin 2023-08-15 15:31:34 +08:00
parent 82312b07cc
commit cadfceaae4
7 changed files with 241 additions and 1468 deletions

View File

@ -202,6 +202,7 @@ function themeConfigEnglish() {
{ text: "callPackage", link: "/nixpkgs/callpackage.md" },
{ text: "Overridding", link: "/nixpkgs/overriding.md" },
{ text: "Overlays", link: "/nixpkgs/overlays.md" },
{ text: "Mutiple Nixpkgs Instances", link: "/nixpkgs/multiple-nixpkgs.md" },
],
},
{
@ -388,6 +389,7 @@ function themeConfigChinese() {
{ text: "callPackage", link: "/zh/nixpkgs/callpackage.md" },
{ text: "Overridding", link: "/zh/nixpkgs/overriding.md" },
{ text: "Overlays", link: "/zh/nixpkgs/overlays.md" },
{ text: "多 Nixpkgs 实例", link: "/nixpkgs/multiple-nixpkgs.md" },
],
},
{

View File

@ -0,0 +1,108 @@
# The Ingenious Uses of Multiple nixpkgs Instances
In the section [Downgrade or Upgrade Packages](/nixos-with-flakes/downgrade-or-upgrade-packages.md), we have seen how to instantiate multiple distinct nixpkgs instances using the method `import nixpkgs {...}`. There are numerous applications for this technique, some common ones include:
1. Instantiate nixpkgs instances with different commit IDs to install various versions of software packages. This approach was used in the previous section [Downgrade or Upgrade Packages](/nixos-with-flakes/downgrade-or-upgrade-packages.md).
2. If you wish to utilize overlays without affecting the default nixpkgs instance, you can instantiate a new nixpkgs instance and apply overlays to it.
- The `nixpkgs.overlays = [...];` mentioned in the previous section on Overlays directly modifies the global nixpkgs instance. If your overlays make changes to fundamental packages, it might impact other modules. One downside is an increase in local compilation (due to cache invalidation), and there might also be functionality issues with the affected packages.
3. In cross-system architecture compilation, you can instantiate multiple nixpkgs instances to selectively use QEMU simulation for compilation and cross-compilation in different locations, or to add various GCC compilation parameters.
In conclusion, instantiating multiple nixpkgs instances is highly advantageous.
## Instantiating `nixpkgs`
Let's first understand how to instantiate a non-global nixpkgs instance. The most common syntax is as follows:
```nix
{
# a simple example
pkgs-xxx = import nixpkgs {
system = "x86_64-linux";
};
# a more complex example (cross-compiling)
pkgs-yyy = import nixpkgs {
localSystem = "x86_64-linux";
crossSystem = {
config = "riscv64-unknown-linux-gnu";
# https://nixos.wiki/wiki/Build_flags
# this option equals to adding `-march=rv64gc` to CFLAGS.
# CFLAGS will be used as the command line arguments for gcc/clang.
gcc.arch = "rv64gc";
# equivalent to `-mabi=lp64d` in CFLAGS.
gcc.abi = "lp64d";
};
overlays = [
(self: super: {
google-chrome = super.google-chrome.override {
commandLineArgs =
"--proxy-server='https=127.0.0.1:3128;http=127.0.0.1:3128'";
};
# ... other overlays
})
];
};
}
```
We have learned in our study of Nix syntax:
> The `import` expression takes a path to another Nix file as an argument and returns the execution result of that Nix file.
> If the argument to `import` is a folder path, it returns the execution result of the `default.nix` file within that folder.
`nixpkgs` is a flake with a `default.nix` file in its root directory. So, `import nixpkgs` essentially returns the execution result of [nixpkgs/default.nix](https://github.com/NixOS/nixpkgs/blob/master/default.nix).
Starting from this file, you can find that the implementation of `import nixpkgs` is in [pkgs/top-level/impure.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix), as excerpted below:
```nix
# ... skipping some lines
{ # We put legacy `system` into `localSystem` if `localSystem` was not passed.
# If neither is passed, assume we are building packages on the current
# (build, in GNU Autotools parlance) platform.
localSystem ? { system = args.system or builtins.currentSystem; }
# These are needed only because nix's `--arg` command-line logic doesn't work
# with unnamed parameters allowed by ...
, system ? localSystem.system
, crossSystem ? localSystem
, # Fallback: The contents of the configuration file found at $NIXPKGS_CONFIG or
# $HOME/.config/nixpkgs/config.nix.
config ? let
# ... skipping some lines
, # Overlays are used to extend Nixpkgs collection with additional
# collections of packages. These collection of packages are part of the
# fix-point made by Nixpkgs.
overlays ? let
# ... skipping some lines
, crossOverlays ? []
, ...
} @ args:
# If `localSystem` was explicitly passed, legacy `system` should
# not be passed, and vice versa.
assert args ? localSystem -> !(args ? system);
assert args ? system -> !(args ? localSystem);
import ./. (builtins.removeAttrs args [ "system" ] // {
inherit config overlays localSystem;
})
```
Therefore, `import nixpkgs {...}` effectively calls this function, and the subsequent attribute set becomes the arguments for this function.
## Considerations
When creating multiple nixpkgs instances, there are some details to keep in mind. Here are some common issues to consider:
1. According to the article [1000 instances of nixpkgs](https://discourse.nixos.org/t/1000-instances-of-nixpkgs/17347) shared by @fbewivpjsbsby, it's not a good practice to use `import` to customize `nixpkgs` in submodules or sub-flakes. This is because each `import` evaluates separately, creating a new nixpkgs instance each time. As the number of configurations increases, this can lead to longer build times and higher memory usage. Therefore, it's recommended to create all nixpkgs instances in the `flake.nix` file.
2. When mixing QEMU simulation and cross-compilation, care should be taken to avoid unnecessary duplication of package compilations.

View File

@ -0,0 +1,109 @@
# 多 nixpkgs 实例的妙用
我们在前面 [降级与升级软件包](/zh/nixos-with-flakes/downgrade-or-upgrade-packages.md) 一节中见过,怎么通过 `import nixpkgs {...}` 这样的方法实例化多个不同的 nixpkgs 实例。
这种方法有很多的用途,常见的有:
1. 通过实例化 commit id 不同的 nixpkgs 实例,用于安装不同版本的软件包。前面的 [降级与升级软件包](/zh/nixos-with-flakes/downgrade-or-upgrade-packages.md) 一节中就是这样使用的。
2. 如果希望使用 overlays但是又不想影响到默认的 nixpkgs 实例,可以通过实例化一个新的 nixpkgs 实例,然后在这个实例上使用 overlays。
- 上一节 Overlays 中提到的 `nixpkgs.overlays = [...];` 是直接修改全局的 nixpkgs 实例,如果你的 overlays 改了比较底层的包,可能会影响到其他模块。坏处之一是会导致大量的本地编译(因为二进制缓存失效了),二是被影响的包功能可能也会出问题。
3. 在跨系统架构的编译中,你可以通过实例化多个 nixpkgs 实例来在不同的地方分别选用 QEMU 模拟编译与交叉编译,或者添加不同的 gcc 编译参数。
总之,实例化多个 nixpkgs 实例是非常有用的。
## `nixpkgs` 的实例化
先看看如何实例化一个非全局的 nixpkgs 实例,最常见的语法是:
```nix
{
# a simple example
pkgs-xxx = import nixpkgs {
system = "x86_64-linux";
};
# a more complex example(cross-compiling)
pkgs-yyy = import nixpkgs {
localSystem = "x86_64-linux";
crossSystem = {
config = "riscv64-unknown-linux-gnu";
# https://nixos.wiki/wiki/Build_flags
# this option equals to add `-march=rv64gc` into CFLAGS.
# CFLAGS will be used as the command line arguments for the gcc/clang.
gcc.arch = "rv64gc";
# the same as `-mabi=lp64d` in CFLAGS.
gcc.abi = "lp64d";
};
overlays = [
(self: super: {
google-chrome = super.google-chrome.override {
commandLineArgs =
"--proxy-server='https=127.0.0.1:3128;http=127.0.0.1:3128'";
};
# ... other overlays
})
];
};
}
```
我们学习 Nix 语法时就学过:
> `import` 表达式以其他 Nix 文件的路径作为参数,返回该 Nix 文件的执行结果。
> `import` 的参数如果为文件夹路径,那么会返回该文件夹下的 `default.nix` 文件的执行结果。
`nixpkgs` 是一个 Git 仓库,它的根目录下刚好有一个 `default.nix` 文件,那么答案就呼之欲出了:`import nixpkgs` 就是返回 [nixpkgs/default.nix](https://github.com/NixOS/nixpkgs/blob/master/default.nix) 文件的执行结果。
从这个文件开始探索,就能找到 `import nixpkgs` 的实现代码是 [pkgs/top-level/impure.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix),这里截取部分内容:
```nix
# ... skip some lines
{ # We put legacy `system` into `localSystem`, if `localSystem` was not passed.
# If neither is passed, assume we are building packages on the current
# (build, in GNU Autotools parlance) platform.
localSystem ? { system = args.system or builtins.currentSystem; }
# These are needed only because nix's `--arg` command-line logic doesn't work
# with unnamed parameters allowed by ...
, system ? localSystem.system
, crossSystem ? localSystem
, # Fallback: The contents of the configuration file found at $NIXPKGS_CONFIG or
# $HOME/.config/nixpkgs/config.nix.
config ? let
# ... skip some lines
, # Overlays are used to extend Nixpkgs collection with additional
# collections of packages. These collection of packages are part of the
# fix-point made by Nixpkgs.
overlays ? let
# ... skip some lines
, crossOverlays ? []
, ...
} @ args:
# If `localSystem` was explicitly passed, legacy `system` should
# not be passed, and vice-versa.
assert args ? localSystem -> !(args ? system);
assert args ? system -> !(args ? localSystem);
import ./. (builtins.removeAttrs args [ "system" ] // {
inherit config overlays localSystem;
})
``````
因此 `import nixpkgs {...}` 实际就是调用了上面这个函数,后面的 attribute set 就是这个参数的参数。
## 注意事项
在创建多 nixpkgs 实例的时候需要注意一些细节,这里列举一些常见的问题:
1. 根据 @fbewivpjsbsby 补充的文章 [1000 instances of nixpkgs](https://discourse.nixos.org/t/1000-instances-of-nixpkgs/17347),在子模块或者子 flakes 中用 `import` 来定制 `nixpkgs` 不是一个好的习惯,因为每次 `import` 都会重新求值并产生一个新的 nixpkgs 实例,在配置越来越多时会导致构建时间变长、内存占用变大。所以这里改为了在 `flake.nix` 中创建所有 nixpkgs 实例。
2. 在混合使用 QEMU 模拟编译与交叉编译时,搞得不好可能会导致许多包被重复编译多次,要注意避免这种情况。

View File

@ -25,10 +25,10 @@
nixpkgs.overlays = [
# overlayer1 - 参数名用 self 与 super表达继承关系
(self: super: {
google-chrome = super.google-chrome.override {
commandLineArgs =
"--proxy-server='https=127.0.0.1:3128;http=127.0.0.1:3128'";
};
google-chrome = super.google-chrome.override {
commandLineArgs =
"--proxy-server='https=127.0.0.1:3128;http=127.0.0.1:3128'";
};
})
# overlayer2 - 还可以使用 extend 来继承其他 overlay

View File

@ -33,7 +33,6 @@
# Set Puppeteer to not download Chrome, cause it doesn't work on NixOS
export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1
# Set Puppeteer to use Chromium from Nixpkgs
export PUPPETEER_EXECUTABLE_PATH=${pkgs.chromium.outPath}/bin/chromium
'';
};
});

View File

@ -1,8 +1,7 @@
{
"type": "module",
"devDependencies": {
"vitepress": "1.0.0-rc.4",
"vitepress-export-pdf": "1.0.0-beta.0"
"vitepress": "1.0.0-rc.4"
},
"scripts": {
"docs:dev": "vitepress dev docs",

File diff suppressed because it is too large Load Diff