Portable Submodules
I have been working on Aux's burgeoning package set, nicknamed Tidepool. In developing this project I've had the opportunity to rethink many of the design decisions that other projects like Nixpkgs have made. One notable change is the use of the module system to declare packages rather than functions operating on plain values around a fixed point. These changes have wide-reaching implications for users which requires each to be thoughtfully considered. One such implication is overrides.
Overrides
In Nixpkgs a series of helpers allow for the customization of function arguments and build system modification. However, these are fairly blunt tools and many of the intricacies of package development require custom implementations in order to function effectively.
Firstly, each package created with callPackage is assigned a .override attribute which allows the user to call the package function again with different arguments. This feature is used today for overriding both dependencies and package configuration. Unfortunately, certain patterns like the use of pkgs as an argument in a package definition complicate this override behavior significantly. These overrides are also not discoverable unless a user is willing to read the package definition and understand what values are acceptable to provide. Often this is unclear.
Secondly, packages built with stdenv.mkDerivation come with a .overrideAttrs attribute. This attribute allows for the modification of configuration provided to stdenv.mkDerivation, letting the user modify the build steps, environment, and more. However, this tool, too, is a blunt one. Configuration provided via .overrideAttrs is not typically able to effect changes in the package's original definition. This means that any build steps, sources, or anything else that the package configures must either be used as-is or fully re-implemented. While a newer finalAttrs pattern exists to try and help somewhat, it is still far from commonplace in Nixpkgs.
Users often find themselves having to operate on a package using both of these disjointed helpers, producing strange code:
let
myModifiedPackage = (package.override {
dependencyA = pkgs.a;
someSetting = true;
}).overrideAttrs (old: {
buildInputs = old.buildInputs ++ [ pkgs.b ];
});
in
# ...
These functions should not be separate. In fact, there really should be a first-class, reactive solution for augmenting packages.
Submodules
A submodule is a fancy name for part of a module configuration which is evaluated separately before having its resulting value assigned. This gives developers the ability to construct more complex structures by having multiple layers of dynamic elements, many of which automatically configuring themselves. The resulting value, though, remains static. Once evaluated at a location this value must be used in full and the underlying definitions which produced the value cannot be reused elsewhere. Imagine, for a moment, that you wish to assign config.a to config.b, each of which uses the same submodule definition. What should the resulting value be for config.b? Take some time to digest this example and try to come up with the answer for what config.b.static and config.b.dynamic should be.
{ config, lib }:
let
type = lib.types.submodule ({ config }: {
options = {
static = lib.options.create {
type = lib.types.int;
default.value = 0;
};
dynamic = lib.options.create {
type = lib.types.int;
default.value = config.static + 1;
};
};
});
in
{
options = {
a = lib.options.create {
inherit type;
default.value = {};
};
b = lib.options.create {
inherit type;
default.value = {};
};
};
config = {
b = lib.modules.merge [
config.a
{
static = 99;
}
];
};
}
... ... ... ... ... ... ... ... ... ... ... ...
Here is the answer:
{
static = 99;
dynamic = 1;
}
Did you get that right? Even if you did, is that result truly desired? I don't believe so. These dynamic pieces of configuration need to propagate in places like this, but the existing tooling in Nixpkgs does not allow for it. Instead, we need a solution for bringing the whole submodule definition along rather than just the static output.
Portables
In order for configuration at different locations to propagate dynamic behavior, the definitions of those behaviors must also be provided. To do so, an additional attribute can be added to submodules: __modules__. This name is not special, rather, it relies on its naming convention to inform users that it is not intended to be manipulated directly. By passing the module definitions along themselves, we can then perform a new evaluation with any additional modifications. The ergonomics of such a feature are not pleasant when performed manually as each location must implement the functionality. Instead, a helper type can be used to allow for more intuitive use: lib.types.submodules.portable.
When operated on as typical submodules, a portable submodule will propagate any configuration as a new module definition. For example, setting config.a.static = 99 will also add a definition to the __modules__ attribute for this location. Here is the resulting value of config.a after performing the assignment:
{
__modules__ = [
{ config.static = 99; }
];
config = {
static = 99;
dynamic = 100;
};
}
This may solve the problem of dynamic values not evaluating, but we are now left with a new undesirable pattern. Users frequently need to override parts of configuration when performing assignment. This is performed with the convenient // operator to merge an old value with a new one. If used on a portable submodule, however, the changes will not propagate. Instead, a module definition needs to be added to __modules__ and a new execution of lib.modules.run is required. This can be worked around by separating assignment and augmentation with lib.modules.merge to leverage the portable submodule type's merging behavior, but the solution requires what feels like boilerplate. A first-class solution is preferred and we can look to lib.extend for ideas. A similar concept can be applied to portable submodules, allowing the user to call .extend on a configuration value and have any desired modifications applied in a way that respects the type's propagation functionality. Let's rewrite the original submodule example to use portables instead.
{ config, lib }:
let
type = lib.types.submodules.portable ({ config }: {
options = {
static = lib.options.create {
type = lib.types.int;
default.value = 0;
};
dynamic = lib.options.create {
type = lib.types.int;
default.value = config.static + 1;
};
};
});
in
{
options = {
a = lib.options.create {
inherit type;
default.value = {};
};
b = lib.options.create {
inherit type;
default.value = {};
};
};
config = {
b = config.a.extend {
static = 99;
};
};
}
As expected, the resulting output respects the original module definition. The value of config.b is the following (omitting __modules__).
{
static = 99;
dynamic = 100;
}
Implications
The simple examples given in this post hint at some larger benefits that this technique provides. One use case being exploited today is package management in Tidepool to allow for package definitions to be portable and easily modifiable. Other problems such as system services can take advantage of this behavior to allow for easy reuse of module definitions to retarget a user environment or ephemeral development instance whereas existing solutions are required to reimplement everything for each unique location.
Overall I am quite happy with the results of this experiment and the developer experience for portable submodules is rather pleasant.