Hermetic Enough: Nix Across the Software Lifecycle

Intro

Nix is a great tool for several things. I got started with declarative developer environments. It then became the operating system for my workstation, server, router, and VPS. Over the past year, I’ve moved more and more of the software lifecycle into Nix. This is an attempt to summarize what I’ve learned, what works for me, and perhaps convey why I get some amount of enjoyment from it :)

Three rather different projects have gradually converged on a similar Nix structure: Minne, Norna, which I’m in the process of open-sourcing, and a private CMS project. The details differ, but the structure has stuck around long enough that I think it is worth sharing.

Example, Norna

Norna is a small distributed-state and policy project. It has a host daemon and firmware for an ESP32-C3. Two nodes should discover each other over UDP, exchange state, and notice when the other node disappears.

I could test the protocol in Rust and test the NixOS service separately. That still would not tell me whether two deployed services can actually find each other.

The Nix test boots two complete NixOS machines and gives them one virtual network:

  flowchart LR
    accTitle: Norna two-node test
    accDescr: Two Norna nodes exchange UDP traffic over a virtual network while the test controls both machines and checks their diagnostics.
    A[node A<br/>192.168.100.1] <-->|UDP| B[node B<br/>192.168.100.2]
    T[test] --> A
    T --> B

The test waits until both services are running and each node can see the other through its diagnostics socket. Then it stops node B. Node A should eventually mark it as no longer live. The test starts B again and waits until both nodes rediscover each other.

This is what made the larger pattern click for me. The same flake provides the daemon package, NixOS module, machine configuration, virtual network, and the command that decides whether the whole thing works. I can run it again after changing any of those pieces.

I have neighboring tests for packet loss, reordering, fanout, and a ten-node shape. QEMU is not a real network, but this catches considerably more than “both crates compiled”.

Hermetic enough

A perfectly hermetic build has a strict meaning. My projects do not always reach it.

What I care about is knowing the important inputs and the places where outside state enters the build. If I download a prebuilt binary, the URL and hash should be visible. If a test needs an unsandboxed builder, that should be visible too. Unsupported targets should fail deliberately rather than work by accident on my machine.

Minne uses prebuilt dependencies for parts of its stack. On x86_64-linux, for example, it fetches a SurrealDB archive by hash and patches the binary against libraries from the locked nixpkgs input. Other supported systems use SurrealDB from nixpkgs. The downloaded archive is pinned, but compatibility with the selected libraries is still something I need to test.

The important versions and hashes are gathered in one file:

{
  minneVersion = "1.0.5";
  ortVersion = "1.23.2";
  rustVersion = "1.91.1";
  rustManifestSha256 = "sha256-SDu4snEWjuZU475PERvu+iO50Mi39KVjqCeJeNvpguU=";
  # ...
  surrealdbVersion = "2.6.5";
  # Platform-specific hashes follow.
}

Native dependencies are still annoying, but I know where to start looking when one changes.

One set of inputs, several build graphs

I used to call this “one locked graph”. That is close enough for conversation, but not quite accurate.

flake.lock pins the flake inputs. Cargo.lock, fixed-output hashes, and other project files pin more of the application. Nix then evaluates different derivation graphs depending on the host, target, features, and requested output. A Windows archive and ESP32-C3 firmware obviously do not share one identical closure.

The relationship looks more like this:

  flowchart LR
    accTitle: Project outputs derived from locked inputs
    accDescr: Locked project inputs feed development shells, CI checks, packages, release archives, a Docker image, NixOS modules, VM tests, and firmware checks.
    lock[locked inputs<br/>flake.lock + Cargo.lock + hashes] --> dev[dev shell]
    lock --> ci[CI checks]
    lock --> pkg[packages]
    pkg --> release[release archives]
    pkg --> image[Docker image]
    pkg --> module[NixOS module]
    module --> vm[VM tests]
    lock --> firmware[firmware]

Norna’s nix/ directory shows how I currently arrange this:

nix/
├── build
│   ├── build-std-vendor-crates.nix
│   ├── firmware-esp32c3.nix
│   ├── firmware-qemu-esp32c3.nix
│   ├── no-std-kernel.nix
│   ├── release.nix
│   ├── toolchain.nix
│   └── write-cargo-checksum.py
├── modules
│   ├── agent.nix
│   ├── checks.nix
│   ├── context.nix
│   ├── dev-shell.nix
│   ├── formatter.nix
│   └── packages.nix
└── tests
    ├── embedded-smoke.nix
    ├── network-chaos.nix
    ├── network-fanout.nix
    ├── network-scale.nix
    ├── network-sync.nix
    ├── smoke.nix
    └── tui-smoke.nix

build/ contains package and cross-target recipes. modules/ exposes them as packages, checks, apps, and development shells. tests/ contains the NixOS tests that boot whole systems.

I initially put much more directly in flake.nix. Eventually it became a long mix of toolchains, packages, and scripts. This split is easier for me to navigate, and there is very little magic to it.

From development shell to release artifact

For Minne, the development shell selects the Rust toolchain and native build dependencies. nix run .#dev starts a process-compose stack with SurrealDB, MinIO, Tailwind, and the setup around them. nix run .#dev-down stops it again.

Native libraries were what pushed this beyond a developer-shell project. When I added reranking to Minne, ONNX Runtime worked locally and then failed in Nix, Docker, and release builds in slightly different ways. I wrote a separate post about that adventure.

What I ended up doing was fairly simple: disable the crate’s automatic download, make ONNX Runtime an explicit input, and supply ORT_DYLIB_PATH from the package. The application accepts a path. Nix, Docker, or the release archive decides where that path points.

Embedded targets

Norna’s daemon uses a stable host toolchain. The firmware has its own ESP32-C3 toolchain and target. The portable core also needs to compile with no_std and alloc for riscv32imc-unknown-none-elf.

The check runs:

cargo build --offline \
  --target riscv32imc-unknown-none-elf \
  -Z build-std=core,alloc \
  -p norna-core --no-default-features

Getting this offline was decidedly not elegant. Cargo’s normal vendoring did not include everything required by -Z build-std. The Nix code therefore contains a long list of standard-library crates and hashes, while a small Python script creates the Cargo checksum files. The firmware derivation also writes reduced Cargo manifests for its isolated build.

This creates a shadow dependency description which can drift from the workspace. It currently proves that the portable core and firmware build in the intended environment, but Rust upgrades will require care. I would only copy this machinery for an embedded target that actually needs it.

CI and agents

CI can easily become another build system. In these projects it mostly installs Nix, restores a store cache where one is configured, and builds a named output.

Minne and the CMS run nix flake check in CI. Norna’s normal GitHub workflow only builds checks.x86_64-linux.agent-fast. That gate contains formatting, Clippy, workspace tests, and coverage. The firmware and NixOS VM checks exist as outputs, but they are not currently run by the ordinary pull-request workflow.

Minne’s release workflow needs the ONNX Runtime version, so it reads it from the flake:

nix eval .#lib.ortVersion --raw

The flake also exposes a named check for agreement between that pin and the nixpkgs package:

ortVersion = pkgs.runCommand "ort-version-check" { } ''
  if [ "${pkgs.onnxruntime.version}" != "${ortVersion}" ]; then
    echo "pkgs.onnxruntime.version is ${pkgs.onnxruntime.version}, but flake pins ${ortVersion}" >&2
    exit 1
  fi
  touch $out
'';

Agents use the same outputs. nix develop --command ... gives them the pinned tools and native dependencies instead of inviting them to guess which compiler to install. It is not a security boundary. The development shell does not hide credentials, disable the network, or prevent writes to my home directory.

Norna makes the fast path explicit:

agentFastInputs = [
  self'.checks.treefmt
  workspaceClippy
  workspaceTests
  workspaceCoverage
];

A full check set with cross toolchains and several QEMU machines is too slow for every edit, especially on a cold store. The quick path keeps the feedback loop useful. The heavier outputs still need to run somewhere regularly; merely naming a check does not verify it.

Testing deployment details

The Norna example at the start tests the network between two deployed services. The simpler VM tests are useful too.

Minne imports its actual NixOS module, starts SurrealDB and Minne, and checks the live and ready endpoints:

testScript = ''
  machine.wait_for_unit("surrealdb.service")
  machine.wait_for_unit("minne.service")
  machine.wait_for_open_port(3000)
  machine.succeed("curl -sf http://127.0.0.1:3000/api/v1/live")
  machine.succeed("curl -sf http://127.0.0.1:3000/api/v1/ready")
'';

The CMS tests more of its surrounding system. One VM checks systemd hardening, reaches the application through nginx over TLS, runs the backup service, and validates the archive with gzip -t. Another boots Garage beside the CMS, signs in through the admin interface, uploads an image, and compares the downloaded bytes with the original.

These tests use fake credentials and validate the NixOS deployment. They take time and work best with KVM acceleration, which is why I want the cheap evaluation and package checks as well.

What they give me is confidence in details below the Rust process: systemd settings, environment mapping, ports, proxies, files, and communication between services. Those are exactly the details I otherwise tend to discover after deployment.

The trade

This setup has real costs. Nix evaluation errors can be cryptic. Cross-compilation still requires understanding linkers and targets. The offline build-std workaround is custom maintenance. VM closures are large, and cold caches hurt. A team adopting this structure would also have to learn enough Nix to debug it rather than depending on one person.

It is hermetic enough for me: explicit enough that I can find the compromises and test the important paths.