Bundling ONNX Runtime in Rust with Nix, Docker and GitHub Actions

Context: The Stack

Before diving in, here are the components involved in this conflict:

• fastembed-rs: A Rust crate for embedding generation that wraps the ONNX Runtime. By default, it attempts to download the shared library at runtime.
• ONNX Runtime (ORT): The cross-platform inference engine from Microsoft. It’s a heavy native dependency (.so, .dll, .dylib) that needs to match the host architecture.
• Nix Sandboxing: Nix builds run in an isolated environment with no network access, preventing the “magic download” strategy of many modern crates.

Why I’m even writing this

I like shipping stuff that can be run in several ways: Nix flake, build from source, Docker, and release binaries. That’s what I’ve been doing for Minne. Up until v0.2.6 this was fine.

Then I enabled reranking by pulling in fastembed-rs, which in turn uses ort. Locally it worked great: the crate downloads what it needs. But my deployment methods failed: Nix (sandbox, no network), Docker and through GitHub Actions (Windows in particular) things broke.

So this post is me writing down the approach I ended up with. It’s centered on my case, but the idea in general is to stop relying on crate download magic, and supply dependencies myself.

The Solution Architecture: Explicit Dependency Injection

Before diving into Nix, Docker and CI, here’s the model I ended up with:

1. Turn off auto-download in the Rust crate via features. no build-time network calls.
2. Put ONNX Runtime somewhere predictable for each environment.
3. Teach the application to look for it in a predictable way (ORT_DYLIB_PATH + a small fallback).
4. Repeat the same in Nix, Docker, GitHub Actions, and release archives.

That’s it. Everything below is just doing this four times.

Rust Configuration: Disabling fastembed-rs Auto-Download

This was the dependency line I used:

fastembed = { version = "5.2.0", default-features = false, features = ["hf-hub-native-tls", "ort-load-dynamic"] }

The important parts:

• default-features = false so it doesn’t pull in download stuff I don’t control.
• ort-load-dynamic so it uses dlopen() and respects ORT_DYLIB_PATH.

Then I made the app look for the lib like this:

use std::{env, path::PathBuf};

fn ort_lib_dir() -> PathBuf {
    if let Ok(p) = env::var("ORT_DYLIB_PATH") {
        return PathBuf::from(p);
    }

    // fallback: ./lib next to binary
    let exe = env::current_exe().unwrap_or_else(|_| PathBuf::from("."));
    exe.parent()
        .map(|p| p.join("lib"))
        .unwrap_or_else(|| PathBuf::from("./lib"))
}

So every deployment method only has to do one of these:

• set ORT_DYLIB_PATH to the actual file
• or package libonnxruntime.* in ./lib

I think this is the interesting reusable bit.

NixOS Sandbox: Linking Native Libraries

Nix builds in a sandbox and only sees declared inputs. Quoting the Nix manual: when sandbox = true, builds only see the Nix store, the build dir and what you bind in. Network is isolated for normal derivations.

So a crate that wants to download ONNX Runtime during build or run is not going to work inside that sandbox. That forced me to make ONNX Runtime a real input.

My Nix flake for Minne

This is the shape I used (shortened to the relevant bit):

{
  description = "Minne application flake";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    crane.url = "github:ipetkov/crane";
  };

  outputs = { self, nixpkgs, flake-utils, crane }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        lib = pkgs.lib;
        craneLib = crane.mkLib pkgs;
        libExt = if pkgs.stdenv.isDarwin then "dylib" else "so";
        minne-pkg = craneLib.buildPackage {
          src = lib.cleanSourceWith {
            src = ./.;
            filter = let
              extraPaths = [
                (toString ./common/migrations)
                (toString ./common/schemas)
                (toString ./html-router/templates)
                (toString ./html-router/assets)
              ];
            in
              path: type:
                craneLib.filterCargoSources path type
                || lib.any (x: lib.hasPrefix x (toString path)) extraPaths;
          };

          pname = "minne";
          version = "0.2.6";
          doCheck = false;

          nativeBuildInputs = [ pkgs.pkg-config pkgs.rustfmt pkgs.makeWrapper ];
          buildInputs = [ pkgs.openssl pkgs.chromium pkgs.onnxruntime ];

          postInstall = ''
            wrapProgram $out/bin/main \
              --set CHROME ${pkgs.chromium}/bin/chromium \
              --set ORT_DYLIB_PATH ${pkgs.onnxruntime}/lib/libonnxruntime.${libExt}
            for b in worker server; do
              if [ -x "$out/bin/$b" ]; then
                wrapProgram $out/bin/$b \
                  --set CHROME ${pkgs.chromium}/bin/chromium \
                  --set ORT_DYLIB_PATH ${pkgs.onnxruntime}/lib/libonnxruntime.${libExt}
              fi
            done
          '';
        };
      in {
        packages = {
          minne-pkg = minne-pkg;
          default = minne-pkg;
        };
      });
}

Key points:

• I pulled in pkgs.onnxruntime as a normal build input.
• I wrapped the binaries to set ORT_DYLIB_PATH to that exact file.
• Chromium is also wrapped.

So the Nix solution is: “declare ONNX Runtime as an input, and hard-set the env var when installing.”

Docker Build: Baking in the ONNX Runtime

Docker is looser than Nix, so it could have worked with download on startup, but I don’t want another feature set and I prefer to align all deployment methods to the same pattern. So I downloaded the exact ONNX Runtime release during the build and set the env var in the image.

This is the Dockerfile I used:

# === Builder ===
FROM rust:1.86-bookworm AS builder
WORKDIR /usr/src/minne
RUN apt-get update && apt-get install -y --no-install-recommends \
    pkg-config clang cmake git && rm -rf /var/lib/apt/lists/*

# Cache deps
COPY Cargo.toml Cargo.lock ./
RUN mkdir -p api-router common composite-retrieval html-router ingestion-pipeline json-stream-parser main worker
COPY api-router/Cargo.toml ./api-router/
COPY common/Cargo.toml ./common/
COPY composite-retrieval/Cargo.toml ./composite-retrieval/
COPY html-router/Cargo.toml ./html-router/
COPY ingestion-pipeline/Cargo.toml ./ingestion-pipeline/
COPY json-stream-parser/Cargo.toml ./json-stream-parser/
COPY main/Cargo.toml ./main/
RUN cargo build --release --bin main --features ingestion-pipeline/docker || true

# Build
COPY . .
RUN cargo build --release --bin main --features ingestion-pipeline/docker

# === Runtime ===
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    chromium libnss3 libasound2 libgbm1 libxshmfence1 \
    ca-certificates fonts-dejavu fonts-noto-color-emoji \
    libgomp1 libstdc++6 curl \
  && rm -rf /var/lib/apt/lists/*

ARG ORT_VERSION=1.22.0
RUN mkdir -p /opt/onnxruntime && \
    curl -fsSL -o /tmp/ort.tgz \
      "https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-${ORT_VERSION}.tgz" && \
    tar -xzf /tmp/ort.tgz -C /opt/onnxruntime --strip-components=1 && rm /tmp/ort.tgz

ENV CHROME_BIN=/usr/bin/chromium \
    SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
    ORT_DYLIB_PATH=/opt/onnxruntime/lib/libonnxruntime.so

RUN useradd -m appuser
USER appuser
WORKDIR /home/appuser

COPY --from=builder /usr/src/minne/target/release/main /usr/local/bin/main
EXPOSE 3000
CMD ["main"]

So here the pattern is: download once at build time, put it under /opt/onnxruntime, set ORT_DYLIB_PATH. No runtime download.

GitHub Actions: Cross-Platform Bundling

I added a block in the GH workflow that creates a lib/ dir and downloads the matching ONNX Runtime artifact per OS, then lets cargo-dist package it, and also set the dist toml file to include the lib folder when building.

    - name: Prepare lib dir (Unix)
      if: runner.os != 'Windows'
      run: |
        mkdir -p lib
        rm -f lib/*

    - name: Prepare lib dir (Windows)
      if: runner.os == 'Windows'
      shell: pwsh
      run: |
        New-Item -ItemType Directory -Force -Path lib | Out-Null
        Get-ChildItem -Path lib -Force | Remove-Item -Force -Recurse -ErrorAction SilentlyContinue

    - name: Fetch ONNX Runtime (Linux)
      if: runner.os == 'Linux'
      env:
        ORT_VER: 1.22.0
      run: |
        set -euo pipefail
        ARCH="$(uname -m)"
        case "$ARCH" in
          x86_64)  URL="https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VER}/onnxruntime-linux-x64-${ORT_VER}.tgz" ;;
          aarch64) URL="https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VER}/onnxruntime-linux-aarch64-${ORT_VER}.tgz" ;;
          *) echo "Unsupported arch $ARCH"; exit 1 ;;
        esac
        curl -fsSL -o ort.tgz "$URL"
        tar -xzf ort.tgz
        cp -v onnxruntime-*/lib/libonnxruntime.so* lib/
        [ -f lib/libonnxruntime.so ] || cp -v lib/libonnxruntime.so.* lib/libonnxruntime.so

and similarly for macOS and Windows (onnxruntime.dll).

That way the release artifact can contain:

• the binary
• the lib/ dir with the right ONNX Runtime file

and since the Rust side already looks at ORT_DYLIB_PATH and ./lib, it just works.

Closing

So the final shape is:

• the Rust app is dumb and only cares about ORT_DYLIB_PATH or ./lib
• Nix, Docker, CI, and release all supply ONNX Runtime in their own way
• I don’t rely on auto-download in anymore, and operate on a specific pinned version of onnxruntime

Related posts

• Refactoring a retrieval pipeline through evaluation - The reranking feature that prompted this ONNX Runtime work, and how I evaluated its (lack of) impact on retrieval quality.
• NixOS, an operating system for me? - More context on why I use NixOS and what the Nix sandbox means for development.