Baseline (Hybrid + Reranker) </caption>
dataset</th>	mrr</th>	ndcg</th>	hit@5</th>	avg ms</th> </tr> </thead>
SQuAD v2.0</td>	0.757 </td>	0.769 </td>	0.855 </td>	1488 </td> </tr>
Natural Questions (dev)</td>	0.863 </td>	0.863 </td>	0.87 </td>	3850 </td> </tr> </tbody> </table> Config:</strong> - Strategy: Complex Hybrid (Vector + Lexical + Graph) - Reranker: On</p> </div> </aside> Config:</strong> - Strategy: Complex Hybrid (Vector + Lexical + Graph) - Reranker: On</p> </div> </aside> </div> </div>

Writing

Hermetic Enough: Nix Across the Software Lifecycle

2026-08-31T00:00:00+00:00

TLDR</summary>

I started using Nix for declarative developer environments. It has gradually spread across most of the software lifecycle in my projects.</li>
One flake now describes development shells, CI checks, release archives, Docker images, firmware builds, NixOS modules, and VM tests.</li>
The nicest example is a NixOS test that boots two Norna nodes, connects them through a virtual network, and checks that they lose and rediscover each other.</li>
Some parts are not hermetic. Prebuilt binaries, unsandboxed tests, and real hardware do not become pure because I put them in a Nix file.</li>
Nix still has a steep learning curve, but I find the cost easier to justify when it solves more than local development.</li> </ul> </details>
Intro
Minne</a>, Norna, which I’m in the process of open-sourcing, and a private CMS project. The details differ, but the structure has stuck around long enough that I think it is worth sharing.
Example, Norna</a> </h2>
Norna is a small distributed-state and policy project. It has a host daemon and firmware for an ESP32-C3. Two nodes should discover each other over UDP, exchange state, and notice when the other node disappears.
I could test the protocol in Rust and test the NixOS service separately. That still would not tell me whether two deployed services can actually find each other.
The Nix test boots two complete NixOS machines and gives them one virtual network:
```
 flowchart LR
 accTitle: Norna two-node test
 accDescr: Two Norna nodes exchange UDP traffic over a virtual network while the test controls both machines and checks their diagnostics.
 A[node A 192.168.100.1] <-->|UDP| B[node B 192.168.100.2]
 T[test] --> A
 T --> B
</pre>
The test waits until both services are running and each node can see the other through its diagnostics socket. Then it stops node B. Node A should eventually mark it as no longer live. The test starts B again and waits until both nodes rediscover each other.
This is what made the larger pattern click for me. The same flake provides the daemon package, NixOS module, machine configuration, virtual network, and the command that decides whether the whole thing works. I can run it again after changing any of those pieces.
I have neighboring tests for packet loss, reordering, fanout, and a ten-node shape. QEMU is not a real network, but this catches considerably more than “both crates compiled”.
Hermetic enough</a>
</h2>
A perfectly hermetic build has a strict meaning. My projects do not always reach it.
What I care about is knowing the important inputs and the places where outside state enters the build. If I download a prebuilt binary, the URL and hash should be visible. If a test needs an unsandboxed builder, that should be visible too. Unsupported targets should fail deliberately rather than work by accident on my machine.
Minne uses prebuilt dependencies for parts of its stack. On x86_64-linux</code>, for example, it fetches a SurrealDB archive by hash and patches the binary against libraries from the locked nixpkgs input. Other supported systems use SurrealDB from nixpkgs. The downloaded archive is pinned, but compatibility with the selected libraries is still something I need to test.
The important versions and hashes are gathered in one file:
{
 minneVersion = "1.0.5";
 ortVersion = "1.23.2";
 rustVersion = "1.91.1";
 rustManifestSha256 = "sha256-SDu4snEWjuZU475PERvu+iO50Mi39KVjqCeJeNvpguU=";
 # ...
 surrealdbVersion = "2.6.5";
 # Platform-specific hashes follow.
}
</code></pre>
Native dependencies are still annoying, but I know where to start looking when one changes.
One set of inputs, several build graphs</a>
</h2>
I used to call this “one locked graph”. That is close enough for conversation, but not quite accurate.
flake.lock</code> pins the flake inputs. Cargo.lock</code>, fixed-output hashes, and other project files pin more of the application. Nix then evaluates different derivation graphs depending on the host, target, features, and requested output. A Windows archive and ESP32-C3 firmware obviously do not share one identical closure.
The relationship looks more like this:
 flowchart LR
 accTitle: Project outputs derived from locked inputs
 accDescr: Locked project inputs feed development shells, CI checks, packages, release archives, a Docker image, NixOS modules, VM tests, and firmware checks.
 lock[locked inputs flake.lock + Cargo.lock + hashes] --> dev[dev shell]
 lock --> ci[CI checks]
 lock --> pkg[packages]
 pkg --> release[release archives]
 pkg --> image[Docker image]
 pkg --> module[NixOS module]
 module --> vm[VM tests]
 lock --> firmware[firmware]
</pre>
Norna’s nix/</code> directory shows how I currently arrange this:
nix/
├── build
│ ├── build-std-vendor-crates.nix
│ ├── firmware-esp32c3.nix
│ ├── firmware-qemu-esp32c3.nix
│ ├── no-std-kernel.nix
│ ├── release.nix
│ ├── toolchain.nix
│ └── write-cargo-checksum.py
├── modules
│ ├── agent.nix
│ ├── checks.nix
│ ├── context.nix
│ ├── dev-shell.nix
│ ├── formatter.nix
│ └── packages.nix
└── tests
 ├── embedded-smoke.nix
 ├── network-chaos.nix
 ├── network-fanout.nix
 ├── network-scale.nix
 ├── network-sync.nix
 ├── smoke.nix
 └── tui-smoke.nix
</code></pre>
build/</code> contains package and cross-target recipes. modules/</code> exposes them as packages, checks, apps, and development shells. tests/</code> contains the NixOS tests that boot whole systems.
I initially put much more directly in flake.nix</code>. Eventually it became a long mix of toolchains, packages, and scripts. This split is easier for me to navigate, and there is very little magic to it.
From development shell to release artifact</a>
</h2>
For Minne, the development shell selects the Rust toolchain and native build dependencies. nix run .#dev</code> starts a process-compose stack with SurrealDB, MinIO, Tailwind, and the setup around them. nix run .#dev-down</code> stops it again.
Native libraries were what pushed this beyond a developer-shell project. When I added reranking to Minne, ONNX Runtime worked locally and then failed in Nix, Docker, and release builds in slightly different ways. I wrote a separate post about that adventure</a>.
What I ended up doing was fairly simple: disable the crate’s automatic download, make ONNX Runtime an explicit input, and supply ORT_DYLIB_PATH</code> from the package. The application accepts a path. Nix, Docker, or the release archive decides where that path points.
Embedded targets</a>
</h3>
Norna’s daemon uses a stable host toolchain. The firmware has its own ESP32-C3 toolchain and target. The portable core also needs to compile with no_std</code> and alloc</code> for riscv32imc-unknown-none-elf</code>.
The check runs:
cargo build --offline \
 --target riscv32imc-unknown-none-elf \
 -Z build-std=core,alloc \
 -p norna-core --no-default-features
</code></pre>
Getting this offline was decidedly not elegant. Cargo’s normal vendoring did not include everything required by -Z build-std</code>. The Nix code therefore contains a long list of standard-library crates and hashes, while a small Python script creates the Cargo checksum files. The firmware derivation also writes reduced Cargo manifests for its isolated build.
This creates a shadow dependency description which can drift from the workspace. It currently proves that the portable core and firmware build in the intended environment, but Rust upgrades will require care. I would only copy this machinery for an embedded target that actually needs it.
CI and agents</a>
</h2>
CI can easily become another build system. In these projects it mostly installs Nix, restores a store cache where one is configured, and builds a named output.
Minne and the CMS run nix flake check</code> in CI. Norna’s normal GitHub workflow only builds checks.x86_64-linux.agent-fast</code>. That gate contains formatting, Clippy, workspace tests, and coverage. The firmware and NixOS VM checks exist as outputs, but they are not currently run by the ordinary pull-request workflow.
Minne’s release workflow needs the ONNX Runtime version, so it reads it from the flake:
nix eval .#lib.ortVersion --raw
</code></pre>
The flake also exposes a named check for agreement between that pin and the nixpkgs package:
ortVersion = pkgs.runCommand "ort-version-check" { } ''
 if [ "${pkgs.onnxruntime.version}" != "${ortVersion}" ]; then
 echo "pkgs.onnxruntime.version is ${pkgs.onnxruntime.version}, but flake pins ${ortVersion}" >&2
 exit 1
 fi
 touch $out
'';
</code></pre>
Agents use the same outputs. nix develop --command ...</code> gives them the pinned tools and native dependencies instead of inviting them to guess which compiler to install. It is not a security boundary. The development shell does not hide credentials, disable the network, or prevent writes to my home directory.
Norna makes the fast path explicit:
agentFastInputs = [
 self'.checks.treefmt
 workspaceClippy
 workspaceTests
 workspaceCoverage
];
</code></pre>
A full check set with cross toolchains and several QEMU machines is too slow for every edit, especially on a cold store. The quick path keeps the feedback loop useful. The heavier outputs still need to run somewhere regularly; merely naming a check does not verify it.
Testing deployment details</a>
</h2>
The Norna example at the start tests the network between two deployed services. The simpler VM tests are useful too.
Minne imports its actual NixOS module, starts SurrealDB and Minne, and checks the live and ready endpoints:
testScript = ''
 machine.wait_for_unit("surrealdb.service")
 machine.wait_for_unit("minne.service")
 machine.wait_for_open_port(3000)
 machine.succeed("curl -sf http://127.0.0.1:3000/api/v1/live")
 machine.succeed("curl -sf http://127.0.0.1:3000/api/v1/ready")
'';
</code></pre>
The CMS tests more of its surrounding system. One VM checks systemd hardening, reaches the application through nginx over TLS, runs the backup service, and validates the archive with gzip -t</code>. Another boots Garage beside the CMS, signs in through the admin interface, uploads an image, and compares the downloaded bytes with the original.
These tests use fake credentials and validate the NixOS deployment. They take time and work best with KVM acceleration, which is why I want the cheap evaluation and package checks as well.
What they give me is confidence in details below the Rust process: systemd settings, environment mapping, ports, proxies, files, and communication between services. Those are exactly the details I otherwise tend to discover after deployment.
The trade</a>
</h2>
This setup has real costs. Nix evaluation errors can be cryptic. Cross-compilation still requires understanding linkers and targets. The offline build-std</code> workaround is custom maintenance. VM closures are large, and cold caches hurt. A team adopting this structure would also have to learn enough Nix to debug it rather than depending on one person.
It is hermetic enough for me: explicit enough that I can find the compromises and test the important paths.
```



Refactoring a retrieval pipeline through evaluation
2025-11-10T00:00:00+00:00

  TLDR</span></summary>
  
I am the developer behind Minne</a></li>
Minne uses a combined retrieval strategy for the “slow search” consisting of vector comparison, full text search, exploration of a graph database</li>
Current design was created in 2024, inspired by psychology and a Zettelkasten style of knowledge mapping</li>
The post details the process of implementing a evaluation pattern against several established datasets to evaluate and steer continued development</li>
The end results showed improved performance with latency reduced from 1600ms to 36ms</li>
Reranking proved to add significant latency to no improvement</li>
</ul>

</details>
How did I actually get in this mess, and background</a>
</h2>
For the past year my passion project has been developing Minne</a>, which is a save-for-later and personal knowledge management system (PKMS). Since learning about the Zettelkasten</a> method, I became quite enamored with the idea of having a personal knowledge database. I’ve tried a couple of different tools, but none really to my liking. While I truly believe a lot of the benefits of a PKMS comes from the manual process of creating links between knowledge and documents, by means of creating neural pathways and working through knowledge, I’ve learned that I’m not that</em> enamored with PKMS. As a psychologist, I’m fascinated by how we offload memory to digital tools. So development began.</p>
When designing Minne in 2024 I envisioned having content as a base layer, and then extracted knowledge entities as an abstraction of it. And that this higher level plane of knowledge could work well for searching and working on. That then these knowledge entities are based in and connected to the underlying documents/content that it was derived from. I think it came out quite nifty, and to some respect it mimics a human learning pattern.</p>
  flowchart TB
subgraph Ingestion
A[Content] --> B[Chunks]
B --> C[Knowledge Entities]
end
subgraph Retrieval
C --> D[Graph Exploration]
C --> E[Vector + Text Search]
end
</pre>
We humans learn these higher level abstractions connected to different entities which we then can reference and explore if we need more detail. That these second layer knowledge entities exists as a compressed version and that some fidelity loss occurs but that’s fine. Can’t overload the context anyway, or we lose quality.</p>
Current retrieval pipeline</a>
</h3>
In the application we look through this knowledge entity database in two separate occurrences. For simplicity I call this slow search</em>. As of writing, this retrieval pipeline consists of vector comparison, full text search, and exploration of the graph database.
  flowchart LR
Q[Query/Embedding] --> V[Vector Search]
Q --> L[Lexical Search]
V --> M[Fuse Scores]
L --> M
M --> D[Expand Graph]
D --> E[Attach Chunks]
E --> F[Rerank]
F --> G[Assemble]
</pre></p>

When we ingest new content, we run slow search so the LLM that creates new knowledge entities can link them to existing ones and work with a context-aware view of the graph.</li>
When we chat with an LLM in the application, it gets the result of slow search as context, which lets it answer in a knowledge-aware way and attach references.</li>
</ul>
While I like</em> the current design, it’s not the fastest. We run several different database operations on quite a lot of data, using the knowledge entities as primary candidates. The project has started to see some real users, and I’d like to explore optimizing the retrieval pipeline. It should serve its purpose first, not just give me the pleasure of maintaining a complex retrieval pipeline.</p>

    
    
    What does this *nice* and complex retrieval pipeline actually do for me, and is it doing it well enough to justify the complexity?</figcaption>
    
</figure>
Getting started</a>
</h2>
So to proceed I opted to set up a evaluation framework, measuring both speed and retrieval accuracy. At this point in time I’m not evaluating the actual LLM performance, but rather the accuracy of the slow search itself. Which means answering the question, is the answer to a question inside the data retrieved from our knowledge system?</p>
Datasets</a>
</h3>
For datasets, I faced an interesting decision. What type of data should this “slow search” be optimized for. What are the users actually ingesting in the system? What am I ingesting? I’ve found I use it for an assortment of things, written notes, audio notes, links for reading later, some pdf documents. But creating a meaningful representative dataset would be hard. So even though it would have been preferred I opted to abandon creating my own dataset, and use established ones. I chose two:</p>

SQUAD v2, it acts as a simpler and lighter dataset. Shorter paragraphs.</li>
Natural Questions (dev). It’s a dataset on wikipedia articles. Some articles are really long and could prove to be challenging.</li>
</ul>
Initially, I stuck to these two. After tuning for these sets I tried adding hybrid search, which consequently lowered results. Which ran some bells about me not understanding something enough… So I eventually brought in some more tests in from the BEIR test suite. More on that those later.</p>
Evaluation harness</a>
</h3>
For my current design, each evaluation means running each article through the ingestion pipeline to create the knowledge entities and relationships. This is a somewhat costly operation. To test retrieval precision effectively, I needed distractors. So for each question, I opted to ingest nine incorrect contexts along with the correct one. This meant that to answer the first 200 questions, I had to ingest a total of 2000 articles.</p>
Considering the computational effort involved in converting the raw dataset to my knowledge design, I needed persistence. This allows me to decouple the ingestion cost from the retrieval experimentation. I can iterate on the retrieval pipeline against a static, pre-ingested set. If I need to change the ingestion logic, I can simply recreate the set. This setup also gives me the flexibility to modify parameters at runtime, like the number of snippets to pass, the total amount to retrieve from the initial search, or whether to enable re-ranking.</p>

  Infrastructure: The in-memory detour</span></summary>
  I initially opted to use an embedded in-memory database instance for the evaluations. While great for unit tests, not the best approach for this.</p>
Since the DB was ephemeral, I had to re-ingest and re-index the entire dataset (2000+ articles) for every single evaluation run. Creating HNSW and BM25 indexes is computationally expensive, and the embedded SurrealDB engine handled index creation in a single thread. This meant waiting hours just to run a quick retrieval test.</p>
The solution was obvious: move to a persistent SurrealDB instance. This decoupled the heavy indexing step from the retrieval tests, allowing me to iterate on the pipeline in seconds rather than hours.</p>

</details>
Initial findings</a>
</h3>
Initial runs on datasets showed the following results, which in my opinion are fairly respectable, except for latency. The majority of latency comes from the embedding and reranking, or so I thought! More detailed logging showed that for moderately sized datasets the majority of latency comes from the search and chunk loading. It worked fine for my smallish production dataset, but scaled really badly. Which makes it clear I need to do some heavy refactoring. So preliminary tests shows good hit@k1-5, but abhorrent latency.</p>

  SurrealDB: Pains</span></summary>
  When running the evals for the Natural Questions dataset (about 1000 wiki articles), another issue became clear. When creating HNSW indexes, memory usage in SurrealDB spiked greatly, to well over 100GB, and it was creating intermediate disk entities in excess of 20GB. It was clear something was wrong.</p>
In efforts to resolve this I tried:</p>

Separating embeddings to a separate table (negligible impact).</li>
Modifying HNSW parameters (helped, but not dramatically).</li>
Removing HNSW indexes from the initial migration and only applying them after</em> ingestion.</li>
</ul>
The last point turned out to be the key. The initial process causing the spike was: applying migrations containing a 1536 HNSW index -> removing the index -> ingesting data -> creating index.</p>
Update:</strong> I have since moved to a memory-backed SurrealDB instance running in Docker. Interestingly, RAM usage has become even better during ingestion with this setup compared to the rocksdb backed host. I’m not 100% sure why, but the previous behavior remains a bit of an annoyance/quirk.
Update 2</strong> Considering how well memory backed surrealdb works compared to rocksdb backed, at least when running via docker. It’s clear that running rocksdb backed via docker was a very bad idea. Have not explored yet if this is to be expected or a bug.</p>

</details>
Vector search</a>
</h2>
So, now we’ve got a decent enough evaluation pipeline. Let’s start by establishing a baseline.</p>
Experimental setup</a>
</h3>
For all experiments, unless otherwise noted, we are using:</p>

Embeddings</strong>: nomic-ai/nomic-embed-text-v1.5</code></li>
Tokenizer</strong>: bert-base-cased</code></li>
Reranker</strong>: jinaai/jina-reranker-v1-turbo-en</code></li>
</ul>
Reranking surprise</a>
</h3>
I initially ran all my tests with a reranker enabled, assuming it would boost performance. However, when I finally ran a test without</em> it, I was shocked.</p>

    
    
    Wait, whaat?</figcaption>
    
</figure>
The reranker was adding significant latency (~550ms per query) but actually lowered</em> performance on the Natural Questions dataset.</p>

Lesson Learned:</strong> Always verify your assumptions.</p>
</blockquote>
For the rest of this post, results are presented without reranking</strong>, unless otherwise specified.</p>
Ablations</a>
</h3>
Here are the results for different vector chunk sizes and strategies.</p>








  
  
  
  

  
    
      Baseline (Hybrid + Reranker)
    </caption>
    
      
        dataset</th>
        mrr</th>
        ndcg</th>
        hit@5</th>
        
        avg ms</th>
        
      </tr>
    </thead>
    

      
      
      
        
          SQuAD v2.0</td>

        
        
          0.757
          
        </td>

        
        
          0.769
          
        </td>

        
        
          0.855
          
        </td>

        
        
        
          1488
          
        </td>
        
      </tr>
      
      
      

        
          Natural Questions (dev)</td>

        
        
          0.863
          
        </td>

        
        
          0.863
          
        </td>

        
        
          0.87
          
        </td>

        
        
        
          3850
          
        </td>
        
      </tr>
      
    </tbody>
  </table>
  
  
    Config:</strong>
- Strategy: Complex Hybrid (Vector + Lexical + Graph)
- Reranker: On</p>
</div>
  </aside>
  

    Config:</strong>
- Strategy: Complex Hybrid (Vector + Lexical + Graph)
- Reranker: On</p>
</div>
  </aside>
  
</div>

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
</div>

Reranking surprise
Ablations

Ablations

Writing

Hermetic Enough: Nix Across the Software Lifecycle

Intro
Minne</a>, Norna, which I’m in the process of open-sourcing, and a private CMS project. The details differ, but the structure has stuck around long enough that I think it is worth sharing.</p>

Refactoring a retrieval pipeline through evaluation