How a Flaky NixOS Build Uncovered Four Layers of Broken
NixOS 25.11 container builds started failing in January 2026 and I couldn’t reproduce it. The stderr would cut off mid-error; I’d retry and it would succeed. That went on for months. When I finally dug in, the “flaky build” turned out to be a race condition, and fixing it peeled back three more layers of problems I didn’t know I had: a broken cache substituter setup (corrupt pubkeys plus missing firewall rules), a flat overlay architecture that couldn’t handle multiple NixOS versions, and eventually an OOM crash that took down a Proxmox host. Four tickets, three commits, six weeks.
What Happened
The symptom was simple enough: NixOS 25.11 container image builds failed unreproducibly. I’d run a build, it would fail, stderr would cut off mid-error, I’d retry, and it would succeed. No pattern. The original ticket, KB#8063: 25.11 builds weird, captured it in one line: “Can’t reproduce the failure and stderr cuts it off.”
That was January 21, 2026. I let it sit because it wasn’t blocking anything critical; the retries worked. But by May I needed 25.11 builds to be reliable, so I started digging.
The first thing I found was that create_image.sh, the script that builds container images, ran nix-channel --update and nixos-rebuild switch before the container finished booting. Specifically, before systemd reported running state and before nix-daemon was reachable. The commands hit a system that wasn’t ready for them, failed in unexpected ways, and the stderr truncation was a symptom of that failure, not a bug in the build output itself.
While investigating that race condition, I noticed something else: containers were taking 20 minutes to activate instead of 30 seconds. That’s a big red flag. It meant Nix was building packages from source instead of pulling them from the cache. I opened KB#8167: Fix nix cache pubkey propagation and substituter priority to track it.
Two bugs surfaced there. First, the nix_cache_pubkey default was an empty string, which rendered a corrupt key that Nix rejects. All 14 consumer stacks passed the pubkey correctly, but the default was wrong, so any stack that didn’t explicitly set it got a broken key. Second, the andrewcz-net and andrewcz-org env-level security groups were missing an outbound port 5000 rule for nix-serve. Containers could reach the nginx proxy on port 80 but not nix-serve on port 5000. Custom overlay packages like ghost-blog, consul, and grafana plugins aren’t on cache.nixos.org, so when the substituter was unreachable, Nix silently fell back to building from source instead of failing loudly.
Then, while fixing the cache issues, I realized the overlay structure itself couldn’t support what I needed. The flat layout (nixpkgs/pkgs/ and nixpkgs/modules/) had no way to hold different package versions for different NixOS channels. When 25.11 came along, there was no mechanism to keep 25.05 and 25.11 packages side by side. That became KB#8168: Multi-version cache architecture and stale artifact cleanup.
And then, on June 21, 2026, proxmini01 crashed during a build-local-nix-cache run on scaffold-cache-ct0. All 6 scaffold-cache ct0 containers lived on proxmini01, each allocated 8GB. That’s 48GB committed on a host with 30GB RAM. The OnBootSec=20s timer meant all 6 fired nix builds simultaneously after reboot. proxmini02 had to be power-cycled from a zombie state during recovery. That became KB#8198: Redistribute scaffold-cache ct0 containers + harden build-local-nix-cache.
Root Cause
The core issue for the original symptom was a race condition in create_image.sh. The script ran nix commands before the container reached a usable state. systemd’s is-system-running reports running only when all startup tasks complete; before that, services like nix-daemon may not have started yet. Running nixos-rebuild switch against a half-booted system produces unpredictable failures, and the stderr truncation was a consequence of commands dying in unexpected ways mid-execution.
The silent source builds had two root causes. The nix_cache_pubkey default of an empty string produced a key that Nix rejects as corrupt; any consumer that didn’t explicitly override the default got a broken substituter configuration. And the missing outbound port 5000 rules in the env-level security groups meant containers couldn’t reach nix-serve at all. Nix’s substituter fallback behavior is to silently build from source when a substituter is unreachable, which is a footgun; you only notice when a container takes 20 minutes instead of 30 seconds.
The flat overlay architecture was an unknown unknown. It worked fine for a single NixOS version. The moment I needed 25.05 and 25.11 side by side, there was no place to put the different package versions. The structure assumed one version of everything, and that assumption was never tested until it broke.
The OOM crash was straightforward memory overcommit. 6 containers at 8GB each on a 30GB host is 48GB committed. Proxmox overcommit works fine until all 6 containers fire nix-build at the same time, which is exactly what the OnBootSec=20s timer guaranteed after a reboot. The kernel OOM killer kicked in, and proxmini01 went unresponsive.
The Fix
The first fix landed on May 18, 2026 in commit 7d2cc28. I added a two-stage wait to create_image.sh:
- Wait for
/etc/set-environmentto exist, which means PATH is available. - Wait for
systemctl is-system-runningto reportrunning, which means systemd finished all startup tasks.
Then I added an explicit systemctl start nix-daemon before any nix commands, plus a retry loop that waits for the nix-daemon socket to be reachable:
nix --extra-experimental-features nix-command store infoThe loop runs 30 retries and restarts nix-daemon on failure. The commit message captured it: “Fix nix-daemon race condition in create_image.sh: wait for systemd running state, start nix-daemon explicitly before nix commands, and retry nix-daemon socket until reachable.”
The key insight here: don’t just check if a file exists, check if systemd reports the system is actually up. /etc/set-environment existing tells you PATH is set; systemctl is-system-running tells you the system is ready for work.
The cache fixes landed on May 26-27, 2026. Commit 3496044 changed the nix_cache_pubkey default from an empty string to None with ValueError validation, so a missing pubkey fails loudly instead of silently producing a corrupt key. The firewall fix added port 5000 outbound rules to both andrewcz-net and andrewcz-org env-level security groups via pvesh. I verified all 6 environments after the change.
The architecture restructure landed on May 27, 2026 in commit 3e5b8c8: 29 files changed, 1048 insertions, 269 deletions. I restructured from flat to per-channel:
nixpkgs/25.05/{pkgs,modules}/
nixpkgs/25.11/{pkgs,modules}/Each package default.nix has version, hash, and build instructions inline. No central versions.nix. I replaced file pushing to containers with a custom Nix channel subscription. The cache server runs generateOverlayNixExprs, an activation-blocking systemd service that creates overlays/nixexprs.tar.xz per channel at /var/public-nix-cache/$ch/overlays/. Containers subscribe to the overlay channel at activation time with a retry loop: 6 attempts, 5 second intervals, hard fail if all attempts fail.
I also changed nix-channel --update from bare (which updates all channels at once and fails entirely if any single channel fails) to per-channel: nixos first, then overlays separately. The overlay subscription is guarded by:
grep -q 'overlays/pkgs\|overlays/modules' /etc/nixos/service.nixso template images skip it. Stale artifacts got cleaned up too: removed /var/public-nix-cache/24.05 and 24.11 from all 6 cache servers, plus 23.11 from andrewcz-org.
A follow-up commit 036280d on June 25, 2026 moved opensearch to the nixbuild overlay, renamed build-local-nix-cache to build-overlay-packages, added the nixbuild substituter and resource limits, and added litestream 0.5.12 to the overlay.
The OOM hardening landed around June 21-24, 2026. I redistributed the scaffold-cache ct0 containers across hosts so no single box held all 6. Then I added build hardening to the build-overlay-packages service:
Nice=19
IOSchedulingClass=idle
NODE_OPTIONS=--max-old-space-size=3072
UV_THREADPOOL_SIZE=4
CPUQuota=200%The CPUQuota=200% doesn’t actually propagate through nix-build sandboxing, but the rest of the limits help. Recovery was controlled: I started containers one at a time with 4-5 minute pauses between each, checking host resources before starting the next one.
What We Learned
- Race conditions in build scripts are notoriously hard to diagnose because the symptom looks like a flaky build, not a timing issue. The stderr truncation sent me down the wrong path for months; the real problem was that
nixos-rebuild switchran against a half-booted system. - Nix’s silent fallback to source builds is a footgun. When a substituter is unreachable, Nix should fail loudly, not silently spend 20 minutes building
ghost-blogfrom source. The missing port 5000 rules were invisible for exactly this reason. - The flat overlay structure was an unknown unknown. It worked fine until I needed multiple NixOS versions, and then it didn’t work at all. The per-channel restructure is more careful and prudent; it handles version differences explicitly instead of assuming one version of everything.
- Memory overcommit on Proxmox is fine until all containers fire builds at the same time. The
OnBootSec=20stimer turned a reasonable allocation into a guaranteed OOM. Redistributing containers and addingNice=19plusIOSchedulingClass=idleprevents the thundering herd. - The
systemctl is-system-runningwait was the key insight. Don’t just check if a file exists; check if systemd reports the system is actually up. That distinction took me from “flaky build” to “race condition” in one afternoon.