in category:HomeLab program:Nix process:Install ~ read.

How We Stopped Building the Same Nix Packages Twelve Times

Twelve containers. Every single one of them building the same overlay packages from source. Ghost, Litestream, Consul plugins… all compiled independently on machines that were already starved for memory. When build-local-nix-cache fired on all of them at once, it ate through RAM caps and crashed the hypervisors underneath them. More than once.

We needed a different approach. Instead of twelve independent build pipelines all doing the same work, we built one server that does it once and shares the result.

What Happened

The homelab runs NixOS containers across six domains, each with a pair of cache servers running nix-serve for local binary caches. Each cache server also runs build-local-nix-cache, a systemd service that compiles overlay packages (packages not in the upstream cache.nixos.org binary cache) from source. With two cache servers per domain across six domains, that’s twelve containers all building the same set of packages independently.

KB#8201: nixbuild: dedicated build server for overlay packages describes the problem clearly: these builds are memory-hungry (the service is capped at 8GB) and CPU-heavy, and they’ve been crashing hypervisors on more than one occasion. When multiple cache servers start building simultaneously, the host machines run out of RAM and the OOM killer starts taking down critical processes.

The existing setup had nix-serve running on each cache server, sharing a signing key across domains. But each server built locally because there was no shared binary cache between them. The first cache server to finish a build would sign its output, but the second server in the same domain wouldn’t use that signed output; it would build from source again.

Root Cause

There were two interconnected problems:

1. No shared binary cache for overlay packages. The Nix substituter mechanism allows machines to pull pre-built binaries from a remote cache instead of building from source. Each cache server was configured with cache.nixos.org as its only substituter, which works for upstream packages but not for overlay packages like ghost-blog or litestream. With no substituter for those packages, nix-build had no choice but to compile from source every time.

2. The build-local-nix-cache service had no coordination. It ran on every cache server on a daily timer, independently. There was no mechanism to say “skip this build, the other server already has it.” Twelve containers, twelve identical builds, twelve opportunities to crash a hypervisor.

The interaction between these two problems is what made it dangerous. On a single domain, both cache servers would start building at roughly the same time, doubling the memory pressure. Across six domains, that’s twelve parallel builds of the same packages, each one consuming up to 8GB of RAM.

The Fix

We built a dedicated Nix binary cache server; a singleton LXC container called nixbuild that builds overlay packages once and serves them over HTTP.

The architecture is straightforward:

  • nixbuild runs on hub-proxbak in the andrewcz-net domain, with 22GB RAM and 7 CPUs
  • It runs nix-serve on port 5001, proxied through Traefik at https://nixbuild.andrewcz.net
  • It has its own NixCacheKeypair (separate from the cache server keypair), created in the exfra Pulumi stack
  • Cache servers point build-overlay-packages at nixbuild via NIX_SUBSTITUTERS and NIX_TRUSTED_PUBLIC_KEYS environment variables, scoped to that service only
  • Consumer containers are unchanged; they still point at their local cache server

The enable_local_builds toggle on cache servers controls fallback behavior. When false (the default), build-overlay-packages runs with --max-jobs 0, forcing it to substitute from nixbuild and fail if nixbuild is unreachable. When true, it tries nixbuild first and falls back to building locally.

The old build-local-nix-cache service was renamed to build-overlay-packages across both scaffold-cache and nixbuild templates, which more accurately describes what it does.

Deployment Gotchas

Deploying this hit several issues that are worth documenting because they’re the kind of thing that’ll bite you in a Nix setup:

Consul health check returning 404. The initial health check path returned a 404 status code, but we had configured Traefik to route based on a (200) matcher. Changed the check to return 200.

Traefik routing on the wrong port. The initial Traefik config routed websecure+TLS on 443 to the nixbuild entrypoint on port 5001 (plain HTTP), which is correct… but the Traefik router needed to point to port 5001 on the internal network, not the external HTTPS endpoint. The final config uses http://nixbuild.andrewcz.net:5001 as the internal substituter URL, with Traefik handling TLS termination separately.

Trusted key name mismatch. The scaffold-cache template was constructing the trusted key name using its own env_name variable instead of passing the full key string from the infrastructure. This meant the NIX_TRUSTED_PUBLIC_KEYS value was wrong; nix-build would reject the signatures from nixbuild because it didn’t recognize the key name. Fixed by passing the complete trusted key string from the Pulumi stack down through role template args.

Race condition in service ordering. The build-overlay-packages service had an After= dependency list that didn’t include downloadNixExprs.service and materializeNixpkgsChannels.service, while prefetchUpstreamPackages did include them. This meant build-overlay-packages could start before the nix expressions were available. Added both to the After= list to match prefetchUpstreamPackages.

NIX_SUBSTITUTERS replaces, not appends. This was the most subtle bug. Setting NIX_SUBSTITTERS in a service’s environment replaces the global substituters list entirely; it doesn’t append to it. When we set it to only http://nixbuild.andrewcz.net:5001, nix-build couldn’t resolve base nixpkgs dependencies because it no longer had cache.nixos.org as a fallback. Fresh cache servers failed with Unable to start any build because nix-build couldn’t find the transitive dependencies on nixbuild alone.

The nixbuild cache only has overlay packages (like ghost-blog and litestream), not the hundreds of nixpkgs dependencies they reference. Without cache.nixos.org in the substituters list, nix-build couldn’t fetch those transitive deps and failed before it could even attempt substitution from nixbuild.

The fix: include both nixbuild (primary) and cache.nixos.org (fallback for transitive deps) in NIX_SUBSTITUTERS, and their corresponding public keys in NIX_TRUSTED_PUBLIC_KEYS.

What We Learned

  • NIX_SUBSTITTERS replaces the global list, it doesn’t append. This is documented in the Nix manual but it’s easy to miss if you’re setting it as an environment variable for a single service. If you only specify your custom cache, nix-build won’t be able to resolve any transitive dependencies from the upstream cache.
  • A singleton binary cache is simpler than a distributed one. We could have set up mutual substituter relationships between cache servers, but that creates coordination complexity. One build server with one keypair, serving one purpose, is easier to reason about and debug.
  • Service-level environment variables are the right scope for substituters. By setting NIX_SUBSTITTERS and NIX_TRUSTED_PUBLIC_KEYS on the build-overlay-packages service unit rather than globally in nix.settings, everything else on the cache server (nixos-rebuild, nix-channel, nix-serve) continues using cache.nixos.org as its only substituter. The nixbuild substitution is scoped to exactly where it’s needed.
  • Test with a truly fresh machine, not one that already has packages in its store. The NIX_SUBSTITTERS bug only manifested on fresh cache servers that didn’t have transitive deps in their local store. Servers that had already built overlay packages locally had those deps cached, so they didn’t need cache.nixos.org as a fallback. The bug was hidden by the very caching it was supposed to provide.
  • Naming the service accurately matters for future debugging. build-local-nix-cache was a misnomer; it doesn’t build a local cache, it builds overlay packages. Renaming it to build-overlay-packages makes the service’s purpose clear at a glance, which helps when you’re staring at systemctl list-units at 2 AM trying to figure out what’s eating all the RAM.

Tickets