Rebuilding the Nix Overlay Architecture for Per-Channel Packages

The cache server was building overlay packages using a single flat directory for all NixOS channels. When we wanted Ghost 6.10.3 on 25.05 and Ghost 6.10.4 on 25.11, we couldn’t express it. The overlay structure had no concept of per-channel versions.

We restructured the entire overlay to support per-channel packages, built a custom Nix channel distribution system, and deployed it across all six cache servers and every container in the homelab.

What Happened

KB#8168: Multi-version cache architecture and stale artifact cleanup tracks the full restructure.

The old overlay structure mirrored the upstream nixpkgs convention: a single nixpkgs/pkgs/ directory with one default.nix per package. Version strings lived in a central versions.nix registry. This worked when every channel ran the same version of every package, but the whole point of separate NixOS channels (25.05, 25.11) is that they can diverge. When they do, you need nixpkgs/25.05/pkgs/ghost-blog/default.nix and nixpkgs/25.11/pkgs/ghost-blog/default.nix with different versions and potentially different build instructions.

The deeper problem: Pulumi was pushing pkgs/ and modules/ to /etc/nixos/ on every container. Client containers downloaded nixexprs.tar.xz from cache.nixos.org, which only knows about upstream nixpkgs. Overlay packages were only available via binary cache substitution; if nix-serve was down, containers fell back to building from source, and there was no default.nix on disk for them to find.

Root Cause

Two interconnected problems:

1. Flat overlay structure can’t express per-channel divergence. The old nixpkgs/pkgs/ghost-blog/default.nix applied the same version to all channels. A central versions.nix registry created coupling between version strings and build instructions that falls apart the first time a version bump requires a build change.

2. Overlay files were pushed to every container instead of distributed through Nix channels. Pushing files to /etc/nixos/ on client containers meant they existed outside the Nix channel system. There was no way for containers to subscribe to overlay updates; they had to be redeployed through Pulumi every time a package version changed.

Additionally, after removing 24.05 and 24.11 from the build channel loop, stale tarballs and artifacts still sat on all six cache servers. There was also a stale 23.11 tarball on andrewcz-org.

The Fix

Per-channel overlay structure

Restructured nixpkgs/ from the flat layout to per-channel directories:

nixpkgs/
  25.05/
    pkgs/
      default.nix        (overlay for 25.05)
      ghost-blog/default.nix
      litestream/default.nix
    modules/
      default.nix
      ghost-blog.nix
      gramps-web.nix
  25.11/
    pkgs/
      default.nix        (overlay for 25.11)
      ghost-blog/default.nix
      litestream/default.nix
    modules/
      default.nix
      ghost-blog.nix
      gramps-web.nix

Each package default.nix contains version + hash + full build instructions inline. No separate version registry, no shared/ directory. Version strings and build instructions live together because an upgrade can change the build instructions too, and splitting them just creates coupling that falls apart the first time a version bump requires a build change.

Build script changes

build-overlay-packages (formerly build-local-nix-cache) changed from applying a single overlay to all channels:

nix-build -E "
  with import $nixpkgsPath {
    overlays = [ (import /etc/nixos/pkgs) ];
  };
  $pkg
"

To applying the per-channel overlay for each channel:

nix-build -E "
  with import $nixpkgsPath {
    overlays = [ (import /etc/nixos/$ch/pkgs) ];
  };
  $pkg
"

Custom Nix channel distribution

Instead of pushing overlay files to every container, the cache server now generates a lightweight overlays.tar.xz per channel containing just the overlay and module definitions. Client containers subscribe to both channels:

nix-channel --add http://<cache-vip>/25.05/nixexprs.tar.xz nixos
nix-channel --add http://<cache-vip>/25.05/overlays.tar.xz overlays

A new generateOverlayNixExprs service (activation-blocking, no timer) creates the tarballs per channel at /var/public-nix-cache/$ch/overlays/nixexprs.tar.xz. Nginx serves them to client containers.

The overlay subscription in configuration.nix is guarded by a grep check: grep -q 'overlays/pkgs\|overlays/modules' /etc/nixos/service.nix. Template images (empty service.nix) skip the subscription entirely. There’s also a retry loop: 6 attempts, 5-second intervals, hard fail if the overlay tarball isn’t available.

nix-channel --update is now per-channel (nixos then overlays separately) because bare nix-channel --update fails entirely if any single channel fails.

Deployment

Deployed across all stacks:

Cache servers get all channels’ overlay files pushed to /etc/nixos/$ch/{pkgs,modules}
App containers get zero overlay files pushed; everything comes through channel subscription
Old /etc/nixos/pkgs/ and /etc/nixos/modules/ references removed from functional code
Stale artifacts (24.05, 24.11, 23.11) cleaned from all cache servers

What’s still open

A few packages haven’t been extracted from inline service templates into per-channel overlays yet: bareos-fd and atom are still inline in their service templates. The gramps-web.nix module references pkgs.gramps-prometheus-exporter but no package directory exists for it in either per-channel overlay (the package was deleted during migration and never recreated).

What We Learned

Per-channel directories beat per-channel branches for overlay packages. The upstream nixpkgs approach uses git branches (each nixos-XX.XX branch owns its own pkgs/ tree). That doesn’t work for us because we push a single overlay directory. Per-channel directories within that directory (25.05/pkgs/, 25.11/pkgs/) give us the same divergence capability without branch management overhead.
Version strings and build instructions should live together. The old versions.nix registry created coupling between version strings and build instructions. The first time a version bump required a build change, the registry became a liability. Inline versions in default.nix are simpler and more maintainable.
Pushing overlay files to containers is the wrong distribution model. Files in /etc/nixos/ exist outside the Nix channel system. They can’t be updated without a Pulumi deploy, and they disappear if nix-serve is down. Custom Nix channels let containers subscribe to overlay updates the same way they subscribe to upstream nixpkgs updates.
generateOverlayNixExprs should be activation-blocking, not timer-based. Overlay tarballs need to exist before any container can subscribe to the channel. A timer could fire before the tarballs are ready, causing the subscription to fail. Making the service activation-blocking ensures the tarballs exist before nixos-rebuild switch runs.
nix-channel --update fails entirely if any channel fails. Splitting the update into per-channel calls (nixos then overlays) means one channel failure doesn’t block the other.