in category:HomeLab program:Garage process:Troubleshoot ~ read.

How We Lost Garage Clusters to Node Identity Confusion

Garage is our S3-compatible object storage. We run it across seven stacks, each with two containers for redundancy behind Keepalived VIPs. It stores metrics data from Mimir and Loki, Litestream backups for SQLite databases, and anything else that needs durable object storage in the homelab. When Garage goes down, monitoring goes blind and databases lose their backup path. So when the andrewcz-com stack’s Garage cluster split in half and the VIP refused to fail over to the healthy node… we noticed.

What Happened

The andrewcz-com LGTM (Mimir, Loki, Tempo, Grafana) stack went down hard. Mimir and Loki couldn’t reach Garage S3, and the Keepalived VIP wasn’t failing over to the healthy node. Two separate problems, one cluster:

Problem 1: Garage node identity wasn’t persistent (KB#8204)

Garage uses Ed25519 keypairs for node identity. The node_key file (a 64-byte libsodium secret key) is generated on first start and stored in /var/lib/garage/meta/. But our NixOS config doesn’t persist /var/lib/garage/meta/ across rebuilds; it’s on the container’s ephemeral root filesystem. When a container gets rebuilt (which we need to do occasionally to pick up config changes), the metadata directory is empty, Garage generates a new keypair, and the node gets a new identity.

The old node ID is still registered in the cluster layout as FAILED. The new node can’t rejoin because the layout still has the old ID stuck in a dead state. The cluster splits in half: ct0 is running with a random new ID that nobody recognizes, ct1 is running with the old ID but thinks ct0 is dead. Neither node can achieve quorum.

This is a footgun that only fires during container rebuilds. If Garage starts on a fresh metadata directory, you get a new identity every time. The layout never converges because the dead node ID blocks the new one from joining.

Problem 2: Keepalived was checking the wrong port (KB#8204)

The Keepalived health check for Garage was configured to hit port 3900. But Garage v2 doesn’t listen on port 3900. The actual ports are 3901 (RPC), 3902 (S3 API), and 3903 (admin). The health check was running curl -sf http://127.0.0.1:3900/health on every VRRP cycle, getting a connection refused, and… doing nothing. Because the check always failed, Keepalived never triggered a failover. The VIP stayed on whichever node happened to be MASTER when the check started failing, even if that node’s Garage process was completely down.

So when ct0’s Garage crashed and came back with a new identity, the VIP was stuck on ct0. ct1 was healthy but never got the VIP because Keepalived thought ct0 was fine (the check was failing on both nodes equally, so it never switched). Mimir and Loki on the andrewcz-com stack were pointing at the VIP for S3, getting errors, and going dark.

Root Cause

Two independent issues, both with the same theme: the cluster had no mechanism to maintain stable identity or detect real health.

  1. Non-persistent node identity: The node_key file lives in /var/lib/garage/meta/, which is on the container’s ephemeral root filesystem. When NixOS rebuilds the container, the metadata directory is empty, and Garage generates a fresh keypair. The old node ID remains in the cluster layout as FAILED, blocking the new identity from joining. This is the same pattern as NixCacheKeypair (which we already had for cache signing keys), just not applied to Garage.
  2. Wrong health check port: The Keepalived configuration had curl -sf http://127.0.0.1:3900/health as the health check. Port 3900 doesn’t exist in Garage v2. The S3 API is on 3902. The health check was failing on both nodes equally, which means Keepalived never detected a difference between them. The VIP never moved.

The Fix

For node identity persistence:

Created GarageNodeKeypair, a Pulumi dynamic resource that generates Ed25519 keypairs in libsodium format (same pattern as our existing NixCacheKeypair). The resource outputs: - node_key_b64: base64 of the 64-byte libsodium secret key (seed + pubkey) - node_key_pub_b64: base64 of the 32-byte public key - node_id: 64-char hex string of the public key, used in garage layout assign and garage node connect

The keypair is generated once per container and stored as Pulumi secrets. Includes diff()/update() immutability protection so Pulumi never regenerates keys on redeploy.

Added an ExecStartPre to garage.service.nix that writes node_key and node_key.pub from Pulumi secrets before Garage starts. The script is idempotent; it only writes if the files don’t already exist.

Replaced dynamic peer discovery in the garage-cluster script with known node IDs from template args (PEER_NODE_ID, LOCAL_NODE_ID). This eliminates the discovery loop entirely; the cluster script knows exactly which nodes to connect and which IDs to assign.

Replaced dynamic layout assignment with a Jinja2 loop using deterministic node IDs, so garage layout assign always uses the Pulumi-generated IDs instead of whatever the API returns on that particular node (which could be different if the cluster is split).

For the Keepalived health check:

Changed the health check from curl -sf http://127.0.0.1:3900/health to curl -s -o /dev/null -w '%{http_code}' --max-time 2 http://127.0.0.1:3902/ | grep -q '^[0-9]'. This hits the actual S3 API port (3902) and accepts any HTTP response code, including 403 (which Garage returns for anonymous requests). The check now correctly detects whether the Garage process is listening, and Keepalived can actually fail over the VIP when the master goes down.

For the split-brain recovery:

The andrewcz-com cluster was the worst hit. ct0 had crashed and come back with a random new ID, forming its own layout. ct1 had the old dead ID stuck as FAILED in the layout. Recovery steps:

  1. Removed the dead node from the layout: garage layout remove <old-dead-node-id>
  2. Assigned the new ct0 with its deterministic ID: garage layout assign <new-ct0-id> -c 1 -z zone1 -t garage
  3. Applied the layout: garage layout apply --version 2
  4. Wiped ct0’s stale metadata directory and restarted; the ExecStartPre script wrote the deterministic keypair, and Garage rejoined with the correct identity

For the remaining stacks (cziryak-com, curatingcollections-com, andrewcz-net, andrewcz-org LGTM, andrewcz-org opencode, andrewcz-com opencode), we did rolling migrations: update the NixOS config with deterministic keys, rebuild each container, verify resync completes before moving to the next. The critical rule: never proceed to the next node until resync queue length = 0 on ALL nodes (check with sudo garage stats -a | grep "resync queue length").

andrewcz-net lost data on the first attempt because we wiped ct0 before verifying resync was complete. We corrected the procedure for all remaining stacks.

What We Learned

  • Ephemeral storage is a silent footgun for cluster identity. Garage’s node_key lives in /var/lib/garage/meta/, which is on the container’s ephemeral root filesystem. Every rebuild generates a new identity, and the old identity blocks the new one from joining the cluster. The fix is to generate the keypair outside the container (in Pulumi) and inject it before Garage starts. This is the same pattern we already used for NixCacheKeypair; we just hadn’t applied it to Garage yet.
  • Health checks that always fail are worse than no health checks. The Keepalived check hitting port 3900 was failing on both nodes equally, which meant Keepalived never detected a difference between them. The VIP stayed stuck on whichever node happened to be MASTER. A health check that can’t distinguish between healthy and unhealthy is worse than no check at all, because it gives a false sense of confidence.
  • garage layout commands use node IDs, not node names. The garage layout assign and garage node connect commands take the 64-character hex node ID, not a hostname or alias. When your node IDs are random and non-deterministic, you can’t put them in config; you have to discover them at runtime, which produces different results on split-brain nodes. Deterministic IDs eliminate this problem entirely.
  • Rolling migrations need a resync check between nodes. After rebuilding a Garage container, you must wait for data resync to complete before touching the next node. The check is sudo garage stats -a | grep "resync queue length" on ALL nodes, not just the rebuilt one. Skipping this check on andrewcz-net caused data loss on ct0.
  • The ExecStartPre idempotency pattern is worth copying. Writing keys only if they don’t already exist means you can redeploy without regenerating identities. The first deploy generates the keypair; subsequent deploys skip it. This is the same pattern as NixCacheKeypair and it’s become our standard for any secret that needs to survive rebuilds.
  • Accept any HTTP response for health checks on authenticated services. Garage returns 403 for anonymous S3 requests, which is correct behavior; the service is running and healthy, it just requires authentication. The health check should accept any HTTP response, not just 200. This is a general pattern for any service that requires auth: check that it’s listening, not that it’s letting you in.

Tickets