in category:HomeLab program:Opencode process:Troubleshoot ~ read.

How Opencode Kept Crashing

Opencode has been running on two NixOS LXC containers behind keepalived VIP failover since deployment. It kept crashing. Not because of opencode itself; three distinct root causes in the NixOS bootstrap and service configuration, each one making the others worse.

The first root cause silently corrupted the opencode binary. The second caused the VIP to ping-pong between nodes every 18 seconds. The third meant that when failover did work, the restore process picked the wrong backup and lost data.

What Happened

KB#8194: Opencode in homelab documents three root causes that were all in the infrastructure, not in opencode itself.

The symptoms were confusing at first. Opencode would work for a while, then segfault. The VIP would jump between ct0 and ct1 every 18 seconds. Sessions would disappear on failover. The pmcp process (which manages MCP server connections) kept getting SIGTERM’d.

It turned out these were three separate bugs, each making the others worse:

Root Cause #1: patchelf –set-rpath permanently breaks Bun-compiled binaries

The NixOS bootstrap script runs patchelf --set-rpath on binaries to set the library search path. For most binaries this is fine. For Bun-compiled single-file executables (like opencode and its companion tool ocx), it permanently breaks them.

patchelf --set-rpath relocates the ELF dynamic section from near the end of the file (offset ~0x6364000) to near the beginning (offset ~0x318). Bun’s runtime expects the original layout and segfaults immediately. The damage is irreversible; removing the rpath afterward does not fix it.

The fix: changed patch_nixos_binaries() to use --set-interpreter only by default, with an exception list (NIXOS_RPATH_BINS) for the two Node SEA binaries (dav-mcp, webdav-mcp-server) that still need --set-rpath because they dynamically link against libstdc++. The gcc.cc.lib package stays in system packages for the exception list’s rpath.

This is a subtle footgun in the NixOS binary patching workflow. Bun compiles JavaScript into a single executable that embeds the V8 runtime. The executable has a valid ELF header, so patchelf treats it like a normal binary and relocates the dynamic section. But Bun expects the file layout to be unchanged, and the relocation breaks it irreversibly. There’s no warning; the binary just segfaults on launch.

Root Cause #2: keepalived health check too aggressive, causing VIP to flap every 18 seconds

The keepalived health check had fall = 3, which means 3 consecutive failures (at interval = 2, that’s 6 seconds) before declaring FAULT. When a node becomes MASTER, notify_master.sh restores the database from Litestream and restarts opencode, which takes 6-8 seconds. During that window, the health check fails 3 consecutive times, putting the node into FAULT. The other node takes MASTER, runs the same restore+restart cycle, fails its own health check, and the cycle repeats every ~18 seconds.

Both nodes were ping-ponging the VIP. pmcp was a victim of this (getting SIGTERM’d by carter’s restart cycle each time), not a cause.

The fix: changed fall = 3 to fall = 10 (20 seconds before FAULT). With interval = 2, this gives opencode 2.5x its restart time before keepalived gives up on it. rise = 3 stays at 6 seconds for fast recovery. Added documentation to notify_master.sh explaining the timing.

This is a classic keepalived misconfiguration. The fall parameter should be set to tolerate the longest expected startup time. If your service takes 6-8 seconds to start after a failover, fall = 3 at interval = 2 guarantees that every failover will immediately trigger another failover.

Root Cause #3: Litestream generation explosion (284 stale generations in S3)

Sessions were not being transferred between nodes on failover. ct0 had 2 sessions while ct1 had 9; they had completely diverged.

Three interconnected problems:

  1. Generation explosion. Every MASTER transition creates a new Litestream generation because the WAL header doesn’t match after a fresh restore. With the keepalived flapping from Root Cause #2, this created 284 generations in S3, each with one nearly-empty snapshot.
  2. Default restore picks the wrong generation. litestream restore (without -generation) picks the generation with the most recent snapshot timestamp. After flapping, the latest generation has almost no data (ct0’s 2 sessions) while the generation with real data (9 sessions, 253KB) is older and gets ignored.
  3. notify_master.sh doesn’t specify a generation. It runs litestream restore -if-replica-exists without -generation, so it always picks the latest (wrong) one.

Confirmed: manual restore from generation 5eadd4fea51efaea yields 9 sessions with 194 messages. Default restore picks generation b56ed69582de9b0f which yields only 2 sessions with 14 messages.

The fall=10 change from Root Cause #2 will stop the flapping and prevent new generation explosions, but the restore logic in notify_master.sh still needs to be smarter about generation selection. The 284 stale generations in S3 also need cleanup.

The Fix

Root Cause #1 (patchelf): Changed patch_nixos_binaries() to use --set-interpreter only by default, with an NIXOS_RPATH_BINS exception list for Node SEA binaries. Deployed in commit 64319d6.

Root Cause #2 (keepalived flapping): Changed fall = 3 to fall = 10 in the keepalived config. This gives opencode 20 seconds to start before keepalived declares FAULT, which is 2.5x the observed startup time. Deployed in commit 64319d6.

Root Cause #3 (Litestream generations): Not yet fixed. The fall=10 change prevents new generation explosions, but notify_master.sh still needs smarter generation selection, and the 284 stale generations in S3 need cleanup. Tracked as remaining work on the ticket.

Additional fixes deployed in the same commit: - Memory increased from 2048MB to 4096MB - GlusterFS shared mount for projects between ct0 and ct1 - Four vestigial package.json files deleted from the carter dotfiles repo (.opencode/tool/package.json, .opencode/tool/bun.lock, .opencode/plugin/package.json, .opencode/plugin/bun.lock). These created module boundaries that blocked Bun from resolving @opencode-ai/plugin. - Subdomain swap: carter subdomain now points to opencode instead of the decommissioned anythingllm

What We Learned

  • patchelf --set-rpath permanently breaks Bun-compiled binaries. The damage is irreversible; you can’t undo it by removing the rpath. If you’re patching binaries on NixOS, use --set-interpreter only for Bun/Node SEA executables, and maintain an exception list for binaries that genuinely need rpath (like Node SEA binaries that link against libstdc++).
  • Keepalived fall must exceed the service startup time. With fall = 3 and interval = 2, keepalived declares FAULT after 6 seconds. If the service takes 6-8 seconds to start, every failover triggers another failover. fall = 10 gives 20 seconds, which is 2.5x the observed startup time.
  • Litestream generation selection matters on failover. The default litestream restore picks the latest generation, which after flapping will be a nearly-empty generation. You need to either specify -generation explicitly or implement smarter selection logic (e.g., pick the generation with the most data, not the most recent snapshot).
  • Test failover, not just startup. The keepalived flapping only manifested when both nodes were running and competing for the VIP. Testing opencode in isolation showed it working fine. The bug only appeared during actual failover conditions.
  • Four package.json files in the wrong directories can break Bun module resolution. Bun walks up the directory tree looking for node_modules/, and it stops at package.json boundaries. Vestigial package.json files in .opencode/tool/ and .opencode/plugin/ created artificial boundaries that prevented Bun from finding @opencode-ai/plugin in the parent node_modules/.

Tickets