in category:HomeLab program:Proxmox process:Troubleshoot ~ read.

The Week the Proxmini Cluster Tried to Kill Itself

Three hosts, four incidents, one cascade of failures that taught us more about our homelab in a week than six months of normal operations. Here’s what broke, how we found it, and what we changed so it won’t happen again.

What Happened

It started at midnight on June 15th. All three proxmini hosts went completely dark, and what followed was a week of recovering from one failure after another… each one exposing a gap in how we’d set things up.

Incident 1: The fstrim collision (KB#8171)

Around midnight, all three proxmini hosts locked up. IOWait hit 68.7% on host 03. Load average spiked to 211. PSI some hit 99.2, which means near-total I/O stall for 22 minutes. Hosts 01 and 02 went completely dark, with all metrics zeroed from 00:02 to 00:15+.

The root cause was two fstrim operations colliding on shared GlusterFS storage. The hourly fstrim-lxc.service (which trims all LXC container filesystems via pct fstrim) and the weekly fstrim.service (which trims host filesystems including 2 x 954 GiB Gluster bricks on NVMe) both fired around midnight. At 22:00 and 23:00, only fstrim-lxc ran and completed normally in about 1m49s. At 00:00, host 02’s weekly fstrim started at 00:01:17 while fstrim-lxc was already running on all three hosts. The compound I/O demand saturated the shared NVMe/Gluster storage.

Making matters worse, vm.dirty_ratio=20 on 32 GB hosts meant the kernel allowed up to 6.4 GB of dirty pages before blocking writers. When the I/O path was saturated by concurrent TRIM operations, dirty pages couldn’t flush, and the system seized.

And here’s the part that still makes me shake my head: the fstrim-lxc.timer description said "Weekly" but the actual schedule was OnCalendar=hourly. So this collision window was open every single week when the weekly host fstrim landed near an hourly trigger.

Incident 2: The cold start (KB#8173)

After all three hosts came back up simultaneously, we had to do a full cold start. This is where the cascade really started showing itself:

  • Gluster FUSE mounts were missing on proxmini01 and proxmini02; had to mount -a
  • 4 containers had stale fstrim locks and needed pct unlock
  • All 21 MariaDB clusters came back healthy, which was genuinely surprising and a credit to the GTID-based replication setup
  • 1 of 5 PostgreSQL clusters had a broken standby; the data directory was partially present from a previous pg_basebackup, so the NixOS pre-start script tried initdb on a non-empty directory and failed with the error that the data directory already existed. Had to wipe /var/lib/postgresql/16/* completely, re-run pg_basebackup from the primary, and restart.
  • 8 of 13 Redis clusters had all nodes running as replicas pointing to a master IP that no longer existed. Sentinel lost track of all slaves (num-slaves=0). All 8 clusters were in read-only mode. The root cause: NixOS Redis config applies replicaof settings from config. If the Keepalived VIP hasn’t converged or the designated master isn’t running, all nodes start as replicas pointing to the old master IP. Redis doesn’t auto-promote; Sentinel also lost topology. For each cluster, I had to manually promote ct0 to master (REPLICAOF NO ONE), reconfigure ct1/ct2 as replicas of the ct0 IP, reset Sentinel on all 3 nodes (SENTINEL RESET mymaster), and verify topology. A fun gotcha: Redis Sentinel uses mymaster across all clusters, scoped by port and network, not by name. Each cluster has its own unique auth password.
  • PVE firewall for cziryak-com was missing an outbound rule for port 3100 (Loki). It had port 3200 labeled "Loki" but 3200 is actually Tempo, not Loki.
  • Kanboard on andrewcz-org had a schema migration error: Column already exists: color_id from migration version_132, but schema_version was still at 131. Had to manually update schema_version to 132 and restart phpfpm-kanboard + nginx.
  • andrewcz-net WireGuard tunnel was completely non-functional (transfer: 0 B received on both ct0 and ct1). thepipeline-tv ct0 had 33% packet loss on the WireGuard gateway.

Incident 3: The zombie hypervisor and the converge hang (KB#8199)

A few days later, proxmini02 went into a zombie state… memory pressure, unresponsive to SSH, ping, and corosync. It took down 72 containers.

The real kicker was what happened on the surviving nodes. Three MariaDB mariadb-converge services hung for 57+ minutes in activating (start) state. The databases themselves were running fine, but converge was stuck, so they never promoted to writable primary. Kanboard, Authelia, and LGTM were all stuck in read-only.

The root cause was a missing --connect-timeout in the mysql_peer() function. When proxmini02 went down, TCP SYN packets went to an unreachable host. The kernel retried. The mysql client blocked indefinitely. All three hung processes were in sleeping state with 0 bytes I/O, stuck for nearly an hour.

The converge script’s grace period and promotion logic were sound. The problem was step 3, the peer reachability check, never completing because the TCP connection never timed out.

Incident 4: The OOM killer that chose wrong (KB#8200)

proxmini02 crashed twice in two days (June 21 and June 23). The June 23 crash happened during a Pulumi deploy of andrewcz-net scaffold-cache. The host entered a livelocked state where the kernel OOM killer destroyed critical host processes (corosync, frr, journald) instead of container processes. The system was unresponsive for over 90 minutes.

The forensic timeline from journalctl tells the story: - 15:19: corosync reports "Totem is unable to form a cluster" continuously - 16:10: frr.service processes SIGKILL'd (the networking stack, killed by OOM) - 16:47-16:56: systemd-oomd crashes four times - 16:50: corosync "not scheduled for 5,382,272 ms" (that’s 90 minutes) - 16:55: pve-ha-lrm "loop took too long (6,518 seconds)" (108 minutes) - 16:56-16:59: journald "Under memory pressure, flushing caches" multiple times

Why did the OOM killer choose so poorly? All host processes had the default oom_score_adj=0. There was no systemd-oomd installed, no earlyoom daemon, and vm.min_free_kbytes was at the default 67584 (about 66 MB). The kernel had zero guidance on what to kill.

And why did the host lock up entirely? 71 containers with 5.8x memory overcommit (174 GB committed on 30 GB physical RAM). During a Pulumi deploy, nixos-rebuild runs across many containers simultaneously, exhausting physical RAM for host processes. LXC cgroup limits protect from individual containers, but they don’t protect from aggregate memory pressure.

Root Cause

There were four distinct technical causes, but they share a theme: the cluster had no defenses for the failure modes it was operating in.

  1. fstrim collision: Two concurrent TRIM operations on shared NVMe/Gluster storage saturated I/O. The fstrim-lxc.timer was mislabeled as "Weekly" but was actually OnCalendar=hourly, so the collision window existed every week. Dirty page thresholds (vm.dirty_ratio=20 on 32 GB hosts, allowing 6.4 GB of dirty pages) meant the kernel couldn’t flush pages when I/O was blocked.
  2. Cold start cascade: After a full power outage, services that depend on each other came up in random order. Redis clusters all started as replicas pointing to VIPs that didn’t exist yet. PostgreSQL standbys had partially-synced data directories. Gluster mounts were missing. There was no orchestrated start order and no dependency-aware health checks.
  3. mariadb-converge hang: The mysql_peer() function had no --connect-timeout on its TCP client. When a peer host went down, TCP connections blocked indefinitely. The converge script is sound; step 3 (peer reachability) just never completed because the TCP SYN never timed out.
  4. OOM misbehavior: 5.8x memory overcommit with zero OOM tuning. All host processes at default oom_score_adj=0. No earlyoom, no systemd-oomd, minimal vm.min_free_kbytes. When memory pressure hit, the kernel had no idea which processes were critical and which were expendable.

The Fix

For the fstrim collision: - Replaced Conflicts=fstrim.service (initial fix) with After=fstrim.service on fstrim-lxc.service. Conflicts would stop whichever service was already running; we wanted ordering, not mutual exclusion. - Added a drop-in override at /etc/systemd/system/fstrim.service.d/after-fstrim-lxc.conf with After=fstrim-lxc.service, so the weekly fstrim also waits for fstrim-lxc if it’s running. Both ordering directions covered. - Tuned dirty page ratios: vm.dirty_ratio 20 -> 5, vm.dirty_background_ratio 10 -> 3, persisted in /etc/sysctl.d/99-dirty-page-tuning.conf. This means the kernel starts writeback earlier and blocks sooner, preventing the 6.4 GB dirty page accumulation that made the I/O stall so severe. See the Linux kernel documentation on dirty page throttling for why these values matter. - Fixed the timer description from "Weekly" to "Hourly".

For the cold start: - Documented a container start protocol: start one container, wait 2-3 minutes for stabilization, check host resources, then start the next. Minimum 4-5 minutes between consecutive container starts. - Redis clusters required manual promotion for each of the 8 broken clusters. Each cluster needed: promote ct0 (REPLICAOF NO ONE), reconfigure ct1/ct2 as replicas, reset Sentinel on all 3 nodes (SENTINEL RESET mymaster), verify topology. - PostgreSQL standby fixed by wiping the partial data directory and re-running pg_basebackup. - Kanboard schema migration fixed by manually updating schema_version to 132 and restarting the services. - PVE firewall rule added for Loki on port 3100.

For the converge hang: - Added --connect-timeout=5 to mysql_peer() so TCP connections to unreachable peers fail fast. A 5-second timeout instead of indefinite blocking. The MySQL documentation on connect-timeout explains why this matters for HA setups. - Killed the three hung converge services via systemctl stop. The timer re-triggered converge, which completed successfully because the kernel had already exhausted TCP SYN retries. - All three databases (kanboard, authelia, lgtm) promoted to writable primary.

For the OOM killer: - Installed earlyoom on all proxmini hosts. earlyoom monitors memory pressure and kills processes before the kernel OOM killer is invoked, with configurable preferences for which processes to sacrifice. - Set oom_score_adj=-1000 on 8 critical host processes (corosync, frr, journald, etc.) so the kernel will never kill them. The Linux OOM score documentation explains how oom_score_adj works. - Increased vm.min_free_kbytes from 66 MB to 512 MB. This reserves memory for the kernel to function even under pressure. - Reduced vm.swappiness from 60 to 10. Less aggressive swapping means less I/O churn when memory is tight. - Additional items tracked in KB#8205 (tech debt): per-container memory.oom.group=1, host swap configuration, container memory.high limits.

What We Learned

  • fstrim timer descriptions lie. The fstrim-lxc.timer said "Weekly" but was actually OnCalendar=hourly. The label we were reading was wrong, and this collision window existed every week. Always verify the actual schedule, not the human-readable description.
  • Dirty page thresholds matter more than you think. On 32 GB hosts with vm.dirty_ratio=20, the kernel would let 6.4 GB of dirty pages accumulate before blocking writers. When I/O is saturated, those pages have nowhere to go. Lowering to 5% (1.6 GB) starts writeback earlier and blocks sooner, preventing the worst case.
  • Conflicts= vs After= is not a trivial distinction. Conflicts=fstrim.service would stop whichever service was already running. After=fstrim.service means “wait for this to finish first, then run.” We needed ordering, not mutual exclusion, because both operations need to happen; they just shouldn’t happen at the same time.
  • Cold starts are a great test of your assumptions. We assumed services would come back up cleanly after a power outage. They didn’t. Redis clusters that rely on Keepalived VIPs for master election all started as replicas when the VIPs didn’t exist yet. PostgreSQL standbys had partially-synced data directories that confused the init script. A container start protocol with resource checks between each start prevents cascading memory pressure.
  • Missing timeouts are time bombs. The mysql_peer() function with no --connect-timeout was fine for 99% of operations… until a peer host went down and it blocked for 57+ minutes. Every network call needs a timeout. Not because things fail often, but because when they do, you want to know in 5 seconds, not 57 minutes.
  • 5.8x memory overcommit is fine until it isn’t. LXC cgroup limits protect from individual containers, but 71 containers all running their allocations simultaneously during a deploy exhausted physical RAM for host processes. The kernel’s default OOM behavior had no idea which processes were critical. Setting oom_score_adj=-1000 on host processes and installing earlyoom means the next OOM event will kill expendable container processes, not the cluster networking stack.
  • Defense in depth works. The fstrim fix alone (ordering) would have prevented the midnight crash. But we also added dirty page tuning as a second layer, so even if ordering fails or a different I/O storm hits, the kernel won’t let 6.4 GB of dirty pages accumulate. Same with OOM: earlyoom catches it early, oom_score_adj protects critical processes, and min_free_kbytes gives the kernel breathing room.

Tickets