Filesystems Are the Hidden Killer in a Homelab
Over the course of several months, the filesystem stack across my 3-node Proxmox cluster started failing in layers. GlusterFS replication got stuck on an orphaned GFID from May 2025; Bareos backups silently broke because a sed filter that never ran; EXT4 started throwing warnings on 22+ dm devices; and putting Minio on Gluster turned out to be a fundamentally bad idea that forced an architecture change. This is the story of how it all unraveled and what we did about it.
What Happened
The cluster is three Proxmox hosts (hub-proxmini01, proxmini02, proxmini03) running GlusterFS 11.1 in distributed-replicate mode, with changelog enabled and build-pgfid turned on, rolling over every 300 seconds. The bricks sit on XFS on top of LVM. Containers run on EXT4 inside LVM thin pools. Multiple layers of the stack, all failing in related ways over a period of months.
The first sign of trouble was glusterfind not running successfully on subsequent runs. The initial error was ModuleNotFoundError: No module named 'xattr', which I fixed by installing python3-pyxattr on all proxmini servers. But that was just the surface. The real problem was a path mismatch: glusterfind SSHes to the Proxmox brick nodes and executes Python scripts at Fedora-style paths like /usr/libexec/glusterfs/glusterfind/changelog.py, but Debian and Proxmox install those scripts at /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind/. The error in the logs was:
bash: line 1: /usr/libexec/glusterfs/glusterfind/changelog.py: No such file or directoryand
nodeagent.py: No such file or directoryI symlinked /usr/libexec/glusterfs/glusterfind -> /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind on all three proxmini hosts. After that, all 6 gluster PreScripts ran successfully. One generated 8554 files for a Bareos job. But then the sed filter that was supposed to narrow those files down to just the changed ones reduced the list to 0 files for andrewcz-org (no incremental changes since January). And when Bareos got an empty file list, the Storage Daemon dropped the connection with:
Director's comm line to SD droppedthepipeline-tv and andrewcz-net failed nightly with that error. curatingcollections-com was intermittent. I tracked this in KB#7664: glusterfind does not run successfully on subsequent runs.
While debugging the glusterfind failures, I discovered something worse: the sed filter in the Bareos Command strings was never executing at all. Bareos Command directives don’t do shell interpretation; the ; separator I used got passed as a literal character. The backups that did succeed weren’t succeeding because of the filter… they succeeded because the gfapi plugin only backs up files under the mounted Gluster subdirectory. The filter was a no-op the entire time. I split the 0-file SD drop issue into KB#8209 and closed the glusterfind path issue in KB#7664.
Then glusterfind started hanging. Not on recent queries; a glusterfind query for changes since January 8, 2026 (timestamp 1765267225) hung indefinitely. Queries for the last hour worked fine. Multi-day queries had been hanging for weeks, and this broke Bareos backups starting January 3, 2026. I tracked this in KB#7916: glustfs one gfid issue.
gluster volume heal hub-volume info showed 1 entry on brick hub-proxmini01:/srv/glusterfs/bricks/brick2:
<gfid:95c3eddb-c6e4-4ec1-a927-f9bb3d96790b>There was also an unremovable directory in andrewcz-org/lgtm-minio. The GFID 95c3eddb was a symlink to an old directory structure, last modified May 8, 2025. It had no corresponding file in the mount point. The brick xattrs showed pending AFR operations:
trusted.afr.hub-volume-client-4=0x0000000b0000000b0000000band the dirty flag was set. The glusterd.log showed Dict get failed [{Key=key2}, {errno=2}] starting January 3, 2026 at 23:36 UTC. This orphaned GFID with stuck AFR attributes was causing glusterfind query to hang on any time range that included it.
Meanwhile, Minio was causing its own problems. Since I put Minio on Gluster, the changelog exploded in size. glusterfind query had to process an enormous number of changelog entries just to find the few files that actually changed, which compounded the hanging issue from KB#7916. I tracked this in KB#8072: LGTM Minio causing glusterfind to hang for days.
The capacity math told the story: 250GB Minio times 4 nodes = 1TB, 60GB nix-cache times 4 = 250GB, 120GB for LLM models. That’s a lot of churn on a replicated filesystem where every write generates changelog entries. I bid on an SSD on eBay to segment the disk; the plan was to move Minio, LLM models, and the nginx cache to local non-replicated storage and keep Gluster for the things that actually need replication.
And then EXT4 started complaining. On hub-proxmini01 only; proxmini02 and proxmini03 had no errors. I checked with:
journalctl -xe | grep 'EXT4-fs warning'and found 22+ dm devices with warnings. I tracked this in KB#8058: EXT4-fs errors. The lvs output told the rest of the story: thin pools at 100% for several VMs. vm-107, vm-135, vm-143, vm-161 were at 92-99% thin pool utilization. pct fstrim 161 trimmed 11.7 GiB and dropped the thin pool to 26.67%. That’s a massive amount of unmapped space that TRIM should have been reclaiming.
I ran fstrim across all containers:
pct list | tail -n+2 | cut -d ' ' -f 1 | xargs -n 1 pct fstrimThen I did a fsck sweep. For containers that needed offline repair:
for i in 154 141 243 110 213 284 285 289 280 321; do
pct stop ${i} && fsck -y /dev/pve/vm-${i}-disk-0 && pct start ${i}
doneI skipped the MariaDB servers during the first pass to avoid unnecessary downtime, then went back for a clean read. Two filesystems remained stubbornly dirty: CT 188 (hub-cziryak-com-nextcloud-mariadb-ct1, dm-39) and CT 196 (hub-andrewcz-com-vaultwarden-mariadb-ct1, dm-42).
CT 188 had 12 errors when I ran e2fsck: orphaned inodes, block and inode bitmap mismatches. CT 196 was mounting with the errors flag set, but e2fsck came back clean; the errors had been cleared by a prior mount and it just needed the flag reset. I verified replication was clean before and after each repair.
Root Cause
KB#7664: glusterfind path mismatch. GlusterFS ships from Fedora and RHEL-centric build systems, so the glusterfind helper scripts land in /usr/libexec/glusterfs/glusterfind/. Debian and its derivatives (including Proxmox, which is Debian-based) use the Debian multiarch path /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind/. The glusterfind code hardcodes the Fedora path when it SSHes to brick nodes. This is a known packaging issue; see gluster/glusterfs#1532. The python3-pyxattr missing module error was a separate but related dependency gap; glusterfind needs pyxattr for extended attribute operations.
The Bareos sed filter issue was a misunderstanding of how Bareos Command directives work. The Command directive in a Bareos Job or RunScript resource passes the string to the shell as a single argument; it doesn’t interpret shell metacharacters like ; the way a shell pipeline would. So somecommand ; sed 's/pattern/replacement/' doesn’t pipe through sed… it passes the literal ; and sed string as arguments to somecommand. The filter was dead code.
KB#7916: orphaned GFID with stuck AFR attributes. AFR (Africa Replicate) is GlusterFS’s replication mechanism. When a file is modified on one brick but not yet synced to its replica, AFR records the pending operation in extended attributes on the brick. The attribute trusted.afr.hub-volume-client-4 with value 0x0000000b0000000b0000000b indicates 11 pending operations across data, metadata, and entry changelogs.
The problem: the GFID 95c3eddb pointed to a symlink that referenced an old directory structure that no longer existed in the mount point. The file was effectively orphaned; it existed on the brick’s .glusterfs path but had no live entry in the filesystem. However, AFR still saw pending operations and the dirty flag, so gluster volume heal kept trying to heal something that couldn’t be healed. And glusterfind query, which reads the changelog to find modified files, would encounter this stuck entry and hang indefinitely when the query time range included it.
The glusterd.log entry Dict get failed [{Key=key2}, {errno=2}] (errno 2 is ENOENT… file not found) confirmed that glusterd was trying to look up a key that didn’t exist, likely related to the orphaned GFID’s missing directory structure.
KB#8072: Minio changelog explosion. Minio is an S3-compatible object store that does a lot of small writes: metadata files, multipart upload parts, bucket policies, index files. GlusterFS changelog records every file operation on the brick. When you put a high-churn object store on a replicated Gluster volume, every small write generates a changelog entry on every replica brick. The changelog files grew large enough that glusterfind query had to process an enormous number of entries to find the actual changes, which is why multi-day queries hung.
This isn’t a bug in either project; it’s a fundamental mismatch. Object stores with high write churn don’t belong on replicated filesystems with changelog tracking. The operational complexity of running glusterfind query against a changelog full of Minio metadata writes is not worth it.
KB#8058: EXT4 errors from full thin pools. LVM thin pools allocate blocks on demand from a shared pool. When the thin pool hits 100%, the kernel can’t allocate new blocks for writes, and EXT4 starts throwing warnings and errors in the journal. The fstrim operation discards unused blocks from the filesystem and returns them to the thin pool. Without regular fstrim, deleted files inside the container don’t release their blocks back to the thin pool, so the pool fills up even though the filesystem has free space.
The 22+ dm devices with warnings were all on hub-proxmini01, which is the busiest node in the cluster. The thin pools for vm-107, vm-135, vm-143, vm-161, and vm-173 were at 92-99% utilization. pct fstrim 161 reclaiming 11.7 GiB and dropping to 26.67% means that container had 11.7 GiB of deleted-but-not-trimmed space sitting in the thin pool.
The Fix
KB#7664: Symlinks for the glusterfind path. On all three proxmini hosts:
ln -s /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind /usr/libexec/glusterfs/glusterfindThis is an ad-hoc fix; a package upgrade could clobber it. Going forward, I’d like to try to get this into the GlusterFS Debian packaging properly, but for now the symlink works. I also installed python3-pyxattr on all proxmini servers.
The Bareos sed filter issue didn’t need a fix per se; the filter was never running, and backups were succeeding via the gfapi plugin’s subdirectory restriction. I split the 0-file SD drop issue into KB#8209 to handle separately.
KB#7916: Remove stuck AFR attributes and xfs_repair. I removed the orphaned AFR attributes from the .glusterfs path on brick2:
setfattr -x trusted.afr.hub-volume-client-4 /srv/glusterfs/bricks/brick2/.glusterfs/95/c3/95c3eddb-c6e4-4ec1-a927-f9bb3d96790b
setfattr -x trusted.afr.dirty /srv/glusterfs/bricks/brick2/.glusterfs/95/c3/95c3eddb-c6e4-4ec1-a927-f9bb3d96790bAfter that, all bricks showed 0 entries in gluster volume heal hub-volume info. I also wrote a split-brain resolution script (~100 lines) that parses gluster volume heal hub-volume info split-brain, identifies GFID entries, removes AFR attributes via SSH to brick hosts, and runs a full heal. This is the kind of bespoke tooling you end up writing when GlusterFS split-brain resolution is a manual, per-GFID process.
I also ran xfs_repair on proxmini01 to clean up the brick filesystems:
systemctl stop glusterd
killall glusterfs glusterfsd
umount /srv/glusterfs/bricks/brick1 /srv/glusterfs/bricks/brick2
xfs_repair /dev/nvme0n1p1
xfs_repair /dev/nvme0n1p2
mount -a
systemctl start glusterdKB#8072: Deprecate Minio, move to Garage. The decision was to deprecate Minio entirely and move to Garage, a self-hosted S3-compatible object store designed for distributed deployments. The architecture: 2-node active/passive with replication_factor=2, consistency_mode="dangerous", and a keepalived VIP for split-brain prevention.
I evaluated four options: 1. Independent Minio instances with rsync sync… rejected, too much operational complexity and no consistency guarantee 2. Default Garage consistency mode… rejected, requires both nodes available for writes 3. 3-node Garage cluster… rejected, I only have 2 nodes available for this 4. Shared block storage… rejected, no shared storage in this cluster
The selected architecture allows writes with 1 node available and does async replication to the returning node. The tradeoff: a small RPO if the active node fails before replication catches up. I’m unsure as to if this will be a problem in practice; the workload is mostly LGTM (Loki, Grafana, Tempo, Mimir) metrics and logs, which are append-heavy and tolerate small data loss. Prometheus metrics for health and degradation monitoring are part of the setup.
KB#8058: fstrim and fsck sweep. First, fstrim all containers to reclaim thin pool space:
pct list | tail -n+2 | cut -d ' ' -f 1 | xargs -n 1 pct fstrimThen, offline fsck for containers with dirty filesystems:
for i in 154 141 243 110 213 284 285 289 280 321; do
pct stop ${i} && fsck -y /dev/pve/vm-${i}-disk-0 && pct start ${i}
doneFor the two stubborn dirty filesystems, I ran e2fsck directly on the dm devices:
e2fsck -y /dev/pve/vm-188-disk-0 # CT 188: 12 errors, orphaned inodes, bitmap mismatches
e2fsck -y /dev/pve/vm-196-disk-0 # CT 196: clean, just needed errors flag resetI verified replication was clean before and after each repair. proxmini02 and proxmini03 had no EXT4 errors at all.
I also wrote a script to identify corrupted files from EXT4 I/O errors by mapping dm device to LV name to container ID to file path via debugfs ncheck.
KB#8162: Monitoring for EXT4 errors. This is the follow-up. The fix for KB#8058 was reactive; I only found the errors because I went looking. I need monitoring to alert on EXT4-fs errors on all 3 proxmini hosts, mapping dm device to container ID so the alert tells me which container is affected. The options I’m considering: a Vector journald source with pattern matching on EXT4-fs warning, a systemd timer checking dmesg, or Grafana Loki alert rules. I tracked this in KB#8162: Monitor for EXT4-fs errors on proxmini hosts.
What We Learned
- Object stores and replicated filesystems with changelog are a bad combination. Minio on Gluster was a footgun from the start. The changelog explosion wasn’t a bug; it was the inevitable result of putting a high-churn workload on a filesystem that records every operation. Going forward, high-churn workloads go on local storage, and Gluster is reserved for things that actually need replication.
- Bareos Command directives don’t do shell interpretation. I had a sed filter in a Command string that was never running, and backups were succeeding for a completely different reason than I thought. The filter was dead code for months. Especially since Bareos doesn’t warn you about this; it just silently passes the literal string.
- Orphaned GFIDs with stuck AFR attributes can hang glusterfind indefinitely. A file that was last modified in May 2025 broke backups starting January 2026 because the AFR attributes kept the heal process trying to sync something that didn’t exist anymore. The fix (removing xattrs manually) is not documented prominently; I had to piece it together from GlusterFS GitHub issues and trial and error.
- LVM thin pools need regular fstrim. 11.7 GiB of reclaimable space in a single container is a lot. Without fstrim, deleted files inside containers don’t release blocks back to the thin pool, and the pool fills up even though the filesystem reports free space. This is notoriously easy to miss because the filesystem looks fine but the thin pool is at 100%.
- EXT4 errors were only on the busiest node.
proxmini02andproxmini03had zero errors.hub-proxmini01had 22+ dm devices with warnings. The difference is load;hub-proxmini01runs more containers and more write-heavy workloads. Were anything weird to happen on the other nodes, I’d want to catch it early, which is why KB#8162 exists. - The symlink fix for glusterfind is ad-hoc and could be clobbered. A package upgrade could remove the
/usr/libexec/glusterfs/glusterfindsymlink and break glusterfind again. To the best of my knowledge, this hasn’t happened yet, but it’s a linger footgun. I’d like to try to get this fixed upstream in the Debian packaging. - Multiple layers of the stack failed in related ways over months. GlusterFS replication, GlusterFS changelog, Bareos backup filtering, LVM thin pool allocation, and EXT4 journal health all had issues that compounded. The operational complexity of a stacked filesystem setup means problems cascade; a stuck GFID broke glusterfind, which broke Bareos, while separately full thin pools caused EXT4 errors on the block layer. Diagnosing these required understanding the full stack, not just one layer.