in category:HomeLab program:Backup process:Troubleshoot ~ read.

Three Ways Bareos Backups Failed Without Anyone Noticing

Three distinct backup failure modes sat silently in the homelab for roughly three months, from January through May 2026, and none of them tripped a single alert. Bareos kept reporting jobs as “errored” or “cancelled” and nobody noticed until manual investigation turned up the gaps. The shared theme across all three is uncomfortable: backup infrastructure that fails quietly, with no level of observability into whether jobs are actually succeeding.

What Happened

The timeline breaks into three separate discoveries, each its own ticket, each found through manual digging rather than alerting.

January 2026: glusterfind path mismatch (KB#8054: Bareos backups hung)

The Bareos director runs on hub-andrewcz-net-bareos-ct0, a Fedora-based container. It uses glusterfind to enumerate files on GlusterFS volumes before backup. The way glusterfind works, it SSHes out to the Proxmox brick hosts (proxmini01, proxmini02, proxmini03) and invokes Python scripts at /usr/libexec/glusterfs/glusterfind/changelog.py and nodeagent.py.

The problem: those scripts live at /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind/ on Debian/Proxmox, not at the Fedora path. So every remote SSH call came back with “No such file or directory”, glusterfind silently produced empty filelists, and the empty filelists caused the Storage Daemon connection to drop with comm line to SD dropped. All 6 *-hub-volume-gluster incremental jobs cancelled at 03:00. The job-thepipeline-tv-nextcloud-mariadb job also errored around the same time.

On top of that, bareos-sd was sitting at 1.8GB memory from a known leak pattern; segfaults going back to April. That’s a separate issue (addressed by restarting bareos-sd) but it muddied the waters during investigation.

February 2026: wrong domain flag (KB#8114: MySQL Backups failed)

The bareos-add-resource.sh script was run with --domain curatingcollections instead of --domain curatingcollections-com for the lgtm-mariadb backup. That one missing -com created a client pointing to hub-curatingcollections-lgtm-mariadb-ct1… a nonexistent hostname. The real host is hub-curatingcollections-com-lgtm-mariadb-ct1.

Every scheduled run of job-curatingcollections-lgtm-mariadb errored out. 66 failed jobs going back to March 25, all silently accumulating in the catalog with nobody watching.

May 2026: missing MariaDB instances (KB#8163: Get all MariaDB instances added to Bareos backup jobs)

This one was the broadest. bareos-fd (the Bareos File Daemon) was installed on every container, but not all MariaDB instances were actually configured as clients in the Bareos director. MariaDB instances spread across six environments on the hub site, and coverage was inconsistent. An audit cross-referenced all MariaDB containers across proxmini01/02/03 against the director config and found three instances with zero backup coverage:

  • hub-andrewcz-net-lgtm-mariadb-ct1 (the entire andrewcz-net domain had zero database backups)
  • hub-andrewcz-org-ghost-mariadb-ct1
  • hub-andrewcz-org-vaultwarden-mariadb-ct1

During this same audit, the dangling curatingcollections (no -com) resources from KB#8114 got cleaned up as well.

Root Cause

Each failure had a different mechanism, but they all share one thing: the backup system had no way to tell us it was failing, and we had no way to hear it.

Issue 1 (glusterfind paths): GlusterFS ships glusterfind with a hardcoded path to its Python scripts, and that path differs between Fedora and Debian. The director container is Fedora-based; the brick hosts run Debian/Proxmox. When glusterfind SSHes from the Fedora host to a Debian host, it expects /usr/libexec/glusterfs/glusterfind/ but the scripts are at /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind/. The SSH call fails with “No such file or directory”, glusterfind swallows the error and produces an empty filelist, and the SD connection drops on the empty input. There’s no loud failure here… just quiet emptiness where file data should be.

Issue 2 (domain flag typo): bareos-add-resource.sh takes a --domain argument that gets interpolated into the client hostname. Passing curatingcollections instead of curatingcollections-com generated a config pointing at a host that doesn’t exist. Bareos dutifully tried to connect to hub-curatingcollections-lgtm-mariadb-ct1 every scheduled run, failed, logged the error, and moved on. 66 times. With no alerting on job failure state, those errors just sat in the catalog.

Issue 3 (missing clients): This is the most uncomfortable one. Installing bareos-fd on a container is necessary but not sufficient; the director also needs a client definition, a job definition, a schedule, and a pool. The bareos-add-resource.sh script handles all of that, but it has to be run deliberately for each instance. Three MariaDB containers were simply never added. No config, no job, no backup, no error… because from Bareos’s perspective, there was nothing to fail. The andrewcz-net domain is the worst case: an entire domain with zero database backups, and the only way we found out was a manual audit.

The Fix

Issue 1: symlinks on all three brick hosts

Created symlinks on proxmini01, proxmini02, and proxmini03:

ln -s /usr/lib/x86_64-linux-gnu/glusterfs/glusterfind /usr/libexec/glusterfs/glusterfind

After the symlinks were in place, kicked off Full backups for all 6 gluster jobs (job IDs 3629 through 3634). The first one (andrewcz-org) ran past the point where all previous attempts died, confirming the fix works.

Remaining work: The symlink fix is ad-hoc. A glusterfs-server package upgrade could wipe it. This needs to be folded into Pulumi provisioning so it survives package updates and host rebuilds.

Issue 2: cleanup and correct resources

Removed 4 config files from the director: - clients-curatingcollections.conf - jobs-curatingcollections-databases.conf - pools-curatingcollections.conf - fs-curatingcollections-databases.conf

Deleted from the Bareos catalog: 66 job records, client ID 16, pool ID 13, fileset ID 22. Verified no orphaned storage volumes. Reloaded bareos-dir after cleanup.

The correct job-curatingcollections-com-lgtm-mariadb was already in place (the -com resources were created correctly the first time); scheduled for first Full backup at 02:00.

Issue 3: add the three missing MariaDB clients

Ran bareos-add-resource.sh with the correct --domain flag for each missing instance. Verified director reload, client connectivity, and schedule entry for each. First Full backups at 02:00.

Current coverage after the fix: 20 clients (all ct1 instances):

  • andrewcz-com: firefly, ghost, kanboard, nextcloud, vaultwarden, lgtm
  • andrewcz-net: lgtm (NEW)
  • andrewcz-org: firefly, ghost (NEW), kanboard, nextcloud, vaultwarden (NEW)
  • curatingcollections-com: ghost, lgtm
  • cziryak-com: kanboard, lgtm, nextcloud, vaultwarden
  • thepipeline-tv: kanboard, nextcloud

What We Learned

  • Backups that fail silently are worse than no backups. With no backups, you know you have a gap. With silently failing backups, you think you’re covered… until you need to restore. The lack of alerting on job failure state is the real footgun here, not any individual misconfiguration.
  • Path differences between distros are a classic footgun. Fedora and Debian disagree on where glusterfind’s Python scripts live, and glusterfind doesn’t handle that gracefully. Any cross-distro SSH tool invocation needs to account for path differences; symlinks work but they’re fragile.
  • A typo in a domain flag produced 66 silent failures over weeks. The bareos-add-resource.sh script interpolates the domain into a hostname, and there’s no validation that the resulting hostname actually resolves. We should add a DNS/hostname check before the script writes config files.
  • Installing the file daemon is not the same as having a backup. bareos-fd on every container gave a false sense of coverage. The director config is what actually drives backups, and that requires deliberate per-instance setup. Going forward, coverage audits need to be periodic, not one-time.
  • The andrewcz-net domain had zero database backups and we only found out by auditing. That’s the kind of unknown unknowns that keep me up at night. If we hadn’t done the manual cross-reference, that gap could have persisted until a restore was needed and failed.
  • We need alerting on backup job failures. This is the single highest-impact change going forward. Bareos can emit job status; we just need something watching it. Without that, every fix here is a point solution that doesn’t address the root problem: we can’t see failures.
  • The bareos-sd memory leak is a known pattern and not yet fully resolved. Restarting it cleared the immediate 1.8GB issue, but the leak will recur. That’s a separate investigation.
  • The glusterfind symlink fix needs to survive package upgrades. Right now it’s a manual symlink that glusterfs-server upgrades could wipe. Folding it into Pulumi provisioning is the careful and prudent path.

Tickets