in category:HomeLab program:Vaultwarden program:MariaDB process:Architect ~ read.

How Vaultwarden Went From SQLite to Loadbalanced MySQL to Agent Secret Access

Vaultwarden started as a single SQLite container pointing at one instance at a time, and over the course of about eight months it evolved into a loadbalanced pair of containers backed by a replicated MariaDB cluster, pinned to a specific version to match upstream client requirements, and wired into a headless rbw agent so that Carter (the AI orchestrator running in a NixOS container) could pull secrets from a shared Bitwarden organization. This is the story of how that happened, told through the commits and tickets that did the work.

What Happened

The initial Vaultwarden deployment landed in October 2024 as part of a broader hardware and VLAN expansion (commit a4c1c45). The config was straightforward: a single container per environment, SQLite backend, GlusterFS mount at /srv/vaultwarden for the database file, and a hardcoded Argon2id admin token in the YAML config. The container_count for the andrewcz-org environment was set to 2, but with SQLite as the backend and no VIP, there was no real loadbalancing… both containers would have been serving the same SQLite file over GlusterFS, which is a recipe for corruption the moment both containers try to write at the same time.

The prod environment was even simpler: container_count: 1, single node, no redundancy. If that container went down, Vaultwarden was down.

This was fine for a personal password manager… until it wasn’t. The first ticket to acknowledge the problem was KB#7777: Move vaultwarden to mysql instead of sqlite, created in June 2025. The ticket has no description and no comments; it was just a note to self that SQLite wasn’t going to scale. It sat in the backlog for months.

The real push came in January 2026, when commit bb61eed landed with the subject PVE exporter; vaultwarden mariadb; mkForce substituters;. This commit did the heavy lifting of the MySQL migration:

  • Added a new role_vaultwarden-mariadb.yml config with container_count: 2, a VIP at .41, and MariaDB replication via MHA Manager (the standard replication pattern at the time)
  • Changed vaultwarden-backend.service.nix from dbBackend = "sqlite" to dbBackend = "mysql" with DATABASE_URL = "mysql://vaultwarden:{{ args['role_template_args']['db_user_pass'] }}@{{ args['first_two_octets'] }}.{{ args['vlan_id'] | int }}.41/vaultwarden"
  • Split vaultwarden/__main__.py into two components: mariadb (the database cluster) and backend (the Vaultwarden app), with the backend depending on the mariadb cluster’s config_deployments
  • Added firewall rules for MariaDB traffic between the backend containers and the database VIP
  • Generated random passwords for the database root user, the vaultwarden app user, the MHA replication user, the BareOS backup user, and the LGTM monitoring user… all via pulumi_random.RandomPassword

The backend config also changed from environment-specific container_count values to a flat default: 2, and the node_names were spread across all three Proxmox hosts (hub-proxmini01, hub-proxmini02, hub-proxmini03) so the two containers would land on different physical machines.

KB#8024: Use mysql vaultwarden tracked this work. The single comment on that ticket is a collection of research links: the NixOS Discourse vaultwarden thread, the NixOS wiki vaultwarden page, the mynixos vaultwarden package page, and logrotate documentation. No narrative, just the raw research that went into the migration.

At this point Vaultwarden was running as two containers behind Traefik’s loadbalancer, backed by a two-node MariaDB cluster with MHA Manager handling replication and failover. The VIP at .41 was the database endpoint; Traefik handled the HTTP loadbalancing across both backend containers via Consul service registration.

The MHA-to-Converge Migration

By May 2026, the MHA Manager pattern was being retired across the entire homelab in favor of a custom converge script. Commit da9ab49 migrated all MariaDB services… atom, firefly, ghost, kanboard, lgtm, nextcloud, and vaultwarden… from MHA Manager to the converge template. For Vaultwarden specifically, this meant:

  • vaultwarden/vaultwarden-mariadb.service.nix changed from a symlink to ../common_components/mariadb-mha.service.nix to a symlink to ../common_components/mariadb.service.nix
  • vaultwarden/__main__.py dropped pulumi_tls, subprocess, mha_replication_pass, mha_ssh_keys, and db_root_pass… all the MHA-specific plumbing… and replaced mha_replication_pass with replicator_pass
  • The commit removed the SSH firewall rules and /srv/mha mount points from the role configs
  • The converge script itself handled active-passive replication with deterministic election, split-brain detection via GTID comma-check, error 1451/1950 rebuild triggers, a local read_only guard, and ExecStartPre to remove stale master.info/relay-log.info

The converge script was a significant operational improvement over MHA Manager. MHA required SSH keys between nodes, a separate MHA Manager process, and manual intervention when things went sideways. The converge script was self-contained: it ran on every node, elected a primary deterministically, and could rebuild a broken replica by streaming a full mysqldump from the primary in seven steps. No SSH, no external manager, no manual failover.

Version Compatibility: KB#8113

KB#8113: Vaultwarden not compatible with new clients landed in February 2026 in the Breakfix swimlane. The ticket has no description and no comments, but the title tells the story: the Bitwarden desktop and mobile clients had updated, and the version of Vaultwarden running in the homelab wasn’t compatible with the new client API expectations.

This is a known issue with Vaultwarden… the upstream Bitwarden clients enforce a minimum server API version, and Vaultwarden (being an unofficial implementation) sometimes lags behind. When the clients update and start requiring newer API endpoints or response formats, older Vaultwarden versions stop working… logins fail, sync breaks, or specific features stop responding.

The fix was to pin Vaultwarden to a specific version that matched the upstream clients’ requirements. In the NixOS module, this is handled by the nixos_version config and the Nixpkgs version that ships a compatible vaultwarden package. The vaultwarden/__main__.py hardcodes nixos_version = "25.11" (commenting out the environment-config-driven version), which ensures the Vaultwarden package version is consistent across all containers and matched what the clients expected.

Carter’s Bitwarden Setup: KB#8144

KB#8144: Carter Bitwarden Setup came in April 2026 with the description: “This may be more cleaning, but I want to set up a bitwarden setup for Carter - IDK if that’s a group or what, but a way for each of us to give creds to carter, and for it to store some itself.” The single comment added: “This will also have to fit in my workstations, as well as anythingllm.”

The goal was to give Carter… the AI orchestrator running in a hosted NixOS container… access to secrets that Andrew could share with it. Instead of hardcoding API keys in config files or environment variables, Carter would use rbw, the unofficial Bitwarden CLI, to pull credentials from the Vaultwarden instance at runtime.

Commit db6e1a0 landed in June 2026 and implemented the full stack. The opencode container got:

  • A carter@{{ args['external_domain'] }} user in Vaultwarden, added to an organization so Andrew could share credentials with it
  • RBW_BASE_URL=https://vaultwarden.{{ args['external_domain'] }} and RBW_EMAIL=carter@{{ args['external_domain'] }} set as systemd environment variables on both the bootstrap service and the opencode service
  • A file-based secret-tool drop-in replacement at /etc/opencode/secret-tool that cats /etc/opencode/secrets/bitwarden-master-password… because the real secret-tool (libsecret) needs a GNOME keyring and D-Bus session, which doesn’t exist on a headless NixOS container
  • The master password deployed by Pulumi via age-encrypted secret (bitwarden_master_password in the Pulumi config, encrypted with Pulumi’s built-in encryption)
  • rbw and rbw-agent binaries downloaded by dotfiles-bootstrap and patched with patchelf --set-interpreter for NixOS FHS compatibility (NixOS doesn’t have /lib64/ld-linux-x86-64.so.2)
  • A symlink at /var/lib/opencode/.local/bin/secret-tool pointing to /etc/opencode/secret-tool so the file-based version shadows the system one in PATH
  • The carter.md agent config in the carter repo with "rbw *": "allow" in the bash permission rules, so Carter can run rbw get commands without prompting for approval

The flow works like this: when Carter needs an API key (say, the Kanboard credentials or the Ghost Admin API key), it runs rbw get 'Carter Kanboard' or rbw get --field 'Admin API key' 'Carter Ghost'. The rbw agent uses the file-based secret-tool to unlock with the master password from disk, connects to Vaultwarden at the configured base URL, and returns the credential. No hardcoded secrets in config files, no environment variable sprawl… just a Bitwarden organization with shared items that Carter can access.

The secret-tool replacement is worth calling out specifically. The comment in the NixOS config explains the reasoning:

On a desktop, secret-tool stores/looks up secrets in the GNOME keyring using key-value attribute pairs (e.g. application=rbw, type=master_password). The keyring is a general-purpose secure storage… any caller can store or look up any secret by its attributes.

This container is a single-purpose host that only needs one secret: the Bitwarden master password, used by rbw-pinentry-keyring to unlock the rbw agent so the carter wrapper can resolve API keys.

The password is already on disk at /etc/opencode/secrets/bitwarden-master-password (deployed by Pulumi via age-encrypted secret). No need to copy it elsewhere or parse key-value attribute pairs… just cat the source file directly. Accepts any arguments and completely disregards them.

If this host ever needs to store additional secrets, this script should be replaced with a proper attribute-pair-to-filename mapping. Until then, YAGNI.

That last line is the important one. The implementation is deliberately minimal because the container only needs one secret. When that changes, the script needs to evolve. But until then, YAGNI.

What We Learned

  • SQLite over GlusterFS is a footgun. The original deployment had two containers both writing to the same SQLite file on a GlusterFS mount. This works… until both containers write at the same time, at which point you get corruption. SQLite needs a single writer, and a shared filesystem doesn’t give you that. Moving to MySQL with a VIP was the right call, not just for scale but for data integrity.
  • MHA Manager was more operational complexity than it was worth. SSH keys between nodes, a separate manager process, manual failover procedures… the converge script replaced all of that with a self-contained systemd service that handles election, replication, and rebuilds automatically. The migration touched seven services at once, which was a lot of surface area, but the end state is significantly simpler to operate.
  • Version compatibility between Vaultwarden and Bitwarden clients is an ongoing concern. Vaultwarden is an unofficial implementation, and when the upstream clients update, older Vaultwarden versions break. Pinning the NixOS version (and therefore the Vaultwarden package version) is the mitigation, but it means the version pin needs to be reviewed every time the clients update.
  • Headless rbw on NixOS requires a custom secret-tool. The standard secret-tool (libsecret) needs a GNOME keyring and D-Bus session, neither of which exist on a headless NixOS container. The file-based replacement is a pragmatic workaround… it cats the master password from disk, which is deployed by Pulumi via age-encrypted secret. It’s not a general-purpose secret store, but it doesn’t need to be. YAGNI.
  • patchelf is mandatory for pre-built binaries on NixOS. rbw, rbw-agent, uv, and several other binaries downloaded by dotfiles-bootstrap expect /lib64/ld-linux-x86-64.so.2 as the dynamic linker, which doesn’t exist on NixOS. The bootstrap script runs patchelf --set-interpreter on every known binary after each download pass, and it runs three passes because each pass may download new binaries that need patching.
  • Sharing secrets via Bitwarden organizations is cleaner than config files. Instead of hardcoding API keys in NixOS configs or environment variables, Carter pulls credentials from Vaultwarden at runtime via rbw. Andrew can rotate a credential in Vaultwarden and Carter picks it up on the next rbw sync. No config changes, no redeployments.

Tickets