Firefly III, Redis, and the Long Road to Load-Balanced PHP on NixOS

I run Firefly III in my homelab as my personal finance manager. It’s a PHP/Laravel app, and like most of my services it lives on NixOS LXC containers behind Traefik with Consul service discovery, load-balanced across two containers (ct0 and ct1). On paper that’s a clean HA setup. In practice, getting a PHP application to actually behave across load-balanced instances turned out to be a whole journey… one that touched Redis sessions, OAuth key sharing, PHP extension packaging, and a handful of footguns along the way.

This is a retrospective of that work, spread across a handful of tickets and commits over roughly a year. I’m writing it up because the pattern that emerged is reusable, and because the individual problems were the kind of thing that’s easy to miss until you’re staring at a broken login screen wondering why your session evaporated between requests.

The separation: Firefly and the importer

Firefly III has a companion service called the Data Importer, which handles pulling transactions from bank APIs and pushing them into Firefly’s backend. Originally I had both running on the same containers, which is fine for a single-instance setup but doesn’t fit the paradigm I want to use going forward, where each service scales and deploys independently.

In April 2025 I split them. Commit 8b94aef created firefly-importer.service.nix as its own service with its own containers, its own nginx vhost, its own Consul registration, and a unison sync for the data directories. The importer talks to the Firefly backend via a personal access token, which I stored as a Pulumi secret.

That split introduced a small but persistent annoyance: the importer got uid=9013 and gid=9014, whereas Firefly itself uses 9013 for both. Mismatched uid and gid numbers. I noted it on KB#8090: Firefly backend does not share access tokens across instances later (“I hate that firefly-importer has uid=9013 and gid=9014… mis-matched numbers”) and eventually fixed the mount owner in commit 915f3a1, but it’s the kind of thing that sits there quietly causing permission errors until you track it down.

The borked config: when Nix store paths go stale

Before the HA work really started, I hit a more fundamental problem (KB#7781: fix borked firefly and NC in andrewcz-org, November 2025). Firefly crashed with an error about a missing directory at a nix store path for storage/logs:

UnexpectedValueException
There is no existing directory at "/nix/store/b86sn1pix3rkv42hv8mk2kddyl8m1nqn-firefly-iii-6.2.10/storage/logs" and it could not be created: Read-only file system

The NixOS filesystem is read-only by design, so the app couldn’t create its log directory… but the real issue was that a cached config file was pointing at a stale nix store path from a previous generation. The cached config.php had hardcoded paths like /nix/store/b86sn1pix3rkv42hv8mk2kddyl8m1nqn-firefly-iii-6.2.10/storage/logs/laravel.log from an older package version. Removing the cached config and restarting firefly-iii-setup fixed it.

That one was a good reminder of how NixOS’s immutability interacts with PHP apps that expect to write config caches. I noted at the time that I’d want to handle this better, especially vis-a-vis the shared nature of the setup I was building toward. A cached config pointing at a path that no longer exists is a footgun that’ll fire every time you do a generation switch and the store hash changes.

Here’s the core issue, and it’s not specific to Firefly. PHP stores sessions on the local filesystem by default. When you load-balance across two containers, a request that lands on ct0 creates a session file on ct0; the next request might land on ct1, which has no idea that session exists. The user gets logged out. Or worse, they get half-logged-in and the app behaves unpredictably.

The fix is to move session storage into a shared backend. Redis is the obvious choice; it’s fast, it’s already in the homelab, and PHP has native support for it via the redis session save handler.

I established the pattern with Nextcloud first (commit c04ba91, February 2026). The key bit of NixOS config lives in the phpfpm pool:

services.phpfpm.pools.nextcloud.phpOptions = ''
  session.save_handler = redis
  session.save_path = "tcp://<redis-vip>:6379?auth=<pass>&database=1"
'';

That’s it, really. Point session.save_handler at redis, give it a save_path that includes the Redis VIP, the auth password, and a database number. Once that’s in place, sessions live in Redis and both containers see the same session state.

This became the template. Firefly got the same treatment later, along with SESSION_DRIVER = "redis" and CACHE_DRIVER = "redis" in its environment settings.

The Redis HA journey: Sentinel, keepalived, and the health check script

Of course, “point sessions at the Redis VIP” only works if the Redis VIP is actually up and pointing at a healthy master. I run Redis with Sentinel plus keepalived for VIP failover, which is a pattern I use across the homelab (MariaDB uses the same approach with a converge script).

The Redis health check script went through two iterations, and both are worth noting because they’re very NixOS-specific footguns.

The first version (commit 83ef9ef, December 2025) added the keepalived health check script. It queries Sentinel for the master address, compares it to the local IP, and exits 0 or 1 accordingly. Straightforward idea. But it used #!/usr/bin/env bash as the shebang, which… doesn’t exist on NixOS the way you’d expect. NixOS doesn’t have a standard /usr/bin/env. Scripts need to reference the full Nix store path.

The second version (commit b55ebe1, March 2026) fixed this properly. The shebang became #!${pkgs.bash}/bin/bash, and every binary the script called got a full package path: ${pkgs.coreutils}/bin/head, ${pkgs.coreutils}/bin/cut, ${pkgs.gnugrep}/bin/grep, and so on. No PATH resolution, no ambiguity.

The master detection logic also changed. The first version compared a single local IP to the Sentinel-reported master. The second version checks whether the master IP appears anywhere in the full list of local addresses, which handles the case where a container has multiple IPs (it does, because of the keepalived VIP itself). I also added reloadTriggers for the check script so keepalived picks up changes to it.

This is the kind of thing that’s notoriously easy to get wrong, and you only find out when failover doesn’t happen during an actual outage. We should be careful and prudent with health check scripts; they’re the component that’s supposed to save you, and a broken save-you component is worse than no failover at all because it gives false confidence.

KB#8090: Firefly backend does not share access tokens across instances (February 2026). The description was blunt: “Right now it’s only stored in one. I’m not sure what’s going on with that.”

What was going on: Firefly uses Laravel Passport for OAuth, and Passport generates its own RSA keypair on first run. Each container generated its own keys. So a token issued by ct0 wasn’t valid on ct1, and vice versa. The importer (or any OAuth client) would authenticate against one instance, then the load balancer would route the next request to the other instance, and the token would be rejected.

The workaround, which I noted on the ticket in February, was to shut off one of the instances entirely. That works, but it defeats the entire purpose of running two containers. It’s a moot point as an HA strategy.

The fix (commit eaa3a16, March 2026) was to generate a shared RSA keypair in Pulumi and deploy it to both Firefly backend containers. Specifically:

A Pulumi TLS PrivateKey (RSA 4096) for Firefly Passport
firefly_passport_private_key and firefly_passport_public_key passed to both containers
PASSPORT_PRIVATE_KEY and PASSPORT_PUBLIC_KEY added to Firefly’s environment settings

Once both instances share the same keypair, tokens issued by either are valid on both. The load balancer can route freely. I closed the ticket comment in April with “This ended up being a php setting. Fixed.”… which is technically true, in the sense that the environment variables are PHP config, but the real insight was that the keys needed to be shared infrastructure, not per-instance generated state.

The Redis database split: don’t let services step on each other

While I was in there, I noticed Firefly backend and the importer were both pointing at the same Redis instance with the same database numbers. That’s a problem because cache keys and session keys can collide, and even when they don’t, you lose the ability to reason about which service owns which keys.

The split (same commit, eaa3a16):

Firefly backend: REDIS_DB="0", REDIS_CACHE_DB="1"
Importer: REDIS_DB="2", REDIS_CACHE_DB="3"

Clean separation. Each service gets its own database number for both app data and cache. Going forward this is the pattern I’ll use for any shared Redis instance: assign explicit database numbers per service and document them.

The PHP redis extension: NixOS modules don’t include it by default

Here’s a subtle one. The NixOS Firefly III module’s PHP package didn’t include the redis extension by default. So even after setting SESSION_DRIVER = "redis", PHP couldn’t actually talk to Redis because the extension wasn’t loaded.

The fix was overriding phpPackage in the phpfpm pool with pkgs.php.buildEnv, adding the redis extension:

services.phpfpm.pools.firefly-iii.phpPackage = lib.mkForce (pkgs.php.buildEnv {
  extensions = ({ enabled, all }: enabled ++ (with all; [
    redis
  ]));
});

I marked this with a TODO to upstream it. The NixOS Firefly III module should probably include redis as an optional extension, or at least document that you need it if you’re using Redis for sessions. Especially since the module’s own options let you configure a Redis connection, but the PHP runtime can’t actually use it without the extension. That’s a gap.

I also added php-fpm error logging at the same time (php_admin_value[error_log] and php_admin_value[log_errors]), because without it I was flying blind on PHP errors. You need a level of observability into the application layer, not just the container and service level.

The php-fpm exporter ordering: a monitoring stack footgun

A smaller but real problem (commit 7d2cc28, May 2026): the Prometheus php-fpm exporter was starting before the phpfpm pool socket existed, so it would fail on boot. The exporter sets up a mount namespace that needs /run/phpfpm to exist first.

The fix was adding after and wants dependencies on the specific phpfpm pool service, so the exporter waits for the pool to be up before it starts. This affected firefly-backend, firefly-importer, kanboard, and nextcloud… basically every PHP service I run.

It’s a systemd ordering issue, not a PHP issue, but it’s the kind of thing that bites you when you’re running multiple services that depend on the same runtime infrastructure. The exporter doesn’t know which pool it’s monitoring; it just needs the socket directory to exist.

The importer flush issue and the ongoing Redis confusion

Two loose ends worth mentioning.

KB#7774: firefly-importer does not flush correctly (June 2025): the importer was dumping to the OAuth token screen instead of flushing correctly. This was related to the same class of problems: the importer needed proper Redis configuration and a valid personal access token to communicate with the Firefly backend. Once the token sharing and Redis session work landed, this resolved as part of the larger fix.

KB#8129: Firefly nightly imports of transactions: I noticed at one point that Redis was pointing at the Nextcloud Redis VIP instead of the Firefly Redis VIP, and was taking the Nextcloud Redis VIP’s address too. This is a configuration confusion issue across the multiple Redis instances in the homelab. When you have several Redis Sentinels running, each with their own VIP, it’s easy to wire a service to the wrong one… and the symptom is subtle, because it’ll mostly work (Redis is Redis), until you realize your sessions are living in the wrong cluster. I’m unsure as to if this is fully resolved or just currently working; it needs a more careful audit of the Redis VIP assignments.

What the journey taught me

The pattern that emerged is straightforward, even if the path to it wasn’t:

Sessions in Redis, not on disk. Override phpOptions in the phpfpm pool to set session.save_handler = redis and point session.save_path at the Redis VIP. This is the single most important change for load-balanced PHP.
Shared secrets, not per-instance secrets. OAuth keypairs, app keys, anything that needs to be valid across instances has to be generated once and deployed to all containers. Pulumi is a good place for this.
Explicit Redis database numbers per service. Don’t let services share a database number; assign and document them.
The PHP redis extension is not automatic. Check that your phpfpm pool’s phpPackage actually includes it.
Health check scripts need full NixOS package paths. No #!/usr/bin/env bash. Every binary gets a ${pkgs.*}/bin/ prefix.
Watch your systemd ordering. Exporters and dependents need after and wants on the thing they’re observing.

The broader lesson is that running a PHP app in HA on NixOS is a series of small, interconnected configuration problems. None of them are hard individually, but they’re unknown unknowns until you hit them; the app works fine on a single instance, and the failure mode under load balancing is often subtle (sessions that half-work, tokens that work sometimes, caches that collide occasionally). You need to be methodical, and you need enough observability to notice when something is off before it becomes a user-facing problem.

Firefly III is now running across both containers with shared sessions, shared OAuth keys, and separate Redis databases. It’s good enough for now. The remaining work is the Redis VIP audit and upstreaming the PHP redis extension to the NixOS module, both of which are on the list but not urgent. The setup works, and more importantly, I understand why it works… which is the part that matters when it inevitably breaks again.