Stabilizing Gramps Web: From SQLite Corruption to Half the RAM

Running Gramps Web in a homelab sounded straightforward… a genealogy app with a Docker image, what could go wrong. Over the course of eight months across four tickets, we dealt with SQLite database corruption, Celery workers losing their Redis broker, 500 errors on login, and containers eating 50% of their RAM budget because we left debug mode on. Here’s the full story.

What Happened

We first deployed Gramps Web in November 2025, tracking the work in KB#7983: Gramps Web. The initial setup used SQLite as the database backend, which is the default for Gramps Web. Getting trees set up required a manual bootstrap process:

$ mkdir /home/grampsdb/f9811c2d-37b7-43cb-8b07-7f1fe6741093
$ echo 'sqlite' > /home/grampsdb/f9811c2d-37b7-43cb-8b07-7f1fe6741093/database.txt
$ echo 'new' > /home/grampsdb/f9811c2d-37b7-43cb-8b07-7f1fe6741093/name.txt
$ podman exec -it gramps bash
> $ python3 -m gramps_webapi --config /app/config/config.cfg user add <username> <password> --role 5 --tree f9811c2d-37b7-43cb-8b07-7f1fe6741093
$ curl -X POST "http://localhost:5000/api/token/" -H "Content-Type: application/json" -d '{"username":"<username>","password":"<password>"}'
$ curl -X POST "http://localhost:5000/api/trees/" -H "Content-Type: application/json" -H "Authorization: Bearer <access-token>" -d '{"name": "<tree-name>"}'

Then we navigated to https://<url>/firstrun/<tree-uuid> for owner admin creation. Pretty standard for a fresh Gramps Web install.

However, things went sideways quickly. The search indexer started throwing errors in the Celery worker logs:

celery.utils.serialization.UnpickleableExceptionWrapper: HandleError()

This came from gramps/gen/db/generic.py line 1461: raise HandleError(f"Handle {handle} not found") during the update_search_indices_from_transaction task. The Celery worker tried to update search indices for a database handle that didn’t exist, which suggests the SQLite database and the search index got out of sync.

Then we hit the real problem:

sqlite3.DatabaseError: database disk image is malformed

This error came from sifts/core.py line 469 during conn.commit() in the search indexer. The SQLite database itself was corrupted. We found a relevant GitHub issue tracking similar problems, and the Gramps Web docs pointed us toward PostgreSQL as a more robust backend and multi-tree setups. The Gramps PostgreSQL addon wiki page also had useful context.

SQLite corruption in a containerized environment is a known footgun. SQLite uses file-level locking, and if the container filesystem doesn’t handle locks correctly, or if the process gets killed mid-write, you can end up with a malformed database. Especially since we were running Celery workers that concurrently accessed the same SQLite file, which is not a pattern SQLite handles gracefully under load.

We moved past the initial SQLite issues and got Gramps Web running across two environments: hub-andrewcz-org and hub-curatingcollections-com, each with two backend containers for HA via Consul + Traefik load balancing.

A critical piece of the multi-container setup was getting Redis configured correctly. Gramps Web uses Celery for background tasks (search indexing, reports, exports, imports) and Flask-Limiter for rate limiting. Out of the box, Gramps Web defaults to in-memory for both… CELERY_CONFIG is an empty dict {} in the default config, which means Celery falls back to its built-in default broker URL of memory://. The rate limiter similarly defaults to memory:// storage.

That’s fine for a single-process dev setup. But in a multi-container deployment with separate grampsweb and gramps-celery containers, in-memory means the web container can’t dispatch tasks to the worker container… there’s no shared broker. And each Gunicorn worker gets its own independent rate limit counter, so a “10 per minute” limit effectively becomes “10 per minute per worker.”

We had to explicitly configure three environment variables in gramps-backend.service.nix to point everything at our shared Redis VIP (.35):

GRAMPSWEB_RATELIMIT_STORAGE_URI = "redis://:<password>@<vip>.35";
GRAMPSWEB_CELERY_CONFIG__broker_url = "redis://:<password>@<vip>.35/0";
GRAMPSWEB_CELERY_CONFIG__result_backend = "redis://:<password>@<vip>.35/0";

The GRAMPSWEB_CELERY_CONFIG__broker_url tells Celery where to send tasks. The GRAMPSWEB_CELERY_CONFIG__result_backend tells Celery where to store task results (so the web container can check if an export or import finished). The GRAMPSWEB_RATELIMIT_STORAGE_URI makes rate limiting shared across all workers. Without all three, the multi-container HA setup simply doesn’t work.

Then in January 2026, we started seeing Celery workers lose connection to that Redis broker, tracked in KB#8057: Gramps redis. The container in question was hub-curatingcollections-com-gramps-backend-ct1. The logs showed:

redis.exceptions.ConnectionError: Error while reading from 172.16.9.35:6379 : (104, 'Connection reset by peer')

and

redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

The pattern was consistent: Celery would connect to Redis, run for a while, then lose the connection with ConnectionResetError: [Errno 104] Connection reset by peer. It would auto-reconnect, but the cycle repeated. We also saw clock drift warnings:

WARNING/MainProcess] Substantial drift from celery@1a01e1d2fdfc may mean clocks are out of sync. Current drift is 22 seconds.

and

WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...

The missed heartbeats from the peer Celery worker celery@1a01e1d2fdfc accumulated over hours. I noted at the time: “it’s like starting it at the same time fails, one of them needs to restart, and then it’s okay.” Which pointed toward a race condition when both backend containers started simultaneously and both tried to establish Celery connections to the same Redis instance.

This Redis issue cascaded into KB#8015: gramps 500’d on login, where Gramps returned 500 errors on login because Redis was not in a master state. The description was terse: “redis not master - keepalive info”. When Redis isn’t in a master state, Celery can’t dispatch tasks, and the Gramps Web API depends on Celery for search indexing and other background work. So login fails because the task queue is broken.

The fix for both Redis tickets turned out to be simpler than the diagnosis. A restart cleared the issue. I also noted a workaround: mv /home/gramps/home/gramps/recent-files-gramps.xml . then systemctl restart podman-gramps. The recent-files-gramps.xml file might have been causing weirdness due to the load balancing; both containers sharing state that wasn’t designed to be shared. After the restart, we had to re-index from the primary owner’s administration tab to get search back to a working state.

Then in July 2026, we noticed all four Gramps backend containers were running at approximately 50% RAM, the highest in the cluster by percentage. This became KB#8206: Remediate high RAM on Gramps backend containers. The four containers:

• hub-andrewcz-org-gramps-backend-ct0 (VMID 268, proxmini01)
• hub-andrewcz-org-gramps-backend-ct1 (VMID 269, proxmini02)
• hub-curatingcollections-com-gramps-backend-ct0 (VMID 271, proxmini02)
• hub-curatingcollections-com-gramps-backend-ct1 (VMID 270, proxmini03)

Each had 2 cores and was running Gramps with 8 Gunicorn workers in development/debug mode.

Root Cause

The SQLite corruption in KB#7983 came from running SQLite in a containerized environment with concurrent access from Celery workers. SQLite uses file-level locking and is not designed for concurrent write access from multiple processes. When the Celery search indexer and the Gramps Web API both tried to write to the same SQLite database file, and the container filesystem didn’t handle locks correctly (or a process got killed mid-write), the database could end up malformed. The database disk image is malformed error is SQLite’s way of saying the file structure is corrupt, not that the disk itself is bad.

The HandleError() from gramps/gen/db/generic.py was a secondary symptom; the search indexer tried to update indices for handles that no longer existed in the corrupted database, which means the database and the search index had diverged.

The Redis/Celery connection issues in KB#8057 and KB#8015 were related to both containers starting simultaneously and racing to establish Celery connections to the same Redis broker. When both Celery workers connected at the same time, one would end up in a state where it couldn’t maintain its connection. The 22-second clock drift between workers compounded the problem; Celery uses heartbeats and timestamps to track worker health, and significant clock drift causes the broker to think a worker is unresponsive.

The “redis not master” state in KB#8015 was likely a side effect of the connection instability. When Redis loses enough client connections or gets into a degraded state, it can drop out of master mode, especially in a Redis Sentinel or clustered setup. Without a master, Celery can’t enqueue or dequeue tasks, and the Gramps Web API returns 500 on any endpoint that depends on async work.

The high RAM in KB#8206 had a clear and embarrassing root cause: we left debug mode on. All four containers were running with:

• FLASK_DEBUG=1
• FLASK_ENV=development
• GRAMPSWEB_DEBUG=1
• gunicorn --reload

The --reload flag tells Gunicorn to watch for file changes and auto-reload workers, which is useful in development but pointless in production. Each Gunicorn worker loads the full Python/GTK/Gramps stack at approximately 250MB per process. With 8 workers, that’s roughly 2GB before counting Celery (approximately 700MB) and infrastructure services. No swap was configured on any of the containers.

The standard guidance for Gunicorn workers is (2 * cores) + 1. With 2 cores, that’s 5 workers. But for a low-traffic genealogy web app, 3 workers is plenty. We were running 8, which is 3x what we needed even by the standard formula, and 2.67x what made sense for this workload.

The Fix

For the SQLite corruption in KB#7983, the long-term fix was moving away from SQLite for production use. The Gramps Web docs recommend PostgreSQL for multi-user or production deployments. SQLite is fine for testing but not for concurrent access patterns. We also had to re-index search from the primary owner’s administration tab after recovering the database.

For the Redis/Celery issues in KB#8057 and KB#8015, the immediate fix was a restart. Both containers were restarted, which cleared the stale connection state and let Redis return to master mode. We also moved the recent-files-gramps.xml file out of the way, since the load-balanced setup meant both containers were reading a shared state file that wasn’t designed for that pattern.

Going forward, the operational fix is to not start both backend containers simultaneously. Staggering container starts avoids the race condition where both Celery workers try to establish connections to the same Redis broker at the same time.

For the high RAM in KB#8206, we made the following changes in gramps-backend.service.nix:

• Gunicorn workers: 8 to 3
• Removed --reload flag from the Gunicorn command
• FLASK_DEBUG: 1 to 0
• FLASK_ENV: development to production
• GRAMPSWEB_DEBUG: 1 to 0
• GRAMPSWEB_ENV: development to production
• GRAMPSWEB_LOG_LEVEL: DEBUG to INFO
• Celery concurrency: left at 2 (no change needed)

The reasoning: 3 Gunicorn workers is more than enough for a low-traffic genealogy app. Removing --reload eliminates the file watcher process. Switching from debug to production mode disables the Flask debugger and the reloader, which both consume memory and CPU. Dropping the log level from DEBUG to INFO reduces I/O and memory pressure from log buffering.

Since HA is via Consul + Traefik load balancing (no VIP/Keepalived), we restarted containers one at a time with 4-5 minute stabilization windows between each. This follows our container start protocol for proxmini02 recovery, which mandates starting one container, waiting for stabilization, checking host resources, and only then starting the next.

The results, verified after all four containers were restarted:

All four dropped below the 35% target. That’s approximately a 50% RAM reduction per container, just from turning off debug mode and reducing worker count. Commit: 4aac6df.

What We Learned

• SQLite in containers with concurrent access is a footgun. If you’re running Celery workers that write to the same SQLite database as the web API, you’re asking for corruption. Move to PostgreSQL for anything beyond testing.
• Gramps Web defaults to in-memory for Celery and rate limiting. In a multi-container deployment, you must explicitly set GRAMPSWEB_CELERY_CONFIG__broker_url, GRAMPSWEB_CELERY_CONFIG__result_backend, and GRAMPSWEB_RATELIMIT_STORAGE_URI to point at a shared Redis instance. Without these, tasks can’t reach the worker container and rate limits are per-worker instead of shared.
• Debug mode in production is a silent resource killer. FLASK_DEBUG=1 and gunicorn --reload are development tools. Leaving them on in production doubles your memory footprint and adds unnecessary file watcher processes. Always verify your environment variables before deploying.
• The standard Gunicorn worker formula is a starting point, not a rule. (2 * cores) + 1 gives you 5 workers on a 2-core container, but for a low-traffic app, 3 is plenty. More workers means more memory for no benefit when traffic doesn’t justify them.
• Stagger container starts when services share a broker. Starting two Celery workers simultaneously against the same Redis instance can cause connection races. A few seconds of stagger avoids the problem entirely.
• Clock drift matters for distributed task queues. A 22-second drift between Celery workers caused the broker to think a worker was unresponsive. NTP or chrony should be running on all containers that participate in distributed task processing.
• Shared state files don’t work with load balancing. The recent-files-gramps.xml file assumed a single-instance deployment. When two containers share a filesystem and both read/write the same state file, you get weird behavior. Either use a shared storage backend designed for concurrency, or make each container’s state independent.
• Restarting one at a time with stabilization windows is the right pattern for HA setups. The 4-5 minute window between container restarts let us verify each container came back healthy before moving to the next, and kept the service available throughout the change.
• We need a level of observability into container resource usage that catches debug mode early. Finding out eight months later that containers were at 50% RAM because of debug mode means we weren’t looking. A simple alert on RAM percentage would have caught this in January, not July.

Tickets

• KB#7983: Gramps Web
• KB#8015: gramps 500’d on login
• KB#8057: Gramps redis
• KB#8206: Remediate high RAM on Gramps backend containers