From Zabbix to LGTM (and friends)

Over 18 months, the homelab observability stack went from a single Zabbix instance to a full LGTM pipeline with metrics, logs, and network flow visibility across 22+ services and 6 datacenters. Twelve tickets, a few inode exhaustion events, one six-iteration debugging saga into NetFlow timeout semantics, and a lot of bespoke Nix packaging later… here’s how it all came together.

What Happened

It started with Zabbix. Back in October 2024, KB#6746: Monitoring opened as the umbrella ticket for everything observability-related in the homelab. The original scope was broad: monitor abstracted endpoints via synthetic checks, MySQL replication lag, clustering quorum, OS and app versions, GPU, network traffic, wireguard, SNMP, power usage. Zabbix was the initial choice, and I spent time fighting three MySQL MHA issues before even getting to the monitoring part: /var/lib/mysql permissions not being group-readable with tmpfiles, mha-manager not restarting when mysql restarts, and masterha_master_switch systemd unit timing. We fixed those via a mysql-post-start service with execStartPre chmod and systemd service dependencies… but the more I worked with Zabbix, the more I realized it didn’t fit the paradigm I wanted to use going forward.

Zabbix is a polling-based, agent-driven monitoring system. It’s good at “is this thing up and what’s its CPU load” but it’s not great at the level of observability I was after: high-cardinality metrics, log aggregation, flexible dashboards, and a pipeline architecture where I could ship logs and metrics through a common transform layer. The operational complexity of maintaining Zabbix agents, Zabbix server, Zabbix frontend, and a separate MariaDB backend for Zabbix’s own data… it was a lot of moving parts for what I was getting out of it. So KB#6746 eventually closed with 16 follow-up tasks enumerated, and those follow-ups became the tickets that drove the rest of this story.

The replacement was a LGTM stack (Loki, Grafana, Tempo, Mimir). On December 15, 2025, commit da10023 created the lgtm/ directory with 1294 insertions: Grafana for dashboards, Loki for log aggregation, Mimir for metrics storage, and Prometheus for scraping. Commit b2acb53 refined the Mimir and Loki configs the same day. This was the foundation; everything else built on top of it.

Two weeks later, commit ae149d6 (December 28) plumbed the LGTM stack into all 6 environment configs (55 files, 2382 insertions), set up nix-serve on the cache server for precompiled Grafana plugins with pubkey auth, and began removing Zabbix. We also stripped out the grafana.com/googleapis telemetry (commit c57e02f) and disabled analytics reporting in LGTM (commit e75c14c) the same day. I don’t want telemetry from my own monitoring stack phoning home to anyone.

Then on December 30, commit dc063b5 fully decommissioned Zabbix: 17 files, 426 deletions. The entire zabbix/ directory, the Zabbix MariaDB backend, all Pulumi configs… gone. KB#8032: decom Zabbix was a clean break. Services had their Zabbix-related config stripped, and that was that.

With Zabbix gone, the next phase was getting metrics on everything. KB#8031: get metrics on all services was the checklist ticket. The exporter list: systemd exporter (replacing node-exporter), php-fpm exporter, MariaDB exporter, Redis exporter, Nextcloud exporter, and PVE exporter. Baselines needed for firefly, ghost, gramps, kanboard, lgtm, nextcloud, postfix, scaffold, searxng, and vaultwarden.

This played out across several commits. Commit 42f1883 (December 30) added the MariaDB exporter (144 insertions to mariadb-mha.service.nix), Redis exporter (139 insertions to redis.service.nix), and Traefik metrics (200 insertions). Commit bb61eed (January 10) added the PVE exporter to the LGTM stack. Commit 0eed58a (January 27) was the big one: systemd monitoring across all 18 service containers, 2100 insertions. Commit f50cf5a (February 18) fixed the Nextcloud exporter port. And commit 7d2cc28 (May 18) fixed php-fpm exporter startup ordering… we had to add after and wants dependencies on the php-fpm pool service so /run/phpfpm exists before the exporter sets up its mount namespace. This affected firefly-backend, firefly-importer, kanboard, and nextcloud.

I noted in the ticket comments: “want to split this into separate tix, after getting everything set up” and “need to remove the nginx where nginx doesn’t need to live before I consider the first base pass completed.” The scope was intentionally broad; we’d refine it as we went.

Metrics were half the picture. The other half was logs. KB#7697: Log management and retention and searching and alerting had been open since May 2025 with a single comment from me: “grafana + vector does this - already implemented. Alerting is separate, but I dont’ want to alert directly on logs, so I’ll be building dashboards and stuff for that.”

The “already implemented” part was optimistic at the time, but the actual rollout came with commit 896646b (February 17, 2026): Vector across all services, 2845 insertions across 21 files. The pipeline was journald to Vector to Loki, with component_name and app_name labels on every log stream. We also added logrotate for Vector’s own logs and switched the LGTM stack’s object storage backend from MinIO to Garage (which I was already running for other things; no point maintaining two S3-compatible stores).

KB#8074: Optimize logging on all the services refined the Vector rollout. Commit 730bccf (February 25) created a dedicated vector user and group, added a proper logrotate config with su = "vector vector" and path = "/var/log/vector/{{unit}}-%Y-%m-%d.log", and registered the Vector service with Consul. Commit da9ab49 (May 22) standardized the prune_journald_metadata transform across all 22+ services. And commit 6a81e9a (April 25) removed debug file sinks from the GoFlow2 Vector pipeline that were writing ~266 MB of logs to disk… more on that later.

Retention was its own ticket. KB#8030: Customize retention in days for all envs had no comments but a clear trail of commits. Commit ae149d6 (December 28) introduced the lgtm_retention variable. Commit d2f66ba (January 29) plumbed it through all 6 environment configs. Commit 691e56b (January 30) set Prometheus to agent mode with --storage.agent.retention.max-time=48h on all 13 service containers. Agent mode is important here: in agent mode, Prometheus doesn’t store data long-term; it scrapes and forwards to a remote write endpoint (Mimir in our case). So 48h of local retention is just a buffer, not the actual retention window. The real retention lives in Mimir.

Commit 896646b (February 17) added the Mimir retention config: retention_period = "{{ args['role_template_args']['lgtm_retention'] }}", retention_enabled = true, retention_delete_delay = "2h", retention_delete_worker_count = 4. Also bumped Prometheus retention to 72h in the converge template to give a bit more buffer.

Some exporters didn’t have Nix packages, so I had to build them from source. KB#8018: Build community exporters from source covered consul_exporter, MinIO metrics, and a few others. Commit 68f6861 (January 2) created the prometheus-exporter-gramps-web Nix package from GitLab source. Commit cd76749 (April 20) created the ntopng-exporter Nix package from GitHub (aauren/ntopng-exporter v1.4.0). Commit 3e5b8c8 (May 27) reorganized everything into multi-version channels and removed ntopng-exporter (superseded by GoFlow2, which I’ll get to). And commit ae149d6 (December 28) set up nix-serve on the cache server for precompiled Grafana plugins with pubkey auth, so I wasn’t rebuilding Grafana plugins from source on every container.

Carter noted in the ticket: “Rolled into #8169. Community exporters are a subtask in the package build ticket.” The work naturally folded into the broader package build effort.

Now we get to the deep technical section. KB#8143: GoFlow2/NetFlow bandwidth metrics for home network is the best detective story in this whole retro. The goal was network flow visibility: I wanted to see bandwidth usage per source/destination/protocol across the home network, not just aggregate interface counters.

The architecture: pfSense running softflowd exports IPFIX/NetFlow records on port 2055 to GoFlow2, which converts them to JSON and writes to journald. Vector reads from journald, parses the flow records, aggregates them, converts to metrics, and ships to Mimir. A branch also sends raw flow records to Loki for forensics.

Getting this pipeline to produce accurate bandwidth graphs took six iterations.

Iteration 1: Counter values wrong. The metric goflow2_flow_bytes_total showed tiny values like 1.0 when downloading 100MB files. The problem was in Vector’s log_to_metric transform: it was missing increment_by_value = true. Without that flag, the counter increments by 1 for each matching log line instead of summing the field values. So every flow record, regardless of whether it carried 100 bytes or 100 megabytes, just bumped the counter by 1. Adding increment_by_value = true told Vector to use the actual byte count from the flow record as the increment amount.

Iteration 2: Rate calculations returning empty or tiny values. rate(goflow2_flow_bytes_total[5m]) was returning empty results or values like 120 KB/s when the actual throughput was 8-10 MB/s. I pulled raw GoFlow2 logs from journald to see what was actually arriving:

sudo journalctl -u goflow2 --since "5 minutes ago" --no-pager | grep "5.161.7.195"

What I found: flow records were being exported 5+ minutes after the download completed. A 100MB download at 8 MB/s takes about 12 seconds, but the flow record didn’t show up in journald until minutes later. So the rate window was looking at stale data, or no data at all, depending on timing.

Iteration 3: Deep dive into softflowd timeout semantics. This is where it got interesting. I pulled the softflowd source code and found the relevant logic at lines 504-510:

// maximum_lifetime triggers immediate export when exceeded
if (flow->flow_last.tv_sec - flow->flow_start.tv_sec > ft->param.maximum_lifetime) {
    flow->expiry->expires_at = 0;  // Immediate export
    flow->expiry->reason = R_MAXLIFE;
    flow->flowEndReason = IPFIX_flowEndReason_activeTimeout;
}

The key insight: tcp_timeout is an idle timeout. It only triggers export when the connection has been idle. For an active download that’s continuously transferring data, tcp_timeout never fires because the connection is never idle. The only timeout that forces export of an active flow is maximum_lifetime. With maximum_lifetime set to 3600 seconds (1 hour), an active download wouldn’t get its flow record exported until either the download finished and the connection went idle (triggering tcp_timeout) or an hour elapsed (triggering maximum_lifetime). That’s why flow records were showing up minutes late.

Iteration 4: Aggressive timeout changes. I changed all softflowd timeouts on pfSense from 3600/120/300 to 2 seconds:

Parameter	Old Value	New Value
Maximum Lifetime	3600	2
Expire Interval	60	2
TCP Timeout	3600	2
TCP RST	120	2
TCP FIN	120	2
UDP Timeout	300	2
General Timeout	3600	2

Also adjusted Vector’s end_every_period_ms to 1001 (just over 1 second) so the aggregation window aligned with the 2-second export interval. Result: flow records arriving within 2-4 seconds of the actual traffic.

Iteration 5: Dashboard and regex fixes. Changed the Grafana dashboard refresh interval to 10 seconds and rate windows from rate(5m) to rate(15s) to match the now-fast flow export. Also fixed a regex escaping problem: 192\\\\.168\\..* was invalid escape sequences in PromQL, so I simplified to 192.168.*. Commit 8eaefba (April 25).

Iteration 6: Debug sink cleanup. Removed the file sinks from the GoFlow2 Vector pipeline that were writing ~266 MB of logs to disk (commit 6a81e9a). These were debug sinks I’d added during iteration 1 to inspect raw flow records; they served their purpose and needed to go.

After all six iterations, I confirmed the pipeline was working: a download showing Destination: Strawberry Shortcake... 100% of 204.64MiB in 00:00:23 at 8.80MiB/s resulted in ~17.5+ MB/s on the Grafana graphs. The final pipeline architecture:

pfSense (softflowd, maxlife=2s)
    | IPFIX/NetFlow on :2055
    v
GoFlow2
    | JSON to journald
    v
Vector:
  goflow2_reader (journald source)
    -> parse_goflow2_flows (remap: extract fields)
    -> aggregate_flows (reduce: sum by src/dst/proto, 1s window)
    -> normalize_timestamps (remap: set timestamp to now())
    -> flow_to_metrics (log_to_metric: counter with increment_by_value=true)
    -> send_flow_metrics_to_mimir (prometheus_remote_write)
  Branch: parse_goflow2_flows -> send_goflow2_to_loki (raw flow records for forensics)

Storage impact: about 425 MB in Garage for the flow data. Reasonable.

Key commits for this section: cd76749 (ntopng-exporter precursor), 3d89101 (GoFlow2 pipeline primary), e4284a3 (softflowd config), 6a81e9a (debug sink removal), 8eaefba (regex fix), 3e5b8c8 (ntopng-exporter removal since GoFlow2 superseded it).

While all that was happening, I also added smartplug monitoring. KB#8036: Monitor smart plugs was a single commit (da9ab49, May 22) that added systemd.services.shelly-poll to poll a Shelly PlugUS RPC API at 192.168.2.126 via curl, with a systemd.timers.shelly-poll polling timer and Consul service registration. The Vector pipeline: shelly_poll_reader (journald source) parses JSON status from the poll output, then shelly_to_metrics extracts shelly_power_watts, shelly_voltage_volts, shelly_current_amps, shelly_energy_watthours_total, shelly_temperature_celsius, shelly_switch_on, shelly_wifi_rssi, and shelly_uptime_seconds as metrics, shipping them to Mimir. A 972-line Grafana dashboard (homelab-smartplug.json) ties it together. The whole thing is conditional on the andrewcz-net environment only, since that’s where the smartplug lives.

Then the logrotate battles. KB#7941: Figure out a more aggressive logrotate cadence was opened in September 2025 and became a real problem by the time Carter audited it. The root cause: the MariaDB converge template (common_components/mariadb.service.nix) had no logrotate block at all. Every container using that template had Vector logs growing unbounded.

The worst offender: vaultwarden-mariadb-ct0 on px01 had 1009M of Vector logs, 479,338 individual files, and 69% inode usage. Full-blown inode exhaustion. The MHA template also had no SSH/session exclusions in Vector, so every SSH session spawned tiny log files that never got cleaned up.

The fix (commit 39c11d6, May 22) added a services.logrotate block to common_components/mariadb.service.nix matching the standard pattern:

/var/log/vector/*.log:
  rotate 3
  hourly
  maxsize 250M
  copytruncate
  compress
  nodelaycompress

Also added /var/log/mysql/*.log rotation with maxsize 4G, rotate 15, hourly, owned by mysql:mysql. Seven stacks needed rebuilds: atom, firefly, ghost, kanboard, lgtm, nextcloud, and vaultwarden.

The pulumi upgrade saga was its own adventure. KB#8127: New pulumi version and vector logrotating started because an OS update bumped Python, which required updating the Pulumi SDKs across all projects. Commit eb4c7dc (June 2) bumped pulumi_proxmoxve across all 17 project pyproject.toml files. Commit 2362441 (June 4) handled library-level fixes.

The new pulumi_proxmoxve version added calculated params that caused empty update POSTs to the Proxmox API. The fix was running pulumi up --refresh repeatedly until all the calculated params settled. Not elegant, but it worked.

While debugging that, I also discovered Vector wasn’t getting logs rotated on some containers. I demo’d the issue on scaffold-traefik and ended up cleaning it up with a one-liner across all running nodes:

for i in $(cat Downloads/running_nodes); do ssh ${i} "sudo rm -rf /var/log/vector/*-2026-*; sudo chown -R vector:vector /var/log/vector" 2>/dev/null; done

Also fixed component_name/app_name not being passed in for vaultwarden and searxng, which was throwing off monitoring. The component_name/app_name pattern was introduced in commit 7d2cc28 (May 18) in the MariaDB converge template and baselined in commit 896646b (February 17) in the Vector config. Without those labels, logs from those services were showing up in Grafana without proper identification… which made them effectively unqueryable in a multi-service Loki instance.

Finally, operational cleanup. KB#8208: Clean up 3 failed consul client registrations dealt with leftover Consul state from a proxmini02 outage. Three failed client registrations across two datacenters, plus three additional left stale entries. No commits; this was pure CLI work.

The cleanup: consul force-leave on the failed nodes (which transitioned them to left), then consul force-leave -prune on all left nodes to remove them from the registry entirely. After cleanup, all 6 datacenters were clean:

DC	Servers	Clients	Total
`hub-andrewcz-com`	3	20	23
`hub-andrewcz-org`	3	24	27
`hub-andrewcz-net`	3	5	8
`hub-cziryak-com`	3	10	13
`hub-thepipeline-tv`	3	4	7
`hub-curatingcollections-com`	2	8	10

Root Cause

There were four distinct technical mechanisms that caused problems across this work, and they’re worth pulling out individually because they’re the kind of footguns that bite anyone running a similar stack.

The Vector log_to_metric counter bug. Vector’s log_to_metric transform has an increment_by_value option that defaults to false. When false, a counter-type metric increments by 1 for each matching log event, regardless of the value field. When true, it increments by the value of the specified field. For flow records carrying byte counts, this is the difference between “I received 1 flow record” and “I received 47,000,000 bytes.” The default behavior makes sense for event counting (how many errors per minute) but is silently wrong for value aggregation. The documentation mentions this, but it’s not prominent… and if you’re used to Prometheus counter semantics where the value is the metric, the Vector default is surprising.

The softflowd timeout semantics. This was the most subtle issue. softflowd has two categories of timeouts: idle timeouts (tcp_timeout, udp_timeout, general_timeout) and the active timeout (maximum_lifetime). Idle timeouts only fire when a flow has been inactive for the specified duration. For a long-running, continuously active connection (like a large file download), idle timeouts never trigger because the connection is never idle. Only maximum_lifetime forces export of an active flow. With the default maximum_lifetime of 3600 seconds, active flows don’t get exported until they complete or an hour passes. This is documented in the softflowd source code but not in any user-facing documentation I could find. The IPFIX specification defines flowEndReason values like IPFIX_flowEndReason_activeTimeout which correspond to this behavior, but you have to know to look there.

The inode exhaustion from missing logrotate. The MariaDB converge template (common_components/mariadb.service.nix) had no services.logrotate block. Vector writes one log file per journald unit per day (/var/log/vector/{{unit}}-%Y-%m-%d.log), and without rotation, these accumulate indefinitely. The MHA template made it worse: every SSH session to the container spawned a separate journald session, and Vector created a log file for each one. On vaultwarden-mariadb-ct0, this resulted in 479,338 files consuming 69% of inodes. Inode exhaustion is particularly nasty because df -h shows plenty of free space; you only see the problem with df -i. The container keeps working until inodes hit 100%, at which point no new files can be created and everything that writes files starts failing in confusing ways.

The pulumi_proxmoxve calculated params issue. The new version of pulumi_proxmoxve introduced calculated parameters: values that the Proxmox API computes and returns, but that Pulumi tries to send back on the next update. When Pulumi sends an empty or null value for a calculated param, the Proxmox API rejects it. The pulumi up --refresh pattern works because it tells Pulumi to read the current state from the provider first, which populates the calculated params with real values before attempting the update. This is a known pattern with Pulumi calculated properties; the provider needs to see the current values before it can safely update them.

The Fix

The concrete changes, organized by area:

LGTM stack deployment: - Created lgtm/ directory with Grafana, Loki, Mimir, and Prometheus configs (commit da10023) - Switched object storage from MinIO to Garage (commit 896646b) - Disabled all telemetry and analytics reporting (commits c57e02f, e75c14c) - Set up nix-serve on cache server for precompiled Grafana plugins with pubkey auth (commit ae149d6)

Metrics coverage: - MariaDB exporter: 144 insertions to mariadb-mha.service.nix (commit 42f1883) - Redis exporter: 139 insertions to redis.service.nix (commit 42f1883) - Traefik metrics: 200 insertions (commit 42f1883) - PVE exporter added to LGTM (commit bb61eed) - systemd monitoring across all 18 containers: 2100 insertions (commit 0eed58a) - php-fpm exporter startup ordering fix: after and wants dependencies on php-fpm pool service (commit 7d2cc28) - Nextcloud exporter port fix (commit f50cf5a)

Log pipeline: - Vector across all services: journald to Vector to Loki, 2845 insertions across 21 files (commit 896646b) - Dedicated vector user/group with logrotate config: su = "vector vector", path = "/var/log/vector/{{unit}}-%Y-%m-%d.log" (commit 730bccf) - prune_journald_metadata transform standardized across 22+ services (commit da9ab49) - component_name/app_name labels on all log streams (commits 896646b, 7d2cc28)

Retention: - lgtm_retention variable introduced (commit ae149d6) - Plumbed through all 6 environment configs (commit d2f66ba) - Prometheus agent mode with --storage.agent.retention.max-time=48h on 13 containers (commit 691e56b) - Mimir retention: retention_period, retention_enabled = true, retention_delete_delay = "2h", retention_delete_worker_count = 4 (commit 896646b) - Prometheus local retention bumped to 72h in converge template (commit 896646b)

GoFlow2/NetFlow pipeline: - softflowd on pfSense with all timeouts set to 2 seconds (commit e4284a3) - GoFlow2 receiving IPFIX/NetFlow on port 2055, outputting JSON to journald (commit 3d89301) - Vector pipeline: goflow2_reader -> parse_goflow2_flows -> aggregate_flows (1s reduce window) -> normalize_timestamps -> flow_to_metrics (with increment_by_value = true) -> send_flow_metrics_to_mimir - Forensics branch: parse_goflow2_flows -> send_goflow2_to_loki (raw flow records) - Dashboard: 10-second refresh, rate(15s) windows, simplified regex 192.168.* (commit 8eaefba) - Debug file sinks removed (commit 6a81e9a) - ntopng-exporter removed, superseded by GoFlow2 (commit 3e5b8c8)

Smartplug monitoring: - systemd.services.shelly-poll polling Shelly PlugUS RPC API at 192.168.2.126 via curl (commit da9ab49) - systemd.timers.shelly-poll polling timer - Consul service registration - Vector pipeline: shelly_poll_reader -> parse_shelly_status -> shelly_to_metrics -> send_shelly_metrics_to_mimir - 972-line Grafana dashboard (homelab-smartplug.json) - Conditional on andrewcz-net environment only

Logrotate fix: - Added services.logrotate to common_components/mariadb.service.nix (commit 39c11d6) - /var/log/vector/*.log: rotate 3, hourly, maxsize 250M, copytruncate, compress, nodelaycompress - /var/log/mysql/*.log: maxsize 4G, rotate 15, hourly, owned by mysql:mysql - Seven stacks rebuilt: atom, firefly, ghost, kanboard, lgtm, nextcloud, vaultwarden

Consul cleanup: - consul force-leave on 3 failed nodes (transitioned to left) - consul force-leave -prune on all 6 left nodes - Verified all 6 datacenters clean (0 failed, 0 left)

What We Learned

Default values in transform tools are footguns. Vector’s log_to_metric defaulting increment_by_value to false is technically correct for event counting but silently wrong for value aggregation. I should have read the transform docs more carefully before assuming Prometheus counter semantics. Going forward, I’m treating any Vector transform that involves numeric fields as something to verify against raw data before trusting the output.
Flow export timeout semantics are not intuitive. The distinction between idle timeouts and active timeouts in softflowd is buried in the source code, not the documentation. tcp_timeout sounds like “how often flows get exported” but it’s actually “how long after a connection goes idle does the flow get exported.” For active connections, only maximum_lifetime matters. I spent three iterations debugging rate calculations before I thought to read the C source. Lesson: when a tool’s behavior doesn’t match expectations and the docs don’t explain it, go to the source code.
Inode exhaustion is invisible until it isn’t. df -h showed plenty of free space on vaultwarden-mariadb-ct0. The problem was 479,338 files consuming 69% of inodes. I got lucky that Carter caught this during an audit before it hit 100%. The fix is simple (add logrotate) but the detection requires df -i, which wasn’t part of my default troubleshooting checklist. It is now.
Converge templates need the same operational hygiene as service configs. The MariaDB converge template had no logrotate block because it was a template, not a service config… but every container using that template inherited the gap. Templates amplify omissions; one missing block becomes 7 missing blocks across 7 stacks. I need to audit converge templates with the same scrutiny I apply to individual service configs.
Pulumi calculated params require refresh-first updates. Bumping a Pulumi provider version can introduce calculated params that break pulumi up with empty POSTs to the API. The fix (pulumi up --refresh) is simple once you know it, but the error messages from the Proxmox API don’t point you there. I should check for calculated param changes in provider release notes before bumping versions.
Zabbix was the wrong paradigm for this scale. Polling-based, agent-driven monitoring works for “is this thing up” but doesn’t scale to high-cardinality metrics, log aggregation, and flexible dashboards across 22+ services. The LGTM stack (Loki, Grafana, Mimir) with Vector as the ingest pipeline gives a level of observability that Zabbix couldn’t match without significant bespoke configuration. The operational complexity of maintaining Zabbix (server, frontend, database, agents) was higher than maintaining the LGTM stack, which runs as NixOS services with declarative config.
Consul state persists across outages. The proxmini02 outage left 6 stale Consul registrations across 2 datacenters. Consul doesn’t automatically prune failed nodes; you have to explicitly force-leave and then force-leave -prune. This is a known Consul operational pattern but it’s easy to forget after an outage when you’re focused on getting services back up. Adding a Consul membership audit to post-outage recovery procedures would catch this earlier.
Debug sinks are useful but must be cleaned up. The GoFlow2 debug file sinks wrote 266 MB of logs to disk before I removed them. They were essential for iteration 1 (inspecting raw flow records) but became a liability once the pipeline was working. I should treat debug sinks as temporary scaffolding with an explicit cleanup step, not permanent config.

What Happened

Root Cause

The Fix

What We Learned

Tickets