Kafka at 94 % disk with 17 hours left, and retention that kept twice what we asked

At Gambchamp an alert fired: Kafka was filling its volumes. The question was the usual one — can this wait until morning?

It could not. All three brokers were at 36.6 of 39.1 GiB, 94 %. Growth averaged 3.37 GiB a day and reached about 5 GiB a day at the evening peak. That left roughly 17 hours, which put the disk-full moment in the middle of the night, right inside the peak. The next automatic retention cleanup was not due for another two days.

Nothing was broken

There were two stacked causes, and neither was a fault.

Traffic had grown about 150 times. From around 5 messages per second at the start of August to a plateau of ~780 messages per second after a product launch. Consumer lag was zero, brokers logged no errors, no partition was under-replicated. The 40 GiB volumes had simply been sized for a different workload.

Retention was really 7 to 14 days, not 7. The broker config did not set retention at all, so Kafka defaults applied: log.retention.hours=168 and log.roll.hours=168, with 1 GiB segments. Retention only deletes closed segments. Our partitions gained a few hundred megabytes a week and never reached 1 GiB, so a segment closed on the 7-day timer — and only then did its retention clock start. That is the sawtooth on the disk graph: a big drop once a week, nothing freed in between.

The practical consequence: you cannot size such a volume as “7 days × rate”. With these defaults it is 14.

Buying time

The volumes used an expandable gp3 storage class, so the first step was an online PVC expansion from 40 to 100 GiB. No restarts, no under-replicated partitions, the GitOps application stayed in sync — the PVCs come from volumeClaimTemplates and are not in Git, so self-heal had nothing to revert. We went straight to 100 rather than 60 because AWS allows modifying a volume only once every six hours.

Fixing the cause

The next day we learned that 7-day retention was an actual requirement of the backend architects. That flipped the meaning of the fix: the system was not under-retaining, it was keeping almost twice as much as required. Rolling segments every 12 hours (log.roll.ms=43200000) brings real retention back to 7–7.5 days without touching log.retention.hours.

With a projected 4x growth, the steady state came to ~185 GiB as is versus ~99 GiB with 12-hour segments, so the target volume became 200 GiB. Throughput, IOPS, file descriptors and CPU were all far from their limits — the ceiling was space, not performance.

Three traps on the way

• A wrong key that said “Completed”. We first set log.segment.ms, which is not a broker setting (the topic-level one is segment.ms). kafka-configs.sh replied Completed updating default config and changed nothing. The right name is log.roll.ms; only --describe --all shows what is really in effect.
• A dynamic override shadows the file. Dynamic broker config takes precedence over server.properties. Until the override is removed, the value in Git is decorative — and removing it too early briefly reverts to 168 hours.
• “Safe” StatefulSet recreation restarted every broker. An old kubectl.kubernetes.io/restartedAt annotation in the live pod template was missing from the Git render, so the hash changed and all brokers rolled — one at a time, cleanly, but unplanned.

After the change: 200 GiB volumes at 21 % use, everything coming from Git, segments rolling twice a day.

Takeaways

• Compute time-to-full from the peak rate, and compare it with the next retention drop. If the drop comes later, waiting is not an option.
• A requirement that happens to equal a default is an unwritten requirement. Set retention explicitly, even when it matches the default.
• kafka-configs.sh reports success for unknown keys. Verify with --describe --all.
• Before delete sts --cascade=orphan, check the pod template for a restartedAt annotation.