Kafka
Running Kafka in production: consumer lag, rebalances, retention and disk capacity.
4 posts
- The rebalance storm was just a deploy: a threshold that scales with group sizeA Kafka rebalance alert fired at 60 per hour on every release. The metric was summed over 40 consumers and averaged over a sliding hour, so the alert described the past.
- Kafka at 94 % disk with 17 hours left, and retention that kept twice what we askedTraffic grew 150x and nobody resized the volumes. Worse, with default segment rolling a 7-day retention really meant 7 to 14 days. How we bought time and fixed the cause.
- The lag went to zero and nobody was fixed: a Kafka consumer stall hidden by a group.id renameConsumer lag grew for five days, then dropped to zero and stayed there. It looked self-healed. In fact the consumer was stuck, the metric had vanished, and the alert had resolved itself.
- Merged, synced, mounted — and still not alerting: vmalert without hot reloadNew Kafka alert rules went through Git and Argo CD into the ConfigMap, yet vmalert never evaluated them. The GitOps pipeline broke on its very last step.