The rebalance storm was just a deploy: a threshold that scales with group size

At Gambchamp an alert called KafkaConsumerRebalanceStorm fired for one consumer group: 60 rebalances per hour against a threshold of 30, and it stayed red for more than half an hour. Neighbouring groups on the same cluster showed 8 and 12. The on-call reading was obvious: this group rebalances more than it works, go find the crashing pods.

The catch: it happened on every release, and people had slowly learned to ignore the channel. That is the worst outcome an alert can have.

What I ruled out first

• Crashing pods: every pod had RESTARTS=0.
• Broker trouble: only one group was affected, the others were calm.
• An automation loop restarting things: the reloader had been silent for six days.

A diff of the pod templates between the old and the new ReplicaSet showed exactly one change — the image tag. A normal release.

The number measured size, not behaviour

The threshold was absolute, but the metric was summed over the whole group.

1. The ETL service runs 20 consumers per pod in one consumer group, one per topic but with a shared group.id. With two replicas that is 40 members.
2. The rebalance counter has labels for the group, environment, service and version — and none that tells consumers apart. All 20 increment the same series. On a freshly started pod ordinary groups read 1.0; this one read 20.0.
3. A rolling update means 20 joins per new pod plus reassignments on the old ones: about 60, mechanically. The alert firing on release was guaranteed by construction.
4. On top of that the rule used increase(...[1h]), a sliding window. The spike stayed inside it for a full hour after the group had already settled. The alert was describing the past.

I asked the application’s /metrics directly what the counter means instead of guessing from its name. One curl replaced an hour of reasoning.

The fix: normalise, and make the window shorter than for

(
  sum by(cluster, consumer_group)(rate(kafka_consumer_rebalances_total[10m])) * 3600
  /
  clamp_min(sum by(cluster, consumer_group)(kafka_consumer_task_alive), 1)
) > 10

with for: 15m. The unit is now “rebalances per consumer per hour”.

The important trick is that the 10-minute window is shorter than the 15-minute for. A deploy spike lives in the window for exactly ten minutes and then falls out; it cannot hold for fifteen. Deploys are excluded by design, not by threshold tuning.

I validated it against the live engine, including a negative control: a deliberately broken query had to return 422, so an empty result could not be mistaken for “no data”. I also checked that no group had a counter without the denominator series, because division would silently drop it. On history the threshold holds down to about 6; at 4–5 the deploy tail stretches to 16 minutes and the rule lies again.

The real root — 20 consumers in one group with eager rebalancing — stays a code change for the team. This was an observability fix.

Takeaways

• A counter over a sliding window is not “now”. increase[1h] keeps a spike for an hour after it ended.
• An absolute threshold on an aggregate is only fair when cardinality is equal. Normalise.
• A rate window shorter than for: gives immunity to bursts like deploys and scale events.
• Check the checker: a negative control tells “no data” apart from “broken query”.