The lag went to zero and nobody was fixed: a Kafka consumer stall hidden by a group.id rename

On a product platform at Gambchamp a consumer-lag graph grew for five days to about 80,000 messages — and then fell to zero and stayed flat. It looked like the problem had gone away on its own. At the same time the “total consumer lag” panel showed a negative number, and “active consumer groups” showed 1 in a cluster with dozens of live groups.

Three things that each looked fine were wrong at once.

Layer one: a consumer that was alive and doing nothing

After the brokers were recreated, the group’s committed offset froze and did not move for more than eight hours while producers kept writing. The consumer stayed a member of its group, so from Kafka’s point of view it was alive — it just never advanced. The client library had not reconnected properly after the broker restart.

The ground truth came from Kafka itself, bypassing the exporter:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group>

A CURRENT-OFFSET that does not change while LOG-END-OFFSET grows is a stuck consumer, whatever its membership says.

Layer two: zero was not recovery

The drop to zero coincided with a deploy that changed the group.id naming scheme. The old groups were orphaned and the new group started consuming. But the new name did not match the exporter’s --group.filter whitelist, so the metric for it simply disappeared.

The lag alert saw its series vanish and resolved. For three more days the rule watched nothing, and the graph looked healthy because there was nothing on it.

Layer three: negative lag

kafka-exporter reports lag = -1 for partitions with no committed offset — the ones where Kafka itself shows LAG = "-". A panel that summed lag went below zero, and the alert threshold was quietly shifted by the number of such partitions.

As a bonus, the application’s own consumer metrics existed but were never collected: the chart declared metrics port 9001, the application listened on 9006.

The fixes

• Scrape the real port. A vmagent relabel keeps the declared port for deduplication but rewrites the address to the port the app actually listens on.
• Blacklist, not whitelist. --group.filter=.* plus an exclude for ephemeral per-pod groups. A whitelist by name breaks on the first rename — exactly what blinded us.
• clamp_min(..., 0) in lag panels. It removes the -1 while keeping the series; filtering with > 0 would drop the series entirely.
• Alert on the client-side metric. highwater - position from the consumer itself does not depend on group naming or on the exporter’s filter.

Takeaways

• Every threshold alert needs a paired absent(). Otherwise a disappearing metric reads as recovery — which is exactly what happened.
• A whitelist in a metrics exporter is a time bomb waiting for the next rename.
• LAG = "-" is not zero, and an exporter’s -1 is not a number to sum.
• Check that the port your chart declares is the port your app listens on.
• One channel for every alert is no alerting at all: the five-day firing alert drowned among thousands of notifications. An owner and an SLA for warnings matter more than one more rule.