The alert fired and nobody saw it: a cluster label lost in aggregation

At Gambchamp a worker node in a production EKS cluster sat at 90–98% CPU for about four hours overnight and ended with HostOomKillDetected. The OOM alert reached the team’s channel. The node CPU alert did not, and it was missing from the Grafana “alert history” panel too. The first conclusion was natural: the CPU alert never fired, the threshold must be wrong.

“Did not fire” is a claim to check

Firing, routing and display are three separate stages, and the ALERTS series tells you about the first one directly. A range query over the incident window showed that HighCpuLoad did fire: pending just before 23:00 UTC, firing from 02:40 to 03:43.

So the question changed from “why didn’t it fire” to “where did it go”.

The label that vanished

The rule aggregated with avg by(instance). That drops every other label, including cluster. And both consumers selected on cluster:

• the dedicated Alertmanager for this product routes on the cluster label;
• the Grafana panel queries ALERTS{alertstate="firing", cluster=~"$cluster"}.

Without the label the alert landed only in the shared default route, where nobody from this team was looking. The OOM alert was delivered because it aggregates with count by(node, cluster) — it kept the label.

Two other hypotheses I rejected on data before touching anything:

• the and on(instance, application) join breaking — no series carries application, all 84 matched;
• for: 1h being reset by dips — there was a continuous 124-minute run above 90%.

The fix

In the generic node rule: avg by(instance) became avg by(instance, cluster), and for: 1h became for: 15m — an hour is too long for a node heading towards an OOM. A seven-day backtest of the new rule across the fleet fired on only two nodes, so the shorter hold adds no noise.

The change went out through Argo CD, then a POST /-/reload to vmalert after checking that the new rule had actually reached the pod volume. A reload instead of a restart keeps the for: timers of every other rule group.

The saturation itself came from Kafka requesting 200m CPU while using about a full core, so the scheduler packed it next to two ETL replicas. That was fixed separately by raising the requests.

A bonus trap found on the way: cAdvisor exports each container twice, as a .scope and as a pod .slice, both labelled with the container. A naive sum by(pod)(rate(container_cpu_usage_seconds_total)) is doubled.

Takeaways

• When a route or a dashboard selects on a label, every aggregation feeding it must keep that label.
• “The alert did not fire” is a hypothesis. Check ALERTS before changing thresholds.
• Grep your rules for sum/avg/max by(...) without cluster — each one is the same blind spot.
• Prefer a reload over a restart when changing rules, so other alerts keep their pending timers.