A Grafana panel that tells you when a pod will OOM

At Gambchamp I run a multi-tenant product platform on AWS EKS, with all clusters feeding one VictoriaMetrics behind a shared Grafana. Golden-signal dashboards tell you what is broken now. I wanted one that answers a different question: if the current trend holds, when does this pod run out of memory, and when does this volume fill up?

The formula

The core is one line:

(limit - current) / clamp_min(deriv(usage[6h]), 1)

deriv fits a straight line over the window and returns its slope in bytes per second, so dividing the remaining headroom by it gives seconds until the limit. Two windows matter: 6 hours catches acute growth, 24 hours catches slow leaks that short-window noise hides.

clamp_min(..., 1) does two jobs. It avoids division by zero, and it turns a shrinking series into a huge ETA instead of a negative one. Without it the table fills with -Inf and negative garbage. In the table the thresholds run low to high: red below 24 hours, orange up to 72 hours, green beyond. The unit s formats itself into days and hours.

The same idea extended to managed databases: free storage divided by the rate it disappears gives an ETA to a full disk.

It caught a leak on day one

The first render flagged an admin backend with an ETA of zero hours. I checked: it had been OOMKilled with exit code 137, around 900 Mi against a 1 Gi limit, six restarts. A CronJob had been restarting it on a schedule, which hid the symptom without fixing anything.

Bug one: negative ETAs for pods without limits

Some backends set a memory limit only on their sidecar, not on the main container. A pod-level sum(limits) - sum(used) then subtracts the whole pod’s usage from the sidecar’s limit, goes negative, and the panel paints a perfectly stable pod red. The fix was to compute per container with an inner join, so containers without a limit drop out, and to keep only series where the slope is actually positive.

Bug two: cAdvisor double counting

On this setup cAdvisor labels every cgroup level of a pod with the same container value: the pod cgroup, each container and the pause container. sum by(pod) adds the pod-level cgroup, which is already the total, to the containers, and every memory and CPU panel doubled. One pod read 156 % of its limit when the truth was 79 %. The rule I now follow: max by(pod) for cAdvisor metrics, sum by(pod) for kube-state limits and requests.

What the forecast cannot see

A trend only catches monotonic leaks. A pod that already cycles through grow, OOMKill, restart has a regression slope near zero, so the forecast correctly leaves it out. Those pods show up in the restarts and memory-percentage panels instead. An empty forecast table means “no monotonic leak right now”, not “everything is healthy”.

Two more traps from the same week. For restarts, read the raw counter rather than increase[1h]: a recreated pod starts a new series at zero, and increase hides a restart storm on exactly the pod you care about. And an auto-scaled Y axis lies on large, slow values: a database volume looked like it was falling off a cliff, but the change was 15 MB out of 163 GiB, an ETA of almost seven years.

Takeaways

• Remaining headroom divided by deriv is a usable ETA, but clamp the slope and keep only growing series.
• cAdvisor and kube-state metrics aggregate differently; check one pod by hand before trusting any sum by(pod).
• Trend forecasts and restart counters catch different failure classes. You need both.
• Judge slow-moving capacity by the absolute number and the ETA, not by how steep the line looks.