The disk alert worked. Its thresholds still meant 19 days of silence
A database volume was 64% full and growing 3.4 GiB a day, and the team heard nothing. Every link in the alerting chain was healthy. The defect was the distance between thresholds.
A teammate told me the database free-space alert “won’t fire”. The volume of a managed Postgres cluster was 64% full, growing about 3.4 GiB a day, and storage autoscaling was off. Nobody was getting new signals about it.
I expected to find a broken link somewhere. I did not.
Walk the chain, link by link
When an alert “doesn’t fire”, there are five places it can break. I checked each one:
- The metric. The CloudWatch exporter series for free storage was fresh.
- The expression. Run by hand against the metrics store, it returned the expected value.
- The rule. The 50% rule was in state
firingin vmalert. - The route. Alertmanager sent that cluster’s alerts to the team channel. There was no severity filter and no inhibit rule that could swallow it.
- The delivery. The 50% notification had arrived. Once.
Everything worked. The problem was not in the chain. It was in the layout.
The arithmetic of silence
The thresholds were 50 / 75 / 90%. After the one-time ping at 50%, the next band was 25 points away. On a 256 GiB volume at 3.4 GiB per day, that is about 19 days without a single signal, while the disk keeps filling. A 7-day predict_linear forecast existed as well, but it only wakes up in the last week before the disk is full.
So the rules were correct, and the team still had weeks of silence. “The alert won’t fire” was the right feeling, even though the diagnosis behind it was wrong.
The formula is simple enough to check for any capacity alert:
days of silence = (gap between bands in %) × size / growth per day
If that number is longer than the time you need to act — resize, clean up, add retention — the layout is wrong, even if every rule is correct.
The fix
New bands: 60 (info) / 70 (warning) / 80 (warning) / 90 (critical), with the absolute “less than 2 GiB left” rule unchanged. At the current growth, 10-point steps mean a ping roughly every seven days.
Before pushing, I evaluated each new expression against the live data. Only the 60% band matched, as expected. Then the usual GitOps path: rules repository, Argo CD sync, vmalert reload. The reload was blocked for a moment by an expired admin client certificate in my kubeconfig, which is a small reminder to watch the expiry of the credentials your runbooks depend on. After that, all rules were health=ok and the 60% band went pending immediately.
Caveats
- The percentage is calculated against a hardcoded 256 GiB, because the exporter has no allocated-size metric. After any resize, the constant has to be updated too.
- Alerting does not stop the growth. The resize or the retention work is still owed.
Takeaways
- Check band spacing against the burn rate, not just rule correctness. Gap divided by daily growth is the number of days your team hears nothing.
- A 7-day
predict_linearis a last-week alarm, not capacity planning. - When someone says “the alert doesn’t fire”, walk the whole chain. Metric, expression, rule state, route, delivery. Sometimes every link is fine and the design is the bug.