Merged, synced, mounted — and still not alerting: vmalert without hot reload

At Gambchamp the alert rules for the whole platform live in a Git repository. A merge goes through Argo CD with auto-sync into a ConfigMap, and vmalert, part of a central VictoriaMetrics observability hub, evaluates them. One day I merged a set of new Kafka rules: brokers down, under-replicated partitions, consumer lag and rebalance storms.

Argo CD synced. The ConfigMap in the cluster contained the new rules. And nothing happened.

Proving the rules were not running

The check that matters is not “is the rule in Git” but “is the engine evaluating it”. The ALERTS metric that vmalert writes for every pending and firing alert had zero series for alertname=~"Kafka.*". At the same time, running the raw rule expression by hand returned 18 matching consumer groups. So the condition was true, and the rule was simply not loaded.

The cause

This vmalert, deployed from the standard chart, had no config-reloader sidecar and no -rule.configCheckInterval flag. The rules ConfigMap is mounted as a volume. The kubelet does update the files in that volume when the ConfigMap changes, but vmalert reads rule files only when the process starts. Without a reloader or a check interval, nothing tells it that the files changed.

The pipeline merge, sync, ConfigMap worked perfectly and stopped one step before the finish line. The new rules sat on the pod’s disk and never reached the engine.

The immediate fix

I restarted the vmalert Deployment. On startup it read all rule files, including the new groups. The 18 rebalance-storm alerts went to pending immediately and to firing after their for: 15m window, and the alerts panel on the dashboard showed them right away.

A restart has a cost worth knowing: it resets the for: timers of every rule in the fleet, not only the new ones. Later I found a gentler way. vmalert accepts an explicit POST /-/reload, which re-reads the rule files without restarting the process. It reloads only the groups that changed; untouched groups keep their state, and alerts that were already firing stay firing. For rule changes that is the right tool.

The lasting fix

Either of two changes makes GitOps alerts activate on their own:

• add -rule.configCheckInterval=30s, so vmalert re-reads its files periodically, or
• add a config-reloader sidecar that calls the reload endpoint whenever the ConfigMap changes.

Until one of them is in place, every merge to the alerts repository needs a manual reload.

Takeaways

• “The rule is merged” is not “the rule works”. Confirm through ALERTS{alertname="X"} that the engine sees it, pending or firing.
• A ConfigMap mounted as a volume updates the files, not the process. Check whether the consumer watches them.
• Prefer an explicit reload over a restart for rule changes: a restart resets every for: timer.
• Calibrate counter-based thresholds on live data before merging, so a new rule is neither dead nor noisy.