Alerts that cannot fire: a blind ServiceMonitor and a dead delivery channel

At Optimacros a Vault peer sat sealed for 32 hours, and 53 of 55 VaultStaticSecret objects stopped syncing for over a week. Nobody was alerted. A VaultSealed alert existed in the repository, Prometheus was healthy, and Alertmanager was up.

Two independent failures were stacked on top of each other, and either one alone would have been enough to hide the incident.

Failure one: the alert was structurally blind

The Vault Helm chart ships a ServiceMonitor that selects pods labelled vault-active: "true". In a three-node Vault cluster, that is exactly one pod — the active leader. And the active pod is, by definition, unsealed: a sealed node cannot lead the Raft cluster.

So Prometheus scraped only the one pod that could never be sealed. The standby peers, which are precisely the ones that come back sealed after a restart, were never scraped at all. vault_core_unsealed == 0 had no series to evaluate.

The irony: the metric was reachable. Unauthenticated metrics access was enabled, so a sealed pod would have served it happily. The selector, not the exporter, was the blindfold.

Failure two: delivery had been dead for two days

Every Alertmanager notification was failing with connect: connection timed out. The Telegram API hostname resolved, from that cluster, to an address that was unreachable, while another Telegram data-centre address answered normally with a valid certificate. General egress was fine — other external services responded — so it was a per-address block, not a firewall policy. A small status bot that used the same API had been failing silently too.

The alerts fired. They just never left the cluster.

The fix

1. Point the ServiceMonitor at all peers: switch its selector to the label of the headless Service that backs every Vault pod. The chart exposes this through values, no template patching required.
2. Add a ground-truth alert that does not depend on Vault’s telemetry: a sealed pod is never Ready, and kube-state-metrics always reports readiness. An alert on kube_pod_status_ready == 0 for Vault pods catches a seal even when Vault’s own metrics are missing.
3. Alert on absence: absent(vault_core_unsealed). A rule written as == 0 can never fire on a series that does not exist.
4. Restore delivery by pinning the reachable Telegram address with hostAliases on Alertmanager and on the status bot.

The uncomfortable caveats

• Pinning an IP is a workaround. If that address is blocked too, delivery dies silently again. The durable answer is a second, independent channel — email for critical alerts — or an egress proxy.
• AlertmanagerFailedToSendAlerts exists, but it is delivered through the very channel that is broken. Only a dead man’s switch — an external receiver that expects a regular heartbeat and complains when it stops — catches this class of failure.
• The Telegram client library logged the full request URL, bot token included, in plain text. Anyone with log access owned the bot. We rotated it.

Takeaways

• metric == 0 is blind to metric absent. Write a separate absence alert.
• A target selector can quietly exclude exactly the instances that fail. Check how many targets you actually scrape against how many instances exist.
• Prefer ground truth over self-reported health. The broken component is the one least able to tell you it is broken.
• Test the delivery path, not just the rule. A firing alert that never leaves the cluster is indistinguishable from no alert at all.