Three alerting outages from one root: a name where an ID belonged

At Gambchamp alert delivery for a tier-0 production system was completely dead, and nothing said so. Someone had renamed the Slack channel. Alertmanager kept posting to the old name, and Slack answered channel_not_found every ten seconds. Two live alerts sat in the queue the whole time. The only outward sign was alertmanager_notifications_failed_total — a counter nobody watched.

The third time on the same identifier

It was the third failure in a row on the same thing:

• first, a shared incoming webhook ignored the channel field, so alerts landed in a shared channel. Fix: a dedicated Alertmanager;
• then, a bot token honoured channel and delivery worked — but it was bound to the channel name. It looked solved;
• finally, the channel was renamed and delivery died.

Each fix changed the transport and left the binding untouched, so the class of failure survived every repair.

A label is not an identifier

A Slack channel name is a human label: its owner can change it at any time without telling anyone. The channel ID is issued once and never changes — not on rename, not on a privacy switch. Our config referenced the label.

Two things made the failure silent rather than loud. Slack marks channel_not_found as unrecoverable, so Alertmanager does not retry; notifications are dropped, not delayed. And a dead delivery path looks exactly like a quiet system. Nothing fires when nothing can fire.

The fix

1. Pin the channel by ID in the receiver. The channel is private and the bot has no permission to list it, so the API cannot validate the ID read-only. I validated it with a scheduled message that was deleted before it was ever delivered — a probe that resolves the ID without posting anything.
2. Route by an explicit label, not by name. When the same system’s alerts were split into an application channel and an infrastructure channel, the obvious regex on the alert-name prefix would have sent five RDS guardrails — replication slots, storage — into the application channel. A signal label set in the rule group lets each rule declare its own layer instead of depending on how someone named it.

After the change I checked that the configuration was actually re-read, not just synced, and watched the derivative of the counters: failed notifications to zero, sent notifications rising. Cumulative counters on their own say nothing.

Takeaways

• Anything a human can rename is a label, not an identifier. Bind to the immutable handle — an ID — or to an explicit declaration, such as a label the author sets on purpose.
• Test the delivery path, not just the component. A healthy Alertmanager that cannot deliver is indistinguishable from having no alerts.
• Alert on increase(alertmanager_notifications_failed_total[5m]) > 0. It is the cheapest dead-man switch you will ever add.
• Audit the same class elsewhere: consumer group IDs built from pod names, exporters filtering by naming convention, policies matching objects by name.