Six of eight critical alerts were about a service we had already shut down
Blackbox probes for a decommissioned production cluster kept firing for almost a month. The probes lived in another cluster's config, and the decommission only cleaned up its own folder.
During a review of monitoring coverage, I looked at the list of active critical alerts. There were eight. Six of them — ProbeFailed, HttpStatusCode_2XX, HttpStatusCode_4XX — pointed at the public endpoints of a production cluster that had been decommissioned weeks earlier.
They had been firing for almost a month. And they were flapping: startsAt reset several times a day, so they kept re-notifying the channel. People had learned to scroll past them. That is the real cost of noise: the next real critical alert would land in a channel everyone ignores.
Why nothing else hinted at it
The decommissioned cluster sent zero series. No node alerts, no pod alerts, nothing in the cluster’s own dashboards. Only the probes kept talking, because they were not running in that cluster.
The cluster’s manifests had been moved to an _old/ folder in the GitOps repository. But the blackbox scrape jobs that probe its domains lived in another cluster’s vmagent config: the staging cluster probes the production edge from outside, which is a good design. The decommission only touched the production folder, so the probes outlived the service.
Confirm it is dead without trusting metrics
A failing probe says “unreachable”. It does not say “decommissioned”. Before removing anything, I confirmed it from three independent sides: the DNS apex was empty, the API was silent, and the owner confirmed the shutdown.
Metrics can add one useful fact, though: whether the probe has ever succeeded within retention.
max_over_time((timestamp(probe_success{instance=~".*<domain>.*"} == 1))[30d:10m])
Empty means not a single successful probe in 30 days.
Remove carefully
- Find every definition. I searched the GitOps repository for the domain and for the job names, ignoring
_old/and commented-out copies. - Check the blast radius. The alert rules were generic (
probe_success == 0). No dashboard or rule referenced these job names, so removing the series would resolve the alerts and break nothing else. - Remove only what is dead. Two dead targets and one job that contained only dead targets. Live targets and the shared job name stayed.
- Commit with an optimistic lock. Through the GitLab API with
last_commit_id, so the commit is rejected if someone changed the file after I read it.
Verify by data, not by status
Argo CD showing Synced only means the ConfigMap matches git. Whether vmagent has re-read it is a separate question. So I checked the age of the samples:
time() - timestamp(up{job="<job>"}) # kept targets reset every scrape
count(up{instance=~".*<domain>.*"}) # removed ones disappear after the lookback window
Series leave instant queries only after the staleness window, about five minutes, so the alerts resolved with that delay. Active alerts went from 52 to 43, and critical alerts from 8 to 4.
Rollback was one git revert, and Argo CD would restore the jobs.
Takeaways
- Monitoring of a service often lives outside the service. Add a step to the decommission checklist: grep the whole GitOps repository for the service’s domains — probes, egress allow-lists, DNS records.
- Noise is not harmless. A channel full of known-false criticals teaches people to ignore criticals.
- “Is it dead?” is answered by DNS, the API and the owner, not by a failing probe.
Syncedis about git, not about the running process. Verify by sample age.