Observability and alerting
Metrics, alerts and SLOs that point at the cause: Prometheus, VictoriaMetrics, Alertmanager, Grafana and the signals that lie.
17 posts
- The certificates expired on schedule. The alert fired for a week — into nowhereA kubeadm cluster lost its control plane exactly one year after install. The expiry alert had been firing for seven days, but Alertmanager only had a null receiver.
- The alert fired and nobody saw it: a cluster label lost in aggregationA node ran at 90-98% CPU for four hours and ended in an OOM kill. The CPU alert did fire, but avg by(instance) dropped the label every route and dashboard selected on.
- The rebalance storm was just a deploy: a threshold that scales with group sizeA Kafka rebalance alert fired at 60 per hour on every release. The metric was summed over 40 consumers and averaged over a sliding hour, so the alert described the past.
- Argo CD said Synced, and the API server had quietly thrown my fix awayMonths of missing metrics, a fix already in git and a green Synced status. The field did not exist in the CRD, so the API server pruned it and nobody complained.
- Three alerting outages from one root: a name where an ID belongedSomeone renamed a Slack channel and alert delivery for a production system died silently. It was the third failure on the same identifier — every earlier fix changed the transport and kept the name.
- Kafka at 94 % disk with 17 hours left, and retention that kept twice what we askedTraffic grew 150x and nobody resized the volumes. Worse, with default segment rolling a 7-day retention really meant 7 to 14 days. How we bought time and fixed the cause.
- A Postgres replica stuck in a pg_rewind loop for 23 hours: rebuild, do not repairAfter a failover a CloudNativePG replica could not rewind: its WAL segments were zero bytes and its data directory a tenth of the primary. Nobody noticed for 23 hours because nothing scraped the operator metrics.
- An error budget in VictoriaMetrics: recording rules, burn-rate alerts and one honest panelHow we built an error budget for tier-1 services from metrics we already had: per-service SLO targets, multi-window burn-rate alerts, a Grafana row — and why a negative budget right after launch was not an incident.
- One p99 alert, five slow routes, five different causesA service-level latency alert named no endpoint. Splitting it per route showed an external gateway, a captcha check, a provider call — and a set of measurement traps that would have sent us the wrong way.
- The lag went to zero and nobody was fixed: a Kafka consumer stall hidden by a group.id renameConsumer lag grew for five days, then dropped to zero and stayed there. It looked self-healed. In fact the consumer was stuck, the metric had vanished, and the alert had resolved itself.
- Profiling a Python memory leak in production with memray — and the outage we caused on the wayA backend OOMed every 10–15 hours. Hourly memray snapshots on the live pod found the leak in library middleware, not the app — after a first attempt took production down for seven minutes.
- Four checks that lied while the system was fineIn one day of platform migrations, four verification steps reported failure while everything was healthy. When one signal disagrees with all the others, suspect the signal first.
- Alerts that cannot fire: a blind ServiceMonitor and a dead delivery channelA Vault peer stayed sealed for 32 hours and secrets stopped syncing for a week, with a VaultSealed alert in the repo and Alertmanager healthy. Two independent failures hid it.
- A GitLab upgrade and two signals that liedA routine minor upgrade of self-hosted GitLab in Docker. A pre-flight check claimed 32 pending migrations, and the container reported healthy while serving nothing.
- Merged, synced, mounted — and still not alerting: vmalert without hot reloadNew Kafka alert rules went through Git and Argo CD into the ConfigMap, yet vmalert never evaluated them. The GitOps pipeline broke on its very last step.
- A Grafana panel that tells you when a pod will OOMTurning a memory trend into a time-to-OOM and time-to-full ETA in PromQL, the two bugs that made it lie, and the leak it caught on day one.
- Five days at 1/3 with every dashboard green: when a CRD status liesA MySQL InnoDB Cluster ran without HA for five days while its status, the pod phase and the readiness gate all said healthy. All three came from one stalled operator.