Metrics absent for four hours: the cluster was alive, its data was stuck in a queue
Three ServiceAbsent criticals fired for a healthy production cluster. A degraded network path had turned remote-write into a FIFO backlog. The most important decision was not to restart the agent.
Three ServiceAbsent criticals fired for a production cluster. In the central VictoriaMetrics, instant queries for that cluster returned nothing at all. Every other alert for it went quiet, because a rule like errors > threshold evaluates an empty series as “no problem”.
From the dashboards, the cluster looked dead. It was not.
Check the product before the metrics
The first step was to look at the service itself, not at its telemetry. The public endpoints answered 200, and all nodes were Ready. The outage was in observability only. That changed the priority: users were fine, so there was time to understand the problem instead of reacting to it.
Ask the agent, not the central database
Each cluster runs vmagent, which scrapes locally and remote-writes over the network to a central VictoriaMetrics in another provider. When the central side is empty, the agent’s own metrics tell you why:
curl -s localhost:8429/metrics | grep -E \
'^vmagent_remotewrite_(pending_data_bytes|requests_total|send_duration_seconds_total|packets_dropped_total)'
pending_data_byteswas growing: the persistent queue had reached 1.1 GB.- Seconds per request (Δ
send_duration/ Δrequests) had gone from 0.46 to 6.6. - Drops were zero, and every response was 2XX.
Nothing was failing. Every request succeeded, just slowly, so throughput fell below the ingest rate. The queue is FIFO, so the central database received older and older data. “Now” stayed empty for about four hours, even though no sample was lost.
A control group instead of a guess
To find which part of the path was slow, I compared similar paths. Another cloud account in the same region degraded the same way. Clusters inside the central database’s own provider did not. A second remote-write target inside the cloud stayed fast. So the slow part was the link between the cloud region and the other provider, not vmagent and not the central database.
One false lead is worth mentioning. New nodes appeared on the central side in the same minute the link recovered. That is a correlation, not a cause. The traffic path never touched those nodes.
The decision was a non-action
The obvious move was to restart vmagent. It would also have been the worst one. The queue lived on an emptyDir. It survives a container restart within the same pod, but a pod restart or reschedule deletes it, and four hours of metrics with it.
So I waited for the link to recover and watched the queue drain. Afterwards I checked continuity over the whole window:
count(up{cluster="<c>"}) # range query, step 15m: no empty steps means no gap
The queue was empty after about four hours, the alerts resolved, and there were zero gaps.
A small trap in my own tooling
My watch script printed pending=0 and “queue empty” while 790 MiB were still queued. A scrape had timed out, the variable was empty, and printf %d turned that into zero. A script has to tell “could not measure” apart from “0”.
Takeaways
absent()cannot tell dead from late. Pair it with an alert on the agent’s backlog (pending bytes growing, seconds per request rising), so the real cause has a name in the first five minutes.- A silent dashboard is not a green one. Rules without
absent()are blind while data is missing. - Know where your buffer lives before you restart anything. A remote-write queue on
emptyDiris lost with the pod. If the link is known to be flaky, use a PVC. - Use a control group. Comparing similar paths turns “the network is slow” into one specific link.