An error budget in VictoriaMetrics: recording rules, burn-rate alerts and one honest panel

At Gambchamp we needed an error budget for the tier-1 services of one product platform: something you can see at a glance and alert on. Someone suggested a chat slash command that would print the budget. We chose a Grafana panel instead — the data was already in VictoriaMetrics, and a new bot means a new service, a new token and a new thing to keep alive.

The SLI and the targets

The SLI was the simplest honest one: the share of non-5xx responses. Each service got its own target, written down before any rule existed. The rules went into a separate group for this platform, so the generic alert rules stayed untouched.

Three recording rules

• slo:target_error:ratio{app,cluster} — the allowed error ratio (1 − SLO) per service, built with label_replace(vector(x), ...) joined by or;
• slo:errors:ratio_rate30d — the real error ratio over 30 days, written as ... or 0 * total so a service with no errors shows 0 instead of an empty series;
• slo:error_budget:remaining_ratio30d = 1 - ratio30d / target — 1 means a full budget, 0 means it is spent, below 0 means over budget.

Because the target carries the same {app, cluster} labels as the error series, every comparison is a clean one-to-one join.

Burn-rate alerts

Multi-window, multi-burn-rate, the classic SRE-workbook setup:

• fast: 14.4× burn over 5 minutes and 1 hour, critical, for: 2m;
• slow: 6× burn over 30 minutes and 6 hours, warning, for: 15m.

Alongside them, plain RED symptom rules: p99 latency above one second but only with a traffic floor (an idle service’s single slow request is not an incident), “no traffic” for tier-0 services, and “metric absent”.

The panel

One row: a stat per service plus a trend, thresholds red below 0, orange below 0.2, green above. It went into the code-first dashboard generator, not the Grafana UI — UI edits are overwritten on the next deploy.

The budget that was −26% and fine

Right after launch one payment service showed a budget of −26%. It looked alarming. It came entirely from one launch-day spike: about 2,100 deposit errors over 9.5 hours, weeks earlier. The 30-day window simply included it. It would heal itself once that day rolled out of the window. The burn-rate alerts, which look at short windows, stayed quiet the whole time — which is correct.

Small things worth knowing

• A recording rule built on other recording rules appears a few evaluation cycles later: the inputs have to be written and become queryable first. That is not a bug.
• vmalert needs -remoteWrite.url, or recording results go nowhere.
• The dashboard reads the ALERTS series from the TSDB, not from any Alertmanager, so splitting notification channels later did not affect it.

Takeaways

• Start from metrics you already have; an SLO does not need a new service.
• Make the “no errors” case return 0, not an empty series, or every downstream rule breaks quietly.
• Use burn rate for paging and the 30-day budget for planning. They answer different questions.
• Right after launch the 30-day window is biased by launch-period errors. Know when it rolls out before you panic.