Helm says "another operation is in progress" — and nothing is running

One morning at Gambchamp the deploy jobs for two backend services went red at the same time. Both failed with the same line:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Nobody was deploying anything. No pipeline was running, no one had a terminal open against that namespace. Yet Helm insisted that something was in progress.

The first guess, and why it was wrong

Two services failing together usually smells like a shared dependency: the registry, the cluster API, the runner. We checked those first. The API answered, images pulled, other releases in other namespaces deployed fine. The failure was specific to these two releases — and they had one thing in common: they are deployed by the same pipeline group into the same namespace.

What Helm actually remembers

Helm keeps the state of every release revision as a Secret in the release namespace. The status lives in a label: deployed, superseded, failed — or pending-install, pending-upgrade, pending-rollback. A pending-* revision is Helm’s lock. While it exists, any new helm upgrade for that release refuses to start.

helm history showed exactly that: the latest revision of both releases was pending-upgrade, and both had been created at the same minute.

That timestamp told the story. An earlier deploy had been cancelled (or timed out) in the middle of helm upgrade --atomic. --atomic promises an automatic rollback on failure — but only if the Helm process lives long enough to perform it. Kill the job mid-flight and the rollback never happens. The lock stays behind, and every following deploy hits it.

The fix

Release the lock by rolling back to the last revision that actually succeeded:

helm history <release> -n <ns>          # find the last "deployed" revision
helm rollback <release> <rev> -n <ns>   # returns to it and clears the pending lock

After the rollback the CI retry went through normally. Since two releases froze at the same minute, we did not want to fix them one by one and miss a third. One query lists every stuck release in a namespace:

kubectl -n <ns> get secret -l owner=helm -o json \
  | jq -r '.items[] | select(.metadata.labels.status | test("pending"))
    | "\(.metadata.labels.name) rev=\(.metadata.labels.version) status=\(.metadata.labels.status)"'

Takeaways

• another operation is in progress almost always means a pending-* revision. Start with helm history, not with the network.
• --atomic is not a guarantee. It only rolls back if Helm finishes; a cancelled job leaves the lock behind.
• Several releases stuck with the same timestamp are one incident — an interrupted batch deploy or a dead runner. Sweep the namespace instead of fixing releases one at a time.
• Roll back to the last deployed revision rather than deleting release Secrets by hand: it is reversible and keeps Helm’s history consistent.