Vault Secrets Operator went quiet after a seal — and a restart did not wake it

At Optimacros the bare-metal Kubernetes cluster gets its application secrets from Vault through the Vault Secrets Operator (VSO). One day I noticed that 38 of 43 VaultStaticSecret objects had been reporting SecretSynced=False for about two days.

A frozen error is not a retrying error

The condition message was 503 Vault is sealed, followed by a back-off horizon of 4.939 seconds. I polled it several times. The horizon was identical to the nanosecond every time. A controller that is really retrying recalculates that value on every attempt. This one was not retrying at all.

The operator’s logs agreed: nothing since the seal event. The pod was alive, renewing its leader lease and using a few millicores of CPU, but it was not reconciling anything.

Layer one: one sealed node behind a Service

The 503 itself had a simple cause. Vault runs as a three-node Raft cluster, and one of the nodes was sealed. The vault Service load-balances across all pods, so roughly a third of VSO’s reads landed on the sealed node. That also explained the odd mix: some secrets synced, some did not, purely by round-robin luck.

I unsealed the node, and all three Vault pods became ready. The VaultStaticSecret objects stayed False.

Layer two: the reconcile loop had died

Next I tried the two official remedies: a rollout restart of the operator and the force-sync annotation on the stuck objects. Neither changed anything.

This turned out to be a known VSO bug, hashicorp/vault-secrets-operator#840. After a seal and unseal, the per-secret lifetime watchers die, and neither the periodic refreshAfter timer nor force-sync puts the objects back on the work queue. The informer is alive, but nothing re-queues anything.

The wake-up call: change the spec

The informer still reacts to spec changes. So I bumped spec.refreshAfter on each stuck object. That fires a watch event, the reconcile runs, and the object goes to True. VSO does not log successful syncs at info level, so the logs stayed quiet even when it worked; the conditions were the only signal. Once everything was synced, I put the original refreshAfter back with a second patch.

This was safe because the secrets used overwrite: false: the wake-up repaired the status and did not rewrite the existing data.

The real fix is elsewhere

The spec bump is a workaround. In this cluster Vault used Shamir unsealing without auto-unseal, so every Vault pod restart re-sealed a node, and every seal would stall VSO again. The actual remedies were upgrading VSO to a release with the bug fixed and moving Vault to auto-unseal, so that seals stop happening in the first place.

Takeaways

• A condition message that does not change between polls, down to back-off timings, is the fingerprint of a dead work queue, not a slow retry.
• A Service in front of a Raft cluster sends traffic to sealed nodes too. Mixed success across identical objects often means some backends are bad.
• If restarts and force-sync annotations do not help, a harmless spec change goes through the informer and re-queues the object.
• Fix the source: auto-unseal stops the seals, and an upgrade removes the stall.