"Vault keeps sealing itself" — no, it just never unseals

At Optimacros the self-managed Kubernetes cluster runs HashiCorp Vault as a three-node HA raft cluster. One morning one Vault pod showed Ready 0/1, and it had been that way for a long time. vault status said Sealed: true, Unseal Progress: 0/3. The way it was reported to me: “Vault keeps sealing itself”.

Why nobody noticed

Raft needs a majority. With three peers, two are enough, so secrets kept flowing and every application that read from Vault was fine. This is the dangerous part: sealed-but-quorum-holds is a silent state one restart away from an outage. If a second peer restarts, the cluster loses quorum and every secret read fails.

Nothing was sealing it

The phrase “keeps sealing” suggests an active process — a bug, a policy, a misconfiguration that re-seals Vault. There was none.

The seal type was Shamir, with no auto-unseal. With Shamir, Vault starts sealed after every process restart and waits for a human to provide a threshold of key shares. Nothing re-seals it; each restart simply leaves it sealed, and nothing brings it back.

The tell was in the pod itself: restartCount=0 and a fairly old pod age. It sealed once, when it was (re)created, and was simply never unsealed. The restarts behind that were driven by an unstable node: every time a Vault pod landed on it and the node blipped, that pod came back sealed.

Unsealing

The listener had TLS disabled inside the cluster and VAULT_ADDR was set in the pod, so the CLI worked directly through kubectl exec — no HTTP API workaround needed:

kubectl -n <ns> exec <vault-pod> -- vault operator unseal   # three times, three distinct shares
kubectl -n <ns> exec <vault-pod> -- vault status | grep -iE 'Sealed|HA Mode|Raft Applied'

After the third share the node unsealed, rejoined raft through retry-join and caught its applied index up to the leader.

The real fix

Unsealing by hand restores service; it does not fix anything. The real fix is auto-unseal — transit or a cloud KMS — configured in the Vault Helm values, so a restarted pod unseals itself. We moved to transit auto-unseal shortly after. Until then: keep Vault pods off unstable nodes, and make sure a VaultSealed alert actually fires.

A note on keys

Unseal shares never go into chats, tickets, logs or commits. If they are exposed, vault operator rekey is mandatory, and it must be run by a person in a secure terminal: rekey prints the new shares, and routing those through the same exposed channel defeats the purpose. Also keep in mind that shares passed as command arguments briefly appear in the process list; stdin is the safer path on shared machines.

Takeaways

• “Keeps sealing” is usually “never unsealed”. Check restartCount and pod age before hunting for a re-seal mechanism.
• Shamir without auto-unseal turns every node blip into manual work and a hidden loss of redundancy.
• Sealed-but-quorum-holds is invisible to users and must be visible to monitoring.
• Auto-unseal is the fix; unseal keys are break-glass, not an operational procedure.