Vault kept sealing itself. The cause was a 15-minute VM freeze below the cluster

At Optimacros, Vault in one of our bare-metal-style Kubernetes clusters sealed itself four times in three weeks, each time on a different pod. Every time, the Vault Secrets Operator froze behind it and 53 of 55 synced secrets stopped updating with 503 Vault is sealed. Each occurrence was handled as its own incident and fixed by hand.

The detail that changed the investigation: the Vault pods were not restarting. restartCount stayed at zero. They were being recreated with new UIDs, and always in the same night window.

The chain

The trigger turned out to be below the cluster. The hypervisor stuns the VM during VM-level backups, and Kubernetes did the rest:

hypervisor freezes the VM (13-15 min, at night)
  -> liveness probes time out
     -> Longhorn declares the engine dead after 8-10 s
        -> instance-manager pod recreated, iSCSI sessions drop
           -> every pod with a Longhorn volume on the node is recreated
              -> Vault comes back as a new pod -> Shamir -> sealed
                 -> secret sync hits 503 and stays stuck

How I proved the VM was frozen

Slow disk and memory pressure were the obvious suspects, and both were ruled out by data:

• journald timestamps. Kubelet entries stamped 21:27 were delivered to journald at 21:42. The process wrote nothing for 15 minutes and then flushed its buffer. Prometheus had a matching hole in its own scrape data.
• dmesg at the same moment: mass EXT4-fs unmounting, attempt to access beyond end of device, then a new iSCSI initiator session. Volumes detaching and reattaching.
• The disk was idle. IO time and iowait were near zero during the gap. The I/O spike came after, on reattach.
• Memory never got close to the wall: minimum available memory over seven days was 48 to 71 percent.

The decisive test was a gap analysis across 14 days of journald on every node: print every place where the OS wrote nothing for more than ten minutes. Only two of seven nodes ever froze, for 13 to 15 minutes each time, and the instance-manager was recreated right after each thaw. Those were exactly the nodes with VM backups enabled.

One more lesson from the same data: steal time and freezes are different things. A control-plane node sharing the same oversubscribed hypervisor had as much steal as a frozen worker, yet never froze. Backups are enabled per VM; steal is a property of the physical host. And a 15-minute stun is pathological. A healthy snapshot stun is under a second.

What we could fix from our side

The root cause belonged to the virtualization team, so it was escalated with exact timestamps. In the cluster:

1. Longhorn engine-replica-timeout from 8 s to 30 s, its maximum, applied without recreating any instance-manager.
2. The secret operator now talks to the vault-active Service. The plain Service publishes not-ready addresses, so a sealed peer stayed in rotation and about a third of calls hit it. vault-active selects only the unsealed leader.
3. Alerts on the actual chain: instance-manager recreated, node unresponsive like a VM freeze, Vault pod not ready.

I deliberately did not raise the instance-manager liveness timeout: changing it recreates instance-managers on every node at once, and no timeout survives a 15-minute freeze anyway.

The uncomfortable caveat

Excluding the nodes from VM backups looked like the quick fix, but at that point Longhorn had no backup target configured. The harmful VM backups were the only backups that existed, so they had to stay until a real backup target was in place.

Takeaways

• Restart and recreation are different signals. A new pod UID in a fixed window points below the pod.
• A journald gap analysis is a cheap, decisive test for a stunned VM.
• Rule out disk and memory with numbers before you blame them.
• Steal is a host property; freezes come from per-VM operations like backups.
• Longer timeouts are mitigation for short stalls, not a fix for a 15-minute freeze.