Moving Vault from Shamir to transit auto-unseal, and the four traps on the way

At Optimacros our Kubernetes Vault ran as a three-node HA raft cluster on Shamir, with no auto-unseal. That meant every pod restart came back sealed and needed three key shares typed in by hand. Over two weeks it happened five times unattended and four more times during a node-patching run. Each seal froze the Vault Secrets Operator behind it: up to 53 of 55 synced secrets stuck for days, once unnoticed for 32 hours.

It also blocked other work. Node patching cost three or four manual unseals every time, unattended reboots were out of the question, and one Vault pod could not even be moved off a stale revision without someone standing by with key shares. There was no bug to fix: the estate simply had no KMS or transit provider to unseal against.

The design

A small dedicated VM now runs a second Vault in Docker, with raft storage, as a transit provider. It holds no application secrets, only the transit key that unwraps the cluster’s master key.

• TLS with a self-signed CA mounted into the Vault pods and pinned via tls_ca_cert. No tls_skip_verify: the unseal path must not be open to a man in the middle.
• Authentication with a periodic orphan token whose policy allows exactly two paths, encrypt and decrypt on one transit key. It reaches the pods as an out-of-band Kubernetes Secret, never through Git. Vault renews it on its own.
• The transit VM unseals itself on boot from a locally stored key file, otherwise a reboot of that one host would become a cluster outage.

The cluster side went in through GitOps, followed by a one-time seal migration. Before migrating we took a raft snapshot and kept the original Shamir shares.

Trap 1: -config twice

The official image’s entrypoint appends -config=/vault/config when the first argument is server. I added my own -config=<file>, so Vault loaded the config twice and died with bind: address already in use on a port that was demonstrably free. The fix is just command: server.

Trap 2: the token variable

The token rides on the standard VAULT_TOKEN. There is a family of VAULT_TRANSIT_SEAL_* variables for the key name and mount path, which makes a VAULT_TRANSIT_SEAL_TOKEN look plausible. It is not what Vault reads.

Trap 3: app-of-apps hid the change

Argo CD reported the Vault application as Synced at the new revision, yet the live config had no seal stanza. The Application object itself is rendered by a parent app, and the parent was still on the old revision. Refresh the parent first, then the child.

Trap 4: -migrate on every peer

This was the important one. After the pods restarted with the new seal stanza, Vault logs that it has entered seal migration mode and will not auto-unseal even with an autoseal configured. The migration completes only when every peer has been unsealed with -migrate and its Shamir shares. One migrated node alone leaves raft without a quorum and the cluster without a leader. You feed shares to all three, threshold times each.

The acceptance test

The whole point was one check: delete a Vault pod and issue no unseal command. The new pod logged unsealed with stored key and joined the cluster by itself. That was the first unattended restart that did not need a human.

What it costs

• The transit VM is now a hard dependency for Vault startup. Running pods keep serving if it is down, but any restarted pod stays sealed. Its self-unseal, snapshots and monitoring matter as much as Vault’s.
• Secret sync dropped to zero during the migration and did not recover on its own. Bumping refreshAfter on the synced objects woke it.
• Keep the Shamir shares. They remain valid for migrating back.

Takeaways

• Shamir without auto-unseal turns every restart into a manual incident. If you patch nodes, you need auto-unseal.
• A transit provider needs least privilege: one key, two paths, pinned TLS.
• Seal migration is cluster-wide. Run -migrate on every peer, or nothing elects a leader.
• In app-of-apps, Synced on the child means nothing while the parent is stale.
• Accept the change only after an unattended pod restart unseals by itself.