HashiCorp Vault
Vault in Kubernetes: auto-unseal, Raft, ACL policies, the Vault Secrets Operator and why a sealed Vault breaks everything downstream.
8 posts
- A production certificate expired and nobody noticed: the front proxy covered for itEvery public host returned 200 while the cluster served an expired wildcard. A proxy with its own TLS hid the outage; the real finding was the missing expiry alert.
- A star in the middle of a Vault path is not a wildcardA dead token took down a secrets operator. Reissuing it exposed an older bug: 27 Vault policies used * mid-path, silently denied what they declared, and hid an over-privilege.
- Alerts that cannot fire: a blind ServiceMonitor and a dead delivery channelA Vault peer stayed sealed for 32 hours and secrets stopped syncing for a week, with a VaultSealed alert in the repo and Alertmanager healthy. Two independent failures hid it.
- Moving Vault from Shamir to transit auto-unseal, and the four traps on the wayEvery Vault pod restart came back sealed and froze secret sync behind it. How we added a transit auto-unseal provider, migrated a three-node raft cluster, and what bit us.
- Vault kept sealing itself. The cause was a 15-minute VM freeze below the clusterFour Vault seals in three weeks, each treated as a separate incident. A journald gap analysis showed the hypervisor stunning the VM during backups, and Longhorn turning that into a cascade.
- Vault Secrets Operator went quiet after a seal — and a restart did not wake it38 of 43 VaultStaticSecrets stuck for two days with a frozen error. Unsealing Vault and restarting the operator did nothing; a one-field spec change did.
- A WordPress login loop that only happened sometimeswp-admin bounced back to the login page with reauth=1, but not always. The cause was random auth salts generated per pod behind a non-sticky load balancer.
- "Vault keeps sealing itself" — no, it just never unsealsA Vault pod sat at Ready 0/1 for days while raft quorum quietly held on two peers. Nothing was sealing it: Shamir seal without auto-unseal leaves every restart sealed.