Storage
Persistent storage for Kubernetes: Longhorn, LINSTOR and DRBD, volume expansion, multipath and migrations without data loss.
6 posts
- multipathd hijacked a Longhorn volume: "mount point busy" while nothing was mountedA pod sat in Init for five days, 2413 mount retries, and a full power cycle did not help. multipathd had built a map over the Longhorn iSCSI device and held it open.
- Moving a production cluster from Longhorn to LINSTOR — and the expectation that did not come trueWhy we swapped Longhorn for LINSTOR/DRBD on a self-managed cluster, how each workload was moved with a different method, and what went wrong on the way.
- Vault kept sealing itself. The cause was a 15-minute VM freeze below the clusterFour Vault seals in three weeks, each treated as a separate incident. A journald gap analysis showed the hypervisor stunning the VM during backups, and Longhorn turning that into a cascade.
- Updating the OS on seven Kubernetes nodes without anyone noticingA rolling Ubuntu point update across a cluster that holds all its own state. The pre-flight checks turned a scary job into a boring one, and Longhorn decided the pace.
- The "disk errors" were an OOM: how Longhorn volumes lost their backendEIO, read-only ext4, a corrupted MySQL datadir and a sealed Vault looked like a failing disk. Host dmesg showed a node-wide OOM that killed the Longhorn instance-manager first.
- A Longhorn volume that would not grow: an attachment ticket pinned to a bad nodeA PVC sat in Resizing forever while Longhorn retried the expansion every five seconds without an error. The volume was locked to one unhealthy node by a stale attachment ticket.