Linux and hosts
The layer under the cluster: kernels and patching, memory pressure and OOM, virtualization and node maintenance.
7 posts
- multipathd hijacked a Longhorn volume: "mount point busy" while nothing was mountedA pod sat in Init for five days, 2413 mount retries, and a full power cycle did not help. multipathd had built a map over the Longhorn iSCSI device and held it open.
- The loading screen was a retry loop, not a hang — and the reboot was never neededA game on my Linux workstation froze on its loading screen, and the known fix was a full reboot. Five seconds of /proc counters showed a hot retry loop that a single SIGTERM could end.
- NotReady for six hours, and the node was alive the whole timeEvery signal said the node was dead. It was thrashing on memory, and the obvious fix — delete the node and force-detach the volume — was one command away from corrupting a database.
- "Kernel drift" was uptime drift: patches landed, nothing rebootedThree different kernels across seven nodes looked like unfinished patch waves and sent an investigation down a false trail. Every node had the same patches; only uptime differed.
- Vault kept sealing itself. The cause was a 15-minute VM freeze below the clusterFour Vault seals in three weeks, each treated as a separate incident. A journald gap analysis showed the hypervisor stunning the VM during backups, and Longhorn turning that into a cascade.
- Updating the OS on seven Kubernetes nodes without anyone noticingA rolling Ubuntu point update across a cluster that holds all its own state. The pre-flight checks turned a scary job into a boring one, and Longhorn decided the pace.
- The "disk errors" were an OOM: how Longhorn volumes lost their backendEIO, read-only ext4, a corrupted MySQL datadir and a sealed Vault looked like a failing disk. Host dmesg showed a node-wide OOM that killed the Longhorn instance-manager first.