The "disk errors" were an OOM: how Longhorn volumes lost their backend

At Optimacros one node of a self-managed Kubernetes cluster “flapped”. Pods on it got EIO, ext4 remounted read-only, a MySQL datadir was corrupted, a Vault peer sealed, and kubelet briefly went NotReady. Afterwards everything looked fine again: the node was Ready, Longhorn reported every volume as healthy.

The tempting conclusion

I/O errors on block devices, a filesystem going read-only, a database with a damaged datadir — every symptom points at storage. The natural next steps are “replace the disk” or “patch the kernel”. Both would have cost time and fixed nothing.

What stopped me was that kubectl could not explain the event. “Healthy now” is not an explanation of what happened at 03:00, so I went to the host.

What dmesg said

A privileged debug pod on the node (kubectl debug node/... --profile=sysadmin, chroot into the host) and a grep through dmesg gave the real order of events:

1. global_oom, constraint=CONSTRAINT_NONE. The whole node ran out of memory. The OOM killer culled many processes, among them longhorn-instance-manager and the CSI node driver.
2. About two minutes later: EXT4-fs error ... unable to read itable block and JBD2: I/O error ... journal superblock on the Longhorn block devices.

The order matters. The instance-manager is the block backend for every Longhorn volume on that node. Kill it, and in-flight I/O on all of those volumes returns EIO; ext4 remounts read-only; whatever database was writing at that moment gets damaged. The disk errors were a symptom of the OOM, not a disk fault.

CONSTRAINT_NONE is the important detail. It means a node-global OOM, not a single container hitting its own limit. That is a capacity and QoS problem.

Why the storage backend was the victim

Two things lined up.

• Many pods ran as BestEffort, with no memory requests at all. The scheduler only counts requests, so it happily packed the node beyond its real RAM.
• The instance-manager ran as Burstable with no memory request. It had a longhorn-critical PriorityClass — but PriorityClass affects scheduling and preemption, not the kernel’s OOM score. Only a memory request lowers oom_score_adj.

So when memory ran out, the kernel saw a large, low-priority-looking process and killed the one thing every volume on the node depended on.

The fix

• Give the instance-manager a memory request, ideally equal to the limit so it becomes Guaranteed, through Longhorn’s guaranteed instance-manager settings.
• Remove BestEffort from real workloads: set memory requests that reflect actual usage so the scheduler stops overcommitting nodes.

Changing the instance-manager’s resources restarts it, which briefly detaches that node’s volumes, so I did it node by node in a window. Diagnosis itself was read-only — just remember to delete the node-debugger pod afterwards.

Takeaways

• EIO on Longhorn devices right after an instance-manager death is the signature of this failure. Read host dmesg before you order a disk.
• constraint=CONSTRAINT_NONE in an OOM line means the node, not the pod, ran out of memory.
• PriorityClass protects scheduling, not survival under OOM. For that you need memory requests.
• BestEffort pods are not free: they spend headroom the scheduler cannot see, and the bill lands on whatever the kernel picks to kill.