A Longhorn volume that would not grow: an attachment ticket pinned to a bad node

At Optimacros I needed more room for a MySQL member: its PVC was 6 GiB and I patched it to 20 GiB. The StorageClass allows expansion, Longhorn supports online resize, and an identical PVC on another node had grown in about 40 seconds. This one did not.

The symptom

The PVC stayed in Resizing=True indefinitely. Longhorn’s own objects looked half-done: the volume’s spec.size was already 20 GiB and status.expansionRequired was true, but the engine’s currentSize never moved past 6 GiB. Inside the pod the filesystem was still 6 GiB.

longhorn-manager logged Starting engine expansion from 6→20Gi every five seconds, on the same node, with no error. A loop with no error message is the worst kind: nothing to search for.

Narrowing it down

The identical PVC that expanded fine lived on a different node. That made the problem node-local. The node itself reported Ready, so this was not kubelet; it was the volume’s engine on that particular node. The same node had been the origin of an earlier incident and runs an older kernel, which made it the prime suspect.

The obvious move was to reschedule the pod onto a healthy node and let the engine restart there. Kubernetes refused with a Multi-Attach error: the volume was “already exclusively attached to a different node”.

The deadlock

Longhorn tracks who needs a volume attached through attachment tickets on a volumeattachments.longhorn.io object. Listing the tickets showed the culprit: a volume-expansion-controller ticket that pinned the volume to the bad node.

So the system had tied itself in a knot:

• the expansion needed the volume attached to the bad node;
• the bad node could not complete the engine-level expansion;
• the pod could not move to a healthy node, because the expansion ticket still held the volume on the bad one.

The fix

I removed the stuck expansion ticket with a JSON patch on the Longhorn VolumeAttachment. Longhorn reconciles tickets on its own, so this is not a manual state you have to maintain; the controller re-creates what it needs. With the volume free, Longhorn detached it, completed the pending expansion while it was detached, and re-attached it.

To make sure the fresh engine landed elsewhere, I cordoned the bad node and deleted the StatefulSet pod. It came up on a healthy node, the CSI node plugin ran resize2fs, and df inside the pod showed 20 GiB. Then I uncordoned the node.

One caution for stateful workloads: moving the pod restarts a database member. Check that quorum holds first (2 of 3 survives in MySQL Group Replication), and watch for an errant transaction on the member after the restart, which forces a rebuild by clone instead of an incremental rejoin.

Takeaways

• A retry loop with no error is a clue in itself: something is waiting on a precondition it can never get. Look for what pins it in place.
• When an expansion is node-local, compare against the same operation on another node before blaming the storage system.
• Longhorn attachment tickets are worth listing early; a stale ticket explains both a stuck expansion and a Multi-Attach error.
• Fixing the volume is not fixing the node. Keep stateful volumes off a suspicious node until it gets its own root-cause analysis.