multipathd hijacked a Longhorn volume: "mount point busy" while nothing was mounted

In my home lab a SonarQube pod sat in Init:0/2 for five days, retrying the same mount 2413 times:

mount: .../globalmount: /dev/longhorn/pvc-... already mounted or mount point busy.

The obvious reading is a stale mount left behind by an unclean reboot. It was wrong, and the proof was simple: the error survived a full power cycle of every node, which is exactly what clears a stale mount.

Following the device, not the message

The message talks about a mount point, so I checked mounts first:

mount | grep pvc-6e83 || echo "not mounted anywhere"
for f in /proc/[0-9]*/mounts; do grep -q pvc-6e83 "$f" && echo "$f"; done

Nothing, in any mount namespace. So the mount point was not occupied. The other half of the message — “busy” — is about the device. One command ended the search:

lsof /dev/sdb
# multipath 429 root 10r BLK 8,16

The vendor and model of the block device were IET / VIRTUAL-DISK — that is Longhorn’s iSCSI target. And multipath -ll showed a map built right on top of it.

Root cause

Longhorn’s v1 engine exports volumes over iSCSI as ordinary /dev/sdX devices. multipathd was running on all four workers with an empty blacklist, treated the volume as a multipath candidate, built a map over it and held the device open. mount(8) reports that as “already mounted or mount point busy”, which sends you looking for a mount that does not exist. It also explains the reboot resistance: multipathd starts early and claims the device again on every boot.

The data was never at risk: e2fsck -n reported the filesystem clean before any change.

The fix

Blacklist Longhorn’s devices by vendor and product — not with devnode "^sd.*", which disables multipath for everything — then flush the map and restart the daemon:

blacklist {
    device {
        vendor "IET"
        product "VIRTUAL-DISK"
    }
}

followed by multipath -f <map>, systemctl restart multipathd, and a pod delete so it could remount. A backup of multipath.conf came first, so the rollback is one copy and one restart.

The important part was doing it on every node. Only one volume had been caught, but the condition was latent on all four workers: any Longhorn volume could have been taken on the next pod restart. Afterwards the volume went degraded while Longhorn rebuilt a replica; that resolves itself, the right move is to wait for healthy.

Takeaways

• “already mounted or mount point busy” with an empty mount output means the device is held. lsof on the block device is the step that ends the search.
• If a reboot does not clear a “stale” state, it is not stale — something recreates it on boot.
• Any node running Longhorn v1 with multipath-tools installed needs a vendor/product blacklist. Add it to node bootstrap and check it after every rebuild.
• Fix the class, not the instance: a problem found on one node is often present on all of them.