← All posts

I moved one NVMe to another slot, and GPU passthrough stole another VM's disk

After a hardware window two VMs on a Proxmox host refused to start. One error looked like a storage fault. The real cause: a VM passed its GPU through by PCI address, the bus got renumbered, and vfio grabbed the NVMe that held the other VM's volume group.

My home lab runs on a single Proxmox host. During a hardware window I moved an NVMe drive to a different slot and added a second GPU. The host came back fine, but two VMs that are set to start on boot stayed stopped:

  • the media VM: start failed: QEMU exited with code 1;
  • the GitLab VM: can't activate LV '/dev/<vg>/vm-...-disk-0': Cannot process volume group <vg>.

Without GitLab, nothing in the lab could push or build. The second error read like a dying disk, and that is where the first instinct goes. The NVMe was perfectly healthy.

Two failures, one cause

The media VM passes its GPU through to a guest, and it does that by PCI address:

hostpci0: 0000:02:00

Moving cards between slots renumbered the bus. The kernel log of the previous boot and the current one tell the story:

journalctl -k -b -1 --no-pager | grep -E "pci 0000:0[23]:00\.0: \["
# pci 0000:02:00.0: [1002:67df]   <- the GPU
journalctl -k -b 0  --no-pager | grep -E "pci 0000:0[23]:00\.0: \["
# pci 0000:02:00.0: [126f:2263]   <- now an NVMe controller
# pci 0000:03:00.0: [1002:67df]   <- the GPU moved here

The rest followed from the start order. The media VM booted first. vfio-pci took 02:00.0 away from the host’s nvme driver, because that is the address the VM config named. The physical volume on that NVMe disappeared from the host, so the volume group for the GitLab VM could not be activated. And the media VM itself failed anyway, because it got a disk controller instead of a GPU.

The VM that owns a passthrough device boots first and silently takes whatever now sits at that address. That can even be a device another VM’s storage lives on.

The fix

First I made sure 03:00 really was the same GPU: its vendor:device ID appears exactly once on the new bus (the second card is from a different vendor and is used by the host). Then:

cp /etc/pve/qemu-server/<vmid>.conf /root/<vmid>.conf.pre-pci-fix
qm set <vmid> --hostpci0 0000:03:00

D=/sys/bus/pci/devices/0000:02:00.0
echo -n > $D/driver_override
[ -e $D/driver ] && echo 0000:02:00.0 > $D/driver/unbind
echo 0000:02:00.0 > /sys/bus/pci/drivers/nvme/bind

pvscan --cache && vgchange -ay <vg>
qm start <gitlab-vmid> && qm start <media-vmid>

The order matters: the NVMe goes back to the host driver, the volume group gets activated, and only then do the VMs start, GitLab first.

Details that cost time

  • The NVMe came back under a different device name (nvme0n2 instead of nvme0n1): the controller reports odd namespace IDs. LVM finds physical volumes by UUID, so that is harmless. Never hardcode /dev/nvme* names anywhere.
  • Cannot reset device ...:00.1 at VM start is the GPU’s HDMI audio function. The GPU itself reset fine. Not an error.
  • Before blaming a power loss, check the journal for Power key pressed short. A short press is a clean poweroff.

The durable fix

Pinning a new address only lasts until the next hardware change. Proxmox has Resource Mappings (Datacenter → Resource Mappings → PCI): a named mapping bound to the device’s vendor and device ID, and the VM refers to it as hostpci0: mapping=<name>. Then a renumbered bus fails safe instead of handing over a random device.

Takeaways

  • Passthrough by PCI address is a contract with the slot layout. Any PCIe change can break it, and it breaks loudly only for the VM that owns it.
  • An LVM error is not proof of a bad disk. Ask first whether the host still owns the device: lspci -nnk shows which driver is bound.
  • Compare kernel enumeration across boots with journalctl -k -b -1 vs -b 0. That is the fastest way to see what moved.
  • When two things fail after one change, look for the single cause before treating them as two incidents.