Moving a production cluster from Longhorn to LINSTOR — and the expectation that did not come true

The cluster behind Optimacros’ marketing sites runs on self-managed kubeadm VMs, with Longhorn for persistent volumes: a MySQL InnoDB Cluster, a three-node Vault, Prometheus, Grafana and Alertmanager. In August I moved all of it to LINSTOR (Piraeus) with DRBD underneath. This is the story of why, how, and what I got wrong.

Not for throughput

The disks were idle 96% of the time. MySQL had zero slow queries, zero innodb_log_waits and a 99.99% buffer-pool hit rate. Throughput was not the reason.

The reason was operational dynamics, measured before anything changed:

• creating a PVC until Bound: 21 seconds;
• first attach: 14 seconds;
• moving a volume between nodes: 46 seconds, plus a Multi-Attach error;
• write latency: 0.16 ms on a local disk versus 12 ms on a Longhorn volume.

Fewer moving parts in the data path and local-disk latency were worth a migration.

Checking the risky part first

DRBD is a kernel module, and that is the one new failure class LINSTOR brings: Longhorn lives entirely in user space, DRBD does not. So before any plan I built DRBD 9.3.3 against the node kernel using exactly the loader image Piraeus would use in production, loaded and unloaded it on one node. Secure Boot turned out to be irrelevant (legacy BIOS boot), and the Ubuntu repos had drbd-utils but no drbd-dkms — which did not matter, because the Piraeus loader DaemonSet builds the module itself, choosing an image by the node’s OS.

The remaining risk was written down plainly: every kernel update rebuilds the module, and a failed build leaves a node without storage. The mitigation is that the loader builds at pod start, so a failure shows up immediately after a reboot, not days later — and kernels are patched one node at a time anyway.

Sizing with a reason

127 GiB of data with two replicas is 254 GiB across the cluster. I asked for 200 GiB per worker — 3.2× — for three concrete reasons: Longhorn snapshots made a 30 GiB Prometheus volume occupy 68 GiB on disk; both storage systems would run side by side during the migration; and replicas never land evenly (one worker had 51 GiB free while another had 156). The disks came as separate devices; LINSTOR takes them whole into an LVM thin pool.

Least critical first, different method each time

Longhorn stayed installed as the rollback point, and volumes moved one at a time from least to most critical:

• Grafana (a Deployment): rsync between two volumes, then existingClaim in the chart values.
• Prometheus: rsync exactly once into a temporary volume kept with Retain, then unbound and re-bound under the name the StatefulSet expects.
• Vault and MySQL: no copying at all. Raft and InnoDB clone rebuild a member from its peers, so it was enough to delete one member’s PVC at a time and let it come back on the new storage class.

What went wrong

Pods that had survived a DNS change the day before still held the resolver of a component that no longer existed. Vault’s Raft kept failing elections on no such host, and MySQL could not form a group. Only recreating those pods fixes that, and MySQL had to be brought back with rebootClusterFromCompleteOutage and its members re-added by cloning.

And one mistake was mine: I deleted the MySQL StatefulSet with --cascade=orphan without saving the manifest first. The operator then refused to recreate it around pods with no owner reference, and I rebuilt the StatefulSet by hand from a live pod’s spec. The marketing database was unavailable for a few minutes while the primary was recreated; the sites themselves kept answering throughout.

The honest result

Everything moved the same day, linstor-r2 (LVM thin, two replicas) became the default class, and Longhorn was removed completely after that. But the expectation that the Multi-Attach errors would disappear did not come true: they reproduce identically on both systems, because that is how Kubernetes handles RWO volumes, not a Longhorn defect. The volume move got only modestly faster — 14 seconds instead of 19 in the re-measurement. The latency win was real; the attach win was mostly a myth.

Takeaways

• Measure the operations you want to improve before migrating — and re-measure after. It is the only way to notice that a promised win was never real.
• Validate the new failure class first. For DRBD that is the kernel module, built by the same image you will run.
• Different workloads deserve different move methods; clustered databases can often rebuild themselves without a copy.
• Before --cascade=orphan, always save the manifest.