Updating the OS on seven Kubernetes nodes without anyone noticing

My home lab cluster is seven VMs on one Proxmox host with no UPS, and those seven nodes hold everything: Longhorn volumes, a CloudNativePG Postgres, a three-node Vault on Raft and both Cloudflare tunnels that publish my sites. The packages were behind, and I wanted the OS current before touching Kubernetes itself. Updating nodes that carry all the state sounds risky. The pre-flight is what made it boring.

What the pre-flight found

Before running anything, I checked what an upgrade would actually touch:

• No kernel updates were pending, so the reboot would not be for the kernel.
• No Kubernetes components were dpkg packages. kubelet, kubeadm, containerd, runc and etcd were installed as binaries in /usr/local/bin by the original provisioning tool, and there was no Kubernetes apt repository. apt simply could not touch them, so an empty apt-mark showhold was not a problem here.
• No libc, OpenSSH or systemd core among the 48 packages.
• nftables.service was disabled, so upgrading the nftables package could not reload a ruleset and flush Calico’s rules. kube-proxy ran in IPVS mode anyway.
• Zero security updates queued: unattended-upgrades already applied them.

The procedure

One node at a time, canary first. I picked the canary by boredom: the worker carrying only Grafana, a tools app and a single Longhorn volume. The worker carrying a Vault peer, a Postgres member, the Argo CD controller and a replica of both tunnels went last among the workers. Control-plane nodes came after all workers.

For each node: a Proxmox snapshot, kubectl drain, the upgrade with NEEDRESTART_MODE=l, a reboot while still drained, uncordon, and then — the important part — wait until every Longhorn volume is healthy before moving on.

NEEDRESTART_MODE=l was the load-bearing flag. needrestart was unconfigured, so under apt -y it could either hang on a prompt or restart containerd, which kills every container on the node. l means “list only, I decide”. On all seven nodes it reported that no containers needed a restart.

What surprised me

• Draining one node degraded volumes attached to other nodes, because replicas are spread around. One drain left six volumes degraded; the large ones rebuilt over about four minutes on a shared disk. Waiting for healthy between nodes is mandatory, not caution.
• A naive “not healthy” count lies: detached volumes have an empty robustness field and show up as “not healthy”. I had to look at the actual values.
• CloudNativePG switched over by itself whenever the primary’s node became unschedulable. Expect one switchover per such drain, and confirm 2/2 healthy before rebooting.
• AppArmor set reboot-required only after the upgrade, so there was no warning beforehand. Plan a reboot for every node anyway.
• Tunnel replica placement decided whether the sites stayed up. With one replica per tunnel on different workers, the public sites returned 200 throughout. Both replicas on one node would have been a real outage.

Result

48 packages per node, Ubuntu 24.04.4, kernel unchanged, kubelet untouched, all seven nodes Ready, every Longhorn volume healthy, etcd 3/3 with identical raft indexes, Postgres 2/2, Vault 3/3, every Argo CD app Synced and every endpoint answering. The snapshots were deleted after verification.

Takeaways

• Pre-flight first: what exactly will the upgrade touch, and can it touch the cluster’s own components?
• Pick the canary by how little it carries.
• Let storage set the pace. The next node waits for the volumes, not for the clock.
• Check replica placement of anything public before draining.