"Kernel drift" was uptime drift: patches landed, nothing rebooted

At Optimacros a seven-node self-managed Kubernetes cluster was running three different kernels. The earlier reading, written down in our environment notes, was “unfinished patch waves”. The nodes on the oldest kernel were treated as prime suspects in an unrelated incident.

That reading was wrong, and it sent the investigation down a false trail.

What was actually going on

Patches had reached every node identically. unattended-upgrades was enabled across the fleet:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade  "1";

Unattended-Upgrade::Automatic-Reboot was not set, and its default is false. So new kernels were installed on every node and never activated. The only thing that differed between nodes was the time since their last reboot:

Running kernel	Newest installed	Nodes	Uptime
6.8.0-101	6.8.0-137	3	152 days
6.8.0-124	6.8.0-137	2	69 days
6.8.0-134	6.8.0-137	2	29 days

The “drift” was simply a record of when each node had last been restarted for some unrelated reason. libc6 had been updated everywhere too, which meant glibc fixes were not applied in any running process, and /var/run/reboot-required was present on all seven nodes.

Two things looked alarming and were not. /boot was at 11% with only two kernels present; old ones were autoremoved. And reboot-required.pkgs listed about ten kernel packages, which reads like “ten kernels installed” — it is actually the accumulated history of packages that requested a reboot since the last one.

The check that answers “are we patched?”

Comparing running kernels between nodes is the wrong question. The right one is per node: running versus newest installed, plus uptime.

uname -r
ls -1 /boot/vmlinuz-* | sed 's|.*vmlinuz-||' | sort -V | tail -1
awk '{print int($1/86400)}' /proc/uptime
[ -f /var/run/reboot-required ] && echo reboot-required

The fix

A rolling reboot of all seven nodes, one at a time — drain, reboot, uncordon, wait for storage to be healthy — followed by a second pass for 31 pending userspace updates. Afterwards every node ran 6.8.0-137, reboot-required was gone and apt reported nothing left to upgrade. kubelet and containerd were untouched.

Traps found along the way

• apt-mark showhold was empty. Nothing protected kubelet, kubeadm or containerd from apt upgrade. Always list the upgrade set first; a Kubernetes component sneaking in breaks the cluster through version skew.
• The kubeconfig pointed at one control-plane node directly, with no HA endpoint. Rebooting that node killed kubectl mid-procedure while the cluster itself was fine. Control-plane HA is not HA of access to the control plane.
• Packages stay “kept back” simply because apt upgrade never installs new dependencies. Check apt-get -s full-upgrade and proceed only if it removes nothing.
• Do not automate reboots while Vault has no auto-unseal. Every reboot that moves a Vault pod leaves it sealed, so unattended reboots create silent outages.

Takeaways

• Different kernels across nodes usually mean different uptimes, not different patch levels.
• “Installed” is not “running”. Patch compliance is a per-node comparison of the two.
• Unattended upgrades without reboots are half a patching policy.
• Write down what you know, and revisit it. A wrong line in the environment notes cost more time than the reboots did.