Kubernetes 1.32 to 1.37 in one day, five hops, without the kubespray that built the cluster

A bare-metal cluster at Optimacros was on Kubernetes 1.32.8, a branch already past end of life. The plan was to take it to 1.37 — five minor hops, because kubeadm never skips a minor:

1.32.8 → 1.33.13 → 1.34.12 → 1.35.9 → 1.36.5 → 1.37.1

What we were working with

The cluster had been built with kubespray, and the inventory was gone. The binaries lived in /usr/local/bin, not owned by any package, with no Kubernetes apt repository configured — so apt install kubeadm=… was not an option; every hop meant swapping binaries downloaded from the official release site. Three stacked masters, four large workers, an external etcd under systemd, Calico from kubespray’s manifests, kube-proxy in ipvs mode.

The workers held 124 local-path volumes for test environments with no backup. A drain would not move them anywhere, so the workers were upgraded in place, without draining. And the alerting delivered to a null receiver, so the whole operation ran with eyes on dashboards, not on pages — a decision written down before starting, not discovered later.

Prepare so that only one thing changes at a time

The day before, the cluster got:

• a deprecated-API audit with pluto against 1.37 — zero hits in live objects and in all 17 Helm releases;
• an etcd defragmentation: the database file was 743 MB with 73 MB of data — 90% fragmentation, never defragmented since install. One member at a time, leader last, under a second each: 82 MB;
• an etcd snapshot, verified on a copy, plus config backups from all seven nodes and an export of cluster objects;
• an OS release upgrade as a separate window, so that the Kubernetes hops would be the only variable.

The rule for components: upgrade each one before the hop its current version does not support, and re-check the compatibility matrices before every hop, because they change.

The hops

Each hop followed the same script: new kubeadm and kubelet with checksums verified, kubeadm upgrade apply on the first master, upgrade node on the other two, then workers one by one — cordon, upgrade, kubelet, Ready, uncordon, about ten seconds each. Acceptance after every hop: nodes, etcd, kube-system, storage, Argo CD, the database operator, Gateway routes, a policy-engine smoke test and a fresh-pod-to-fresh-pod network test with a timer.

Hops one and two took about ten minutes each. All five were done the same day.

Traps that were not in any plan

• Calico from kubespray needed upstream CRDs and ClusterRoles, not just new images — kubespray’s RBAC was a strict subset. And the new version looped on an XDP program it could not remove, which made fresh pods unreachable for up to 42 seconds until XDP was switched off.
• --apiserver-count was removed in kube-apiserver 1.35. kubespray puts it into the kubeadm config, so it had to be deleted before upgrade apply. Checking flags against the new image’s --help before each hop caught it.
• Longhorn 1.11 requires Kubernetes 1.34 or newer, while 1.10 only supports up to 1.33. So Longhorn had to go up right after the 1.34 hop, strictly one minor at a time.
• CoreDNS is outside kubeadm in kubespray clusters — the addon phase is disabled — so it stays on its old version unless updated by hand.
• The kubelet reads kubespray’s own config file, not the one kubeadm upgrade node writes. clusterDNS has to be checked in the right file after every hop.
• Argo CD caches the cluster version. After each hop its application controller kept rendering Helm with the old --kube-version, and a chart that required 1.34+ would not render. A restart of the controller after every hop fixed it.
• etcd 3.6 is a one-way door: the minimum starting point is 3.5.32, a v2-store check must pass on every member, and once the last member is on 3.6 the only way back is a snapshot.
• On a separate front, moving ingress to Envoy Gateway exposed a kube-proxy strictARP default that made every node answer ARP for LoadBalancer IPs.

The gate we lifted on purpose

The plan had a hard gate before 1.37: Longhorn and the policy engine had to list 1.37 in their support matrices. On the day, they did not. We lifted the gate deliberately and recorded why: every out-of-matrix component used only stable APIs and passed a functional smoke test on 1.37 — a PVC write, webhook and generate rules, routes, fresh-pod networking. Jumping to a brand-new Calico x.0 release that restructures its DaemonSet looked like a bigger risk than running the previous one slightly outside its matrix.

Takeaways

• Without the original installer, kubeadm by hand is fine — but write the procedure once and run it identically on every hop.
• Upgrade components before the hop that breaks them, and re-read the matrices every time.
• Defragment and snapshot etcd before you start; it is cheap and it was 90% fragmented.
• Check removed flags against the new binary before upgrade apply.
• Restart anything that caches the cluster version — Argo CD in particular.
• A gate can be lifted, but only with a written reason and a functional test, never silently.