I rebuilt my home Kubernetes cluster on clean kubeadm. Here is what the rebuild exposed
An inherited kubespray cluster rebuilt in place on plain kubeadm in one day. The plan held; the lessons were six things that only ever worked by accident.
My home lab runs a seven-node Kubernetes cluster: three control-plane nodes and four workers, all VMs on one hypervisor. It was built with kubespray, and the kubespray checkout that built it was long gone. Every upgrade since then had been done by hand on top of what kubespray left behind:
- etcd ran as an external systemd service with its own certificates, which
kubeadm upgradenever touches; - the API endpoint was the IP address of the first master, so the “highly available” control plane had a single point of failure;
- CoreDNS had a hand-edited Corefile, and every upgrade had to skip the CoreDNS phase so the edits survived;
- kube-dns, service and pod ranges were kubespray defaults that nobody remembered choosing.
None of it was broken. All of it was a trap for the next upgrade. So I decided to rebuild in place: drop the old cluster, recreate the VMs from a cloud image with the same MAC addresses (so DHCP hands out the same IPs), and build a clean kubeadm cluster on them. GitOps would bring the workloads back, and a Vault raft snapshot would bring the secrets back.
The plan itself held up. It took one long day. What I want to write down is the six things the rebuild exposed, because each of them had been working only by accident.
Backups first, because there is no rollback
An in-place rebuild has no “switch back” button. Once the old VMs are destroyed, the backups are the only way back. So before touching anything I took a Vault snapshot, an etcd snapshot, a dump of every object in the cluster by type, all secrets, a mirror of the GitOps repository with a tag on the exact commit the old cluster was running, and a full image of one master. Two copies, neither on a cluster VM. The two public sites the cluster serves were moved to a plain Docker host for the downtime, behind the same tunnel connector token.
The first trap showed up right here.
Trap 1: a Vault snapshot on a follower is not a snapshot
Error taking the snapshot: incomplete snapshot, unable to read SHA256SUMS.sealed file
I ran vault operator raft snapshot save on vault-0, which is what I had always done. This time vault-0 was a follower. The follower forwards the request to the leader, and the forwarded stream breaks. vault operator raft list-peers showed the leader was vault-2; the same command there produced a snapshot that snapshot inspect could read, and the checksum of the copy matched the file in the pod.
Lesson: take raft snapshots on the leader, and always inspect and checksum the copy you are going to rely on.
Trap 2: the free IP that was not free
The new control plane gets a virtual IP via kube-vip, so losing master1 no longer takes the API down. I needed one free address. Five candidates between the masters and the workers did not answer ping, and ARP for them failed. Free, I thought.
Then I looked at the router’s DHCP pool. All five were inside it. Nothing held them today, but sooner or later the router would lease one to a phone, and the control-plane VIP would start fighting over ARP with it. That is the kind of outage that appears weeks later and looks like a network ghost. The VIP went above the pool, next to the load-balancer addresses.
Lesson: “nobody answers” means nothing until you know who is allowed to hand the address out.
Trap 3: the interface name changed, and the ingress IP silently went away
After the VMs came up I noticed that six nodes had a NIC called ens18 and one had eth0. The cloud-init network config pins set-name: eth0 by MAC, but on first boot udev had already named the interface before that config existed. On one VM the rename back to eth0 won the race; on six it did not. After one reboot all seven were eth0, deterministically.
I fixed it early because kube-vip binds the VIP to an interface by name. What I did not think about was the old GitOps repository. Hours later, with everything green, the ingress IP did not answer at all. The Cilium L2 announcement policy that answers ARP for it matched ^ens18$. No error, no event; the announcement simply had no interface to live on.
Lesson: an interface name is configuration. When the OS changes it, grep the whole repository for the old one.
Trap 4: volumes created during a cold start had no filesystem
Storage is LINSTOR/DRBD on a dedicated disk per worker. On fresh nodes the DRBD kernel module does not exist yet, so the module loader compiles it from source for the running kernel. That takes a few minutes per node.
Every PVC went Pending, then Bound. Then three pods refused to start:
mount: wrong fs type, bad option, bad superblock on /dev/drbd1001
blkid on the node found nothing; the ext4 superblock area was all zeros. The volumes had been created while the satellites were still coming up, and the filesystem was never made. I checked every volume, not only the failing ones: three out of ten had no filesystem. All of them were empty, so the fix was to delete the PVC and the pod and let the provisioner create a fresh, properly formatted volume.
Lesson: on a cold LINSTOR cluster, wait until every storage pool reports Ok before you let anything claim a volume.
Trap 5: a safety policy versus a self-healing GitOps controller
I moved ingress from Cilium’s built-in Gateway API controller to Envoy Gateway. Its chart wants the Gateway API CRDs from the experimental channel (I need TLSRoute for one passthrough route), and my cluster had the standard channel.
The upstream bundle ships a ValidatingAdmissionPolicy, safe-upgrades, that forbids installing experimental CRDs over standard ones. The documented way out is to delete the policy. I deleted it, and the sync failed again. ArgoCD’s self-heal had recreated the policy, because the new chart ships it too, faster than the same sync applied the CRDs. I tried deleting only the binding and applying the CRDs in the same command; the controller still won.
What worked was removing the race instead of trying to win it: scale the ArgoCD application controller to zero for a minute, delete the policy, apply the CRDs, scale the controller back. The policy came back immediately, and now it guards an experimental installation instead of blocking it.
Lesson: you cannot outrun a self-healing system. Pause it.
Trap 6: the connector that only ever existed by hand
The last step was to point the public tunnel back from the temporary Docker host to the new cluster. Before flipping it I checked the in-cluster tunnel connector. There wasn’t one.
The repository did contain an ArgoCD Application for it, added months ago in a commit titled “adopt the connector”. But it was never listed in the kustomization that ArgoCD reads. The connector in the old cluster had been a hand-made Deployment the whole time, and it died with the cluster. Had I flipped the tunnel first, both sites would have gone down with nothing in the cluster to receive the traffic.
I wired the Application in, waited for both replicas to register, stopped the temporary connector, switched the tunnel, and then confirmed with a tagged User-Agent that requests were reaching the cluster pods and none were reaching the temporary containers. Only after that did I delete them.
Lesson: “it worked in the old cluster” does not mean “it is in Git”. A rebuild from scratch is the most honest audit of your GitOps there is.
What is better now
- etcd is stacked and upgraded by
kubeadm upgradewith everything else; - the API sits behind a VIP. I shut down the master holding it; the VIP moved and
kubectldid not even need a retry; - the cluster runs the latest stable Kubernetes, installed from packages and held, not binaries dropped into a directory;
- service load balancing is done by Cilium itself;
- ingress is Envoy Gateway with two replicas, so restarting the CNI agent no longer takes ingress with it;
- CoreDNS, the Gateway API CRDs and the tunnel connector are in Git, where they always should have been.
The plan was right. The value of the day was in the six places where reality had quietly drifted from it.