One stale DHCP option took down a whole Kubernetes cluster

My home lab runs a seven-node kubeadm cluster on a single Proxmox host. After I moved that host and brought it back up, the cluster came back in the worst possible shape: about fifty pods in CrashLoopBackOff or ImagePullBackOff, cluster-wide. CoreDNS could not reach the API server — connect: no route to host. The GitLab runner could not resolve anything. Image pulls failed everywhere.

And yet all seven nodes were Ready. calico-node and kube-proxy were running, and the IPVS table for the API server’s ClusterIP pointed at all three control-plane nodes, exactly as it should.

The wrong first guess

Everything screamed “CNI is broken”. Pods could not talk to anything, Calico was the obvious suspect, and kube-proxy came a close second. I spent the first stretch of time confirming that both were programmed correctly. They were.

The tell

What finally split the problem was a simple comparison: a hostNetwork pod worked, a normal pod did not. So the node’s own network namespace was fine, and only pod egress was broken.

From inside a broken pod, ip route get to the API server resolved fine, but TCP failed with EHOSTUNREACH. Then ip neigh showed the smoking gun: Calico’s link-local gateway 169.254.1.1 was INCOMPLETE. The pod could not even ARP its own gateway.

On the host, ip route show default was empty. There was no default route at all.

Root cause

Calico answers ARP for 169.254.1.1 with proxy-ARP, and proxy-ARP only replies for a target the host can route. No default route, no reply, no pod egress — to anything.

Why had every VM booted without a default route? My router hands out DHCP option 121 (classless static routes). Years ago I had configured it with a couple of routes through the old gateway, and after renumbering the LAN nobody updated it. RFC 3442 is explicit: a client that receives option 121 must ignore option 3, the default router. So every client installed only the stale classless routes and silently dropped the default gateway.

The cascade went like this: no default route → no proxy-ARP → pods cannot reach the API → CoreDNS crashloops → cluster DNS is gone → everything that resolves names crashloops too. On top of that the nodes’ own resolver stub was stuck, which explained the image pulls.

The fix

At runtime I restored the default route on each node with ip route replace default via <gateway>. The pods recovered the moment 169.254.1.1 became REACHABLE.

The permanent fix was on the router: I rewrote option 121 (and its Microsoft twin, option 249) so that it carries a default route — a zero-length prefix followed by the correct gateway. Removing the option entirely was not possible on that router’s default network, so fixing the value in place was the reliable path. Two more real reboots later, the cluster reformed on its own, with no hand-holding.

Takeaways

• Option 121 present means option 3 is ignored. A stale classless-route push can strip the default route from an entire subnet. Audit DHCP options after any gateway or subnet change.
• 169.254.1.1 INCOMPLETE in a pod’s ip neigh is the Calico smoking gun. The CNI is fine; the host cannot route to the gateway.
• Compare hostNetwork and normal pods early. It cuts the search space in half in one step.
• “All nodes Ready” says nothing about pod egress. Node readiness and pod networking fail independently.