Networking and Gateway API
Cluster networking in practice: Cilium and Calico, Gateway API and Envoy, load balancers, ARP, DHCP and the routes that disappear.
9 posts
- Calico was Ready with no errors, and every new pod was unreachable for half a minuteAfter a Calico upgrade, fresh pods were blackholed for 15–42 seconds. Felix was stuck in an XDP resync loop that only a cross-node, fresh-pod test could see.
- It worked for a minute, then timed out: MetalLB L2 with kube-proxy ipvs and no strictARPMoving hosts to a new LoadBalancer IP worked briefly, then an external load balancer saw Layer 4 timeouts. Every node was answering ARP for the same IP.
- Kubernetes 1.33 → 1.36 and Calico → Cilium in one dayA home-lab rehearsal: three Kubernetes hops, a CNI swap to Cilium with kube-proxy replacement, and ingress-nginx retired in favour of Gateway API — why a bounded big-bang beat live migration, and the traps worth knowing.
- A production certificate expired and nobody noticed: the front proxy covered for itEvery public host returned 200 while the cluster served an expired wildcard. A proxy with its own TLS hid the outage; the real finding was the missing expiry alert.
- A liveness probe tied to someone else's network: 330 restarts that fixed nothingTwo identical cloudflared replicas, 330 and 13 restarts. The liveness probe was checking the tunnel to the edge, so every uplink hiccup turned into a SIGKILL.
- Basic auth vanished on port 80 after an Ingress to Gateway API migrationEvery status was green and my own curl returned 401. Yet the WordPress login was open to the world: Gateway API binds a policy to one listener, while Ingress annotations covered every port.
- From ingress-nginx to Envoy Gateway with zero downtime: 27 Ingresses, 39 HTTPRoutesingress-nginx is retired and blocks Kubernetes upgrades. How we moved 16 public hosts to Envoy Gateway by building the new entry point next to the old one and swapping only the address at the end.
- Replacing MetalLB with Cilium LB IPAM without dropping a single requestThe cluster had exactly one LoadBalancer IP and every route depended on it. How I handed it from MetalLB to Cilium with zero failed requests — by letting both controllers run at once.
- One stale DHCP option took down a whole Kubernetes clusterAfter a host reboot about fifty pods crashlooped while every node was Ready. The cause was a forgotten DHCP classless-route option that silently removed the default gateway.