Replacing MetalLB with Cilium LB IPAM without dropping a single request

After moving my home lab to Cilium with kube-proxy replacement, MetalLB was the last platform component outside GitOps. It was old (a 2023 release) and still served a deprecated AddressPool API that showed up in apiserver_requested_deprecated_apis. Cilium already had LB IPAM switched on and can announce LoadBalancer IPs over L2 by itself, so a second controller was pure redundancy.

The whole risk came down to one number. The cluster had exactly one LoadBalancer Service, the Gateway’s, and every reverse-proxy upstream pointed at its address. Worse, that address had never been pinned: MetalLB picked it dynamically from a pool, and the value it held was a historical accident. A naive migration would have “succeeded”, handed out the first free address in the new pool and broken every route at once.

A trap I had to clear first

Before touching anything I checked which machine answered ARP for that address. The answer was all seven of them.

kube-proxy had been removed weeks earlier, but kubeadm upgrade node quietly re-applied the kube-proxy addon on every upgrade hop. Its IPVS dummy interface, kube-ipvs0, survived two full cluster reboots, still carrying the LoadBalancer IP and about 44 stale ClusterIPs with zero IPVS rules behind them. With the default arp_ignore=0, the kernel answers ARP for any local address on any interface, so every node claimed the IP.

Under MetalLB this was survivable: with externalTrafficPolicy: Cluster and eBPF service handling, any node can serve the address. But it destroyed the only cheap way to verify an L2 handover, which is asking which MAC owns the IP. Leader election would have been decorative. So step one was removing just that one /32 from kube-ipvs0 on each node, with a canary on a non-owner node first and the current owner last.

The counter-intuitive order

1. Clear the stale address so ARP ownership becomes single-valued and observable.
2. Enable l2announcements in the Cilium Helm values through GitOps, then restart the agent and operator. The chart does not restart pods when only the ConfigMap changes.
3. Apply Cilium’s IP pool and L2 policy while MetalLB is still running.
4. Only then scale MetalLB to zero, test failover and delete it.

Step three is the reason there was no outage. Two ARP responders sound like a split brain, but with externalTrafficPolicy: Cluster it does not matter who wins: either node serves the traffic in eBPF. And because the new pool was a single /32, both IPAMs could only compute the same address, so the Service status never flapped. Cilium took the lease and ARP moved to its node while MetalLB was still up.

I pinned the address with a one-address pool rather than an annotation. The Service is owned by the Gateway, whose controller reconciles annotations away. A /32 pool makes the correct result the only arithmetic possibility, which beats trusting a hint.

Measured, not assumed

Step	Result
Pool and policy applied under live MetalLB	ARP moved to the lease holder, 0 failed requests
MetalLB scaled to zero	10 probes in a row, address unchanged
Lease holder killed	address moved in about 2.3 s, 0 of 40 requests failed
MetalLB fully deleted	ingress returned 200 after every single delete

The deletion order mattered too: the validating webhook goes first, because a webhook whose backend is gone blocks deletion of its own custom resources.

The scare that was a bad test

From a node, curl to the LoadBalancer IP returned 000 and the ARP table stayed empty. That read as “nodes cannot reach their own LoadBalancer IP”, and it nearly ate the migration window. It was neither a fault nor pre-existing damage. The exit code was 35, a TLS error, which means the TCP connection was established. Cilium redirects the address to the node’s own Envoy in eBPF, so the packet never reaches the wire and no ARP is ever needed. The TLS failure came from a passthrough listener with no SNI to match on a bare IP. With a proper Host header or --resolve, it returned 200.

Takeaways

• Before an L2 handover, find every ARP responder for the address. Leftover dummy interfaces make ownership meaningless.
• Two controllers announcing the same address are safe when any node can serve it. Use that to overlap instead of cutting over.
• Pin a critical address with a pool that can only produce that address.
• %{http_code} collapses every transport failure into 000. Always read the exit code.
• The fast failover number is the graceful one. A hard node loss waits out the full lease duration.