It worked for a minute, then timed out: MetalLB L2 with kube-proxy ipvs and no strictARP

Part of retiring ingress-nginx on a bare-metal cluster at Optimacros was switching hosts, one by one, from the old ingress to the Envoy Gateway LoadBalancer IP. In front of the cluster sits an external HAProxy in another subnet but on the same L2 segment. I pointed the first SNI host at the new IP and it answered fine.

About a minute later every host routed that way was timing out. HAProxy logged Layer4 timeout and marked the backend down. From my laptop the same IP still worked. MetalLB reported the IP announced from exactly the right node, the one running the Envoy pod. I rolled the HAProxy change back and service recovered — about a minute of impact for four hosts.

The wrong MAC address

The first useful look was from the HAProxy VM itself: ip neigh show <lb-ip>. The cached MAC belonged to a worker that had no Envoy pod. MetalLB was announcing from one node, but HAProxy was talking to another.

The Service used externalTrafficPolicy: Local, which is right for preserving client IPs — and it means a node without a local endpoint drops the traffic. So the SYNs went to the wrong node and disappeared.

Why every node answered ARP

kube-proxy in ipvs mode binds every LoadBalancer IP to the dummy interface kube-ipvs0 on every node. On this cluster that was 306 addresses per node. With strictARP: false — the kubespray and kubeadm default, set since install — the kernel keeps arp_ignore=0, so every node happily answers ARP requests for those IPs, not just the MetalLB speaker that owns the announcement.

Which MAC a client caches is a race. That explains “it worked, then stopped”: the first answer happened to come from the right node, a later one did not. Clients behind the router were lucky with the router’s cache, which is why the problem stayed invisible for so long. The external HAProxy had previously only used NodePorts on the masters; the migration was the first time anything on that segment resolved the LB IP itself.

MetalLB documents this plainly: L2 mode with kube-proxy in ipvs mode requires strictARP: true.

The fix and how I verified it

1. Back up the kube-proxy ConfigMap.
2. Set strictARP: true and restart the kube-proxy DaemonSet. kube-proxy then sets arp_ignore=1 and arp_announce=2, and only the announcing speaker answers ARP.
3. Flush the stale neighbour entry on the HAProxy VM.
4. Probe for a full minute instead of trusting one request:

for i in $(seq 1 60); do timeout 2 bash -c "</dev/tcp/<lb-ip>/443" && echo ok || echo FAIL; sleep 1; done | sort | uniq -c

Then I redid the switch host by host, with a soak period after each. kubeadm reads KubeProxyConfiguration from the same ConfigMap during upgrades, so the setting survived the Kubernetes hops that followed the same day.

Takeaways

• With MetalLB L2 and kube-proxy ipvs, check strictARP first. If it is false, every node is a candidate answer for every LoadBalancer IP.
• externalTrafficPolicy: Local turns a wrong ARP answer into a silent drop.
• One successful request proves nothing in an ARP race. Probe over time and soak each change.
• A latent network bug often waits for the first new client on the segment. Migrations are when that client appears.