Calico was Ready with no errors, and every new pod was unreachable for half a minute

During a multi-hop Kubernetes upgrade of a bare-metal cluster at Optimacros, Calico had to go from 3.29.5 to 3.31.7 first. The upgrade itself looked textbook: new CRDs and RBAC from upstream, images swapped, every calico-node pod 1/1 Ready, ClusterInformation showing the new version, no ERROR lines from Felix. Existing pods talked to each other, DNS resolved, ingress worked.

And yet something was off.

The test that caught it

After any CNI change I run one specific check: create a fresh pod on node A and another on node B, and time how long it takes for the first request between them to succeed.

s=$(date +%s); until kubectl exec nt-src -- wget -qO- -T1 http://$IP:8080/ >/dev/null 2>&1; do :; done
echo "$(( $(date +%s)-s ))s"

It took 15 to 42 seconds after the destination pod was already Ready. Before the upgrade it was 1–2 seconds. Health checks and smoke tests that reuse existing pods passed, because the endpoints of existing pods had been programmed long ago. Only a new endpoint shows the lag.

A loop with no ERROR

The Felix logs on the workers contained a line I had first dismissed because it is logged at warning level:

Applying XDP BPF actions did not succeed, will retry with resync... error=failed to remove XDP program from lo

Counted per node, it appeared 250–470 times every five minutes on every Debian 13 worker — and zero times on the masters, which run a different OS and kernel. The previous Calico version had left an xdpgeneric program on the loopback interface, and the new one could not remove it on this kernel. Every failed attempt queued a full dataplane resync. New workload endpoints were programmed only in the short gaps between resyncs, so each new pod spent tens of seconds in a blackhole.

Why switching XDP off was safe

Calico uses XDP only for untracked and prefilter policies. This cluster had no NetworkPolicies at all, so XDP was doing nothing useful while actively hurting. The fix was one line:

kubectl patch felixconfiguration default --type=merge -p '{"spec":{"xdpEnabled":false}}'

Within seconds the error count dropped to zero on every node, and the fresh-pod test went back to 1–2 seconds. The stale program stays attached to lo — it was there before and filters nothing. Rollback is the same patch with true, which on these kernels brings the loop straight back.

The decision has a revisit trigger: before adopting doNotTrack or host-endpoint policies that rely on XDP, and on the next Calico minor, which restructures the DaemonSet anyway.

Takeaways

• “calico-node Ready, no ERROR” is not network health. Felix can be busy failing at warning level.
• After any CNI change, test fresh pod to fresh pod across nodes, with a timer. Reused pods hide endpoint-programming lag completely.
• Count warnings per node. A loop that only exists on one OS or kernel is a strong hint about the cause.
• A feature you do not use is still code that runs. If XDP serves no policy, turning it off removes a failure mode for free.