A liveness probe tied to someone else's network: 330 restarts that fixed nothing

In my home lab two replicas of the same cloudflared Deployment — same image, same config — had wildly different histories: one with 330 restarts, the other with 13. Exit code 137, reason Error, not OOMKilled.

The asymmetry is the clue

A 25:1 difference between identical replicas already rules out the image and the config. Whatever kills them depends on the node or the network path, not on the code. And 137 with reason: Error means SIGKILL from kubelet — in practice, a failing probe.

What the probe was really asking

The liveness probe polled /ready. I assumed that meant “the process is ready”, so instead of assuming further I asked the endpoint itself through a port-forward:

/ready -> {"status":200,"readyConnections":4}

On cloudflared, /ready reports the state of the tunnel’s connections to the Cloudflare edge. Not whether the process is alive. So whenever the upstream network blipped, the connections dropped, /ready failed, and kubelet killed a perfectly healthy process. The logs said it plainly:

failed to accept QUIC stream: timeout: no recent network activity

Restarting a pod does not repair someone else’s WAN. The final proof that the cause was always external: the restarts stopped on their own the day the home uplink was fixed, with no change to the Deployment at all.

The fix

• Liveness now polls /metrics, which answers for as long as the process itself serves HTTP.
• Readiness stays on /ready, where “is the tunnel up” is exactly the right question: an unready pod stops getting traffic but is not killed.
• timeoutSeconds went from 1 to 5. One second is aggressive for a process busy with QUIC I/O.

Before switching I checked the new path on a live pod. Pointing liveness at a path that returns 404 kills every replica at once, which is a much worse outage than the one you are fixing.

I also deliberately did not bundle the overdue cloudflared upgrade into the same change, so that a regression would have one obvious cause.

The app’s own workers already had the right pattern, and it is the one to copy: /health always returns 200 and carries the Redis state as a field, while /ready returns 503 when Redis is down.

Takeaways

• Liveness answers one question: “is this process wedged, and would a restart help?”
• If the answer depends on a network, a database or a third party, it belongs in readiness or in an alert, never in liveness.
• A liveness probe on an external dependency turns someone else’s outage into your CrashLoop, and the restart counter then hides the real cause.
• Very different restart counts between identical replicas point at the environment, not the code.