← All posts

A 30-second network blip, seven minutes without Kafka metadata: the KRaft election storm

No broker restarted, yet backends drowned in NotLeaderForPartition and rebalances. KRaft default quorum timeouts are five times shorter than the time to re-establish a connection, so one reconnect turns into a self-sustaining election loop.

One evening the backends of a production platform started throwing errors in bulk: NotLeaderForPartitionError, NotControllerException, Heartbeat session expired, consumer groups rebalancing over and over. The Kafka pods showed RESTARTS 0. External 5xx barely moved — it was the event bus that broke, not HTTP. Seven minutes later everything recovered by itself.

“It healed on its own” is not a fix. The question was not how to stop the storm, it was why a short network blip became a seven-minute outage.

What the controllers were doing

The broker logs had dozens of Completed transition to CandidateState with a growing epoch: 52 epochs in 100 seconds. While the quorum is not formed, the controller accepts no metadata writes — no leader elections for partitions, no ISR changes. Producers and consumers of partitions whose leader sat on the affected node simply stop.

First question: who was isolated. Do not trust voteStates={0=UNRECORDED, ...} — UNRECORDED only means “no vote received”, which is normal for a candidate that already voted for itself. Count accepted vote requests per controller instead. Two controllers had about 20 each, the third had 2. Exactly one node was cut off, not all three.

Elected, then lost

The interesting part was that leaders were elected — repeatedly. One cycle, unrolled:

20:54:39.429  c0  Completed transition to CandidateState(epoch=5879)
20:54:39.457  c0  Completed transition to Leader(localId=0, epoch=5879)   ← leader in 28 ms
20:54:39.474  c2  FollowerState(fetchTimeoutMs=2000, leaderId=0)
20:54:41.468  c2  Become candidate due to fetch timeout                   ← 1994 ms later
20:54:42.453  c0  Completed transition to ResignedState(epoch=5879)

And right next to it, the reason the follower could not make it in time:

c2  Disconnecting from node 0 due to socket connection setup timeout. The timeout value is 11429 ms.

Why it loops

The KRaft defaults:

Setting Default
controller.quorum.fetch.timeout.ms 2000
controller.quorum.election.timeout.ms 1000
controller.quorum.request.timeout.ms 2000
socket.connection.setup.timeout.ms 10000 (+ jitter)

The budget for re-establishing a connection is five times the budget for a fetch. Any event that forces controllers to reconnect guarantees an election: the fresh leader cannot deliver the first fetch within 2 seconds because the connection is still being set up, the follower declares itself a candidate, the leader resigns, and the elections tear connections down again. The loop feeds itself. With SASL on the controller listener every reconnect also includes a handshake.

That is the difference between a trigger and an amplifier. The trigger was a 30-second blip on one node. The amplifier turned it into seven minutes. A day later the same kind of blip hit again, but this time connections survived — one election, two seconds, nobody noticed.

It is a known issue (KAFKA-16620), and the defaults are the same everywhere. Checking other environments showed that the quiet production was in fact the least affected: non-production clusters with untouched defaults logged hundreds of Become candidate events in two weeks as background noise.

The fix

controller.quorum.fetch.timeout.ms=10000
controller.quorum.election.timeout.ms=4000
controller.quorum.request.timeout.ms=5000

The invariant: fetch.timeout ≥ socket.connection.setup.timeout.ms, and election.timeout < fetch.timeout. The price: detecting a really dead controller takes ~10 seconds instead of ~2. Against 100 seconds of thrashing that is a good trade.

Two things on the rollout bit us in other places, so I check them first now:

  • podManagementPolicy: Parallel affects rolling updates too. The Kubernetes docs say it only affects scaling, but in practice the controller does not wait for Ready between pod updates (kubernetes#96218). For a three-node quorum that means losing it entirely. Roll in phases with updateStrategy.rollingUpdate.partition 2 → 1 → 0 and check the quorum between steps.
  • Make sure the image can still be pulled. A restart pulls the image again. If the pod has been living on a cached image from a registry that has since been cleaned up, it comes back as ImagePullBackOff.

Also check where the controllers actually run: two of three in one availability zone means losing a zone loses the quorum.

Takeaways

  • RESTARTS 0 and a storm of leader errors means the quorum, not the brokers. Look for Become candidate in the logs.
  • Find the isolated node by counting accepted vote requests, not by voteStates.
  • “Elected and lost” means the timeouts are the amplifier. Keep fetch.timeout at least as long as connection setup.
  • Separate the trigger from the amplifier. Removing the amplifier turns minutes into seconds; the trigger still needs its own investigation.
  • Measure the baseline before calling something an incident: on some environments the storm was chronic background noise.