NotReady for six hours, and the node was alive the whole time

At Gambchamp, on a product platform running on AWS EKS, a database node showed NotReady for six hours. kubectl describe node said Kubelet stopped posting node status on every condition. A ClickHouse StatefulSet member sat in Terminating on that node with its volume still attached, and its peer had crash-looped 96 times.

Every signal read “the machine is gone”. The textbook remediation is just as clear: kubectl delete node to release the stuck pod, let the volume force-detach, let the StatefulSet reschedule. That command was one keystroke away from destroying the data.

Fence first

Before any force-detach, you have to prove the old writer is really dead. So instead of acting, I fenced — read-only:

• the EC2 instance state was running;
• both status checks, system and instance, were ok;
• the SSM agent had stopped pinging, about three minutes apart from kubelet.

A running instance with healthy status checks is not a dead machine. Two independent daemons going silent within minutes of each other was the real clue: daemons do not crash in sync, but they do starve in sync.

Starving, not dead

CloudWatch showed CPU at a flat 55 % for the whole window, spiking to 99.9 %. On a 2-vCPU node, 55 % is one core pinned solid. That is what continuous memory reclaim looks like.

The cause was a single line of arithmetic:

container limits.memory (4Gi)  >  node allocatable (3.07Gi)

A cgroup limit above what the node can actually give will never trigger a cgroup OOM kill. The container keeps growing until the whole machine runs out, and since there is no swap, the kernel spends its time in direct reclaim. kubelet and the SSM agent get no CPU and cannot send heartbeats. NotReady was a statement about kubelet, not about the machine — and the database on it was very likely still writing to its volume.

Detaching a volume from a live writer is how you get a corrupted disk. reclaim=Retain keeps the disk, not its consistency.

What actually happened next

Nothing — on purpose. Fencing answered “is it safe to force-detach?” with no, and I made no mutation. A while later the kernel’s system OOM killer finally picked the ClickHouse process, memory came back, kubelet started posting status again, and the node returned to Ready.

That recovery was luck, not a fix. The durable fix is to make the limit fit: limits.memory below node allocatable, so the cgroup killer takes one container instead of the kernel drowning the whole node. And to right-size the pool, which was simply too small for what ran on it. Raising the limit further would have made things worse.

Takeaways

• NotReady describes kubelet, not the machine. Before any destructive step, fence: instance state, status checks, user-space agent ping, CPU over the incident window.
• running plus status checks ok only means the hypervisor can reach the kernel. Pair it with CPU and an agent heartbeat.
• Two daemons going silent together points at resource starvation, not at a network partition.
• Compare memory limits with node allocatable, not capacity. A limit that does not fit cannot protect the node.
• delete node plus force-detach has no rollback. If you cannot prove the writer is dead, do not do it.