Incident response
Real incidents taken apart: the first wrong guess, how the root cause was proven and what changed afterwards.
9 posts
- The loading screen was a retry loop, not a hang — and the reboot was never neededA game on my Linux workstation froze on its loading screen, and the known fix was a full reboot. Five seconds of /proc counters showed a hot retry loop that a single SIGTERM could end.
- NotReady for six hours, and the node was alive the whole timeEvery signal said the node was dead. It was thrashing on memory, and the obvious fix — delete the node and force-detach the volume — was one command away from corrupting a database.
- Four checks that lied while the system was fineIn one day of platform migrations, four verification steps reported failure while everything was healthy. When one signal disagrees with all the others, suspect the signal first.
- Vault kept sealing itself. The cause was a 15-minute VM freeze below the clusterFour Vault seals in three weeks, each treated as a separate incident. A journald gap analysis showed the hypervisor stunning the VM during backups, and Longhorn turning that into a cascade.
- One stale DHCP option took down a whole Kubernetes clusterAfter a host reboot about fifty pods crashlooped while every node was Ready. The cause was a forgotten DHCP classless-route option that silently removed the default gateway.
- Three incidents that were one: a database IP baked into a configLogin 500s, backends in CrashLoopBackOff and a stuck rollout looked like three problems. All of them were one hardcoded database IP in a pgbouncer config.
- The "disk errors" were an OOM: how Longhorn volumes lost their backendEIO, read-only ext4, a corrupted MySQL datadir and a sealed Vault looked like a failing disk. Host dmesg showed a node-wide OOM that killed the Longhorn instance-manager first.
- A Longhorn volume that would not grow: an attachment ticket pinned to a bad nodeA PVC sat in Resizing forever while Longhorn retried the expansion every five seconds without an error. The volume was locked to one unhealthy node by a stale attachment ticket.
- Helm says "another operation is in progress" — and nothing is runningA cancelled CI job left two Helm releases stuck in pending-upgrade and blocked every deploy. How to find the lock, release it safely and check a whole namespace at once.