Kubernetes
Running Kubernetes in production: upgrades, operators, StatefulSets, probes and the failures that only show up on real clusters.
36 posts
- Calico was Ready with no errors, and every new pod was unreachable for half a minuteAfter a Calico upgrade, fresh pods were blackholed for 15–42 seconds. Felix was stuck in an XDP resync loop that only a cross-node, fresh-pod test could see.
- Kubernetes 1.32 to 1.37 in one day, five hops, without the kubespray that built the clusterAn end-of-life bare-metal cluster, a lost kubespray inventory and binaries outside any package manager. How five kubeadm hops went in one day, and the traps no plan listed.
- It worked for a minute, then timed out: MetalLB L2 with kube-proxy ipvs and no strictARPMoving hosts to a new LoadBalancer IP worked briefly, then an external load balancer saw Layer 4 timeouts. Every node was answering ARP for the same IP.
- The certificates expired on schedule. The alert fired for a week — into nowhereA kubeadm cluster lost its control plane exactly one year after install. The expiry alert had been firing for seven days, but Alertmanager only had a null receiver.
- Kubernetes 1.33 → 1.36 and Calico → Cilium in one dayA home-lab rehearsal: three Kubernetes hops, a CNI swap to Cilium with kube-proxy replacement, and ingress-nginx retired in favour of Gateway API — why a bounded big-bang beat live migration, and the traps worth knowing.
- multipathd hijacked a Longhorn volume: "mount point busy" while nothing was mountedA pod sat in Init for five days, 2413 mount retries, and a full power cycle did not help. multipathd had built a map over the Longhorn iSCSI device and held it open.
- Argo CD said Synced, and the API server had quietly thrown my fix awayMonths of missing metrics, a fix already in git and a green Synced status. The field did not exist in the CRD, so the API server pruned it and nobody complained.
- A liveness probe tied to someone else's network: 330 restarts that fixed nothingTwo identical cloudflared replicas, 330 and 13 restarts. The liveness probe was checking the tunnel to the edge, so every uplink hiccup turned into a SIGKILL.
- Kafka at 94 % disk with 17 hours left, and retention that kept twice what we askedTraffic grew 150x and nobody resized the volumes. Worse, with default segment rolling a 7-day retention really meant 7 to 14 days. How we bought time and fixed the cause.
- Basic auth vanished on port 80 after an Ingress to Gateway API migrationEvery status was green and my own curl returned 401. Yet the WordPress login was open to the world: Gateway API binds a policy to one listener, while Ingress annotations covered every port.
- NotReady for six hours, and the node was alive the whole timeEvery signal said the node was dead. It was thrashing on memory, and the obvious fix — delete the node and force-detach the volume — was one command away from corrupting a database.
- A star in the middle of a Vault path is not a wildcardA dead token took down a secrets operator. Reissuing it exposed an older bug: 27 Vault policies used * mid-path, silently denied what they declared, and hid an over-privilege.
- A Postgres replica stuck in a pg_rewind loop for 23 hours: rebuild, do not repairAfter a failover a CloudNativePG replica could not rewind: its WAL segments were zero bytes and its data directory a tenth of the primary. Nobody noticed for 23 hours because nothing scraped the operator metrics.
- Profiling a Python memory leak in production with memray — and the outage we caused on the wayA backend OOMed every 10–15 hours. Hourly memray snapshots on the live pod found the leak in library middleware, not the app — after a first attempt took production down for seven minutes.
- Moving a production cluster from Longhorn to LINSTOR — and the expectation that did not come trueWhy we swapped Longhorn for LINSTOR/DRBD on a self-managed cluster, how each workload was moved with a different method, and what went wrong on the way.
- From ingress-nginx to Envoy Gateway with zero downtime: 27 Ingresses, 39 HTTPRoutesingress-nginx is retired and blocks Kubernetes upgrades. How we moved 16 public hosts to Envoy Gateway by building the new entry point next to the old one and swapping only the address at the end.
- Moving CloudNativePG to the Barman Cloud plugin without splitting the archiveCNPG is removing native barmanObjectStore support. The switch has no valid intermediate state, the database had no base backups at all, and one path decides whether point-in-time recovery still works.
- Four checks that lied while the system was fineIn one day of platform migrations, four verification steps reported failure while everything was healthy. When one signal disagrees with all the others, suspect the signal first.
- Alerts that cannot fire: a blind ServiceMonitor and a dead delivery channelA Vault peer stayed sealed for 32 hours and secrets stopped syncing for a week, with a VaultSealed alert in the repo and Alertmanager healthy. Two independent failures hid it.
- "Kernel drift" was uptime drift: patches landed, nothing rebootedThree different kernels across seven nodes looked like unfinished patch waves and sent an investigation down a false trail. Every node had the same patches; only uptime differed.
- kubectl patch --type merge wiped CoreDNS, and every health check stayed greenA one-line image bump replaced the whole container spec. CoreDNS lost its config, quietly fell back to the whoami plugin and answered every query with NOERROR and no answer.
- Replacing MetalLB with Cilium LB IPAM without dropping a single requestThe cluster had exactly one LoadBalancer IP and every route depended on it. How I handed it from MetalLB to Cilium with zero failed requests — by letting both controllers run at once.
- Moving Vault from Shamir to transit auto-unseal, and the four traps on the wayEvery Vault pod restart came back sealed and froze secret sync behind it. How we added a transit auto-unseal provider, migrated a three-node raft cluster, and what bit us.
- A GitLab upgrade and two signals that liedA routine minor upgrade of self-hosted GitLab in Docker. A pre-flight check claimed 32 pending migrations, and the container reported healthy while serving nothing.
- Updating the OS on seven Kubernetes nodes without anyone noticingA rolling Ubuntu point update across a cluster that holds all its own state. The pre-flight checks turned a scary job into a boring one, and Longhorn decided the pace.
- One stale DHCP option took down a whole Kubernetes clusterAfter a host reboot about fifty pods crashlooped while every node was Ready. The cause was a forgotten DHCP classless-route option that silently removed the default gateway.
- ONLINE_UNCERTAIN: when the MySQL operator and the database disagreeA router flapping, an operator stuck in a lock for two days, a CRD frozen at 2/3 — while all three InnoDB Cluster members were healthy. The fix was one metadata rescan.
- Vault Secrets Operator went quiet after a seal — and a restart did not wake it38 of 43 VaultStaticSecrets stuck for two days with a frozen error. Unsealing Vault and restarting the operator did nothing; a one-field spec change did.
- A WordPress login loop that only happened sometimeswp-admin bounced back to the login page with reauth=1, but not always. The cause was random auth salts generated per pod behind a non-sticky load balancer.
- Three incidents that were one: a database IP baked into a configLogin 500s, backends in CrashLoopBackOff and a stuck rollout looked like three problems. All of them were one hardcoded database IP in a pgbouncer config.
- The "disk errors" were an OOM: how Longhorn volumes lost their backendEIO, read-only ext4, a corrupted MySQL datadir and a sealed Vault looked like a failing disk. Host dmesg showed a node-wide OOM that killed the Longhorn instance-manager first.
- "Vault keeps sealing itself" — no, it just never unsealsA Vault pod sat at Ready 0/1 for days while raft quorum quietly held on two peers. Nothing was sealing it: Shamir seal without auto-unseal leaves every restart sealed.
- A Longhorn volume that would not grow: an attachment ticket pinned to a bad nodeA PVC sat in Resizing forever while Longhorn retried the expansion every five seconds without an error. The volume was locked to one unhealthy node by a stale attachment ticket.
- Changing an immutable StatefulSet field without restarting a single podGit said 20 GiB, the live StatefulSet ran 6 GiB templates and Argo CD said Synced. How orphan deletion plus OnDelete rebuilt the template with zero downtime.
- Five days at 1/3 with every dashboard green: when a CRD status liesA MySQL InnoDB Cluster ran without HA for five days while its status, the pod phase and the readiness gate all said healthy. All three came from one stalled operator.
- Helm says "another operation is in progress" — and nothing is runningA cancelled CI job left two Helm releases stuck in pending-upgrade and blocked every deploy. How to find the lock, release it safely and check a whole namespace at once.