Writing
Notes from production and the home lab: upgrades, migrations, incidents and the reasoning behind them.
- An AI agent on my ops work: the guardrails that make it safeI run much of my operations work with an AI agent in the terminal: kubectl, logs, manifests, git, cloud APIs. Speed is the easy part. These are the rules that make it safe, and the mistakes they caught.
- RDS refused connections at 10% CPU: default connection pools times replicasLoad tests hit a flat plateau of ~420 database connections and the request was for a much bigger database. The real ceiling was max_connections versus Npgsql default pools.
- Calico was Ready with no errors, and every new pod was unreachable for half a minuteAfter a Calico upgrade, fresh pods were blackholed for 15–42 seconds. Felix was stuck in an XDP resync loop that only a cross-node, fresh-pod test could see.
- Kubernetes 1.32 to 1.37 in one day, five hops, without the kubespray that built the clusterAn end-of-life bare-metal cluster, a lost kubespray inventory and binaries outside any package manager. How five kubeadm hops went in one day, and the traps no plan listed.
- It worked for a minute, then timed out: MetalLB L2 with kube-proxy ipvs and no strictARPMoving hosts to a new LoadBalancer IP worked briefly, then an external load balancer saw Layer 4 timeouts. Every node was answering ARP for the same IP.
- The certificates expired on schedule. The alert fired for a week — into nowhereA kubeadm cluster lost its control plane exactly one year after install. The expiry alert had been firing for seven days, but Alertmanager only had a null receiver.
- The alert fired and nobody saw it: a cluster label lost in aggregationA node ran at 90-98% CPU for four hours and ended in an OOM kill. The CPU alert did fire, but avg by(instance) dropped the label every route and dashboard selected on.
- Kubernetes 1.33 → 1.36 and Calico → Cilium in one dayA home-lab rehearsal: three Kubernetes hops, a CNI swap to Cilium with kube-proxy replacement, and ingress-nginx retired in favour of Gateway API — why a bounded big-bang beat live migration, and the traps worth knowing.
- A production certificate expired and nobody noticed: the front proxy covered for itEvery public host returned 200 while the cluster served an expired wildcard. A proxy with its own TLS hid the outage; the real finding was the missing expiry alert.
- multipathd hijacked a Longhorn volume: "mount point busy" while nothing was mountedA pod sat in Init for five days, 2413 mount retries, and a full power cycle did not help. multipathd had built a map over the Longhorn iSCSI device and held it open.
- The rebalance storm was just a deploy: a threshold that scales with group sizeA Kafka rebalance alert fired at 60 per hour on every release. The metric was summed over 40 consumers and averaged over a sliding hour, so the alert described the past.
- Argo CD said Synced, and the API server had quietly thrown my fix awayMonths of missing metrics, a fix already in git and a green Synced status. The field did not exist in the CRD, so the API server pruned it and nobody complained.
- A liveness probe tied to someone else's network: 330 restarts that fixed nothingTwo identical cloudflared replicas, 330 and 13 restarts. The liveness probe was checking the tunnel to the edge, so every uplink hiccup turned into a SIGKILL.
- The loading screen was a retry loop, not a hang — and the reboot was never neededA game on my Linux workstation froze on its loading screen, and the known fix was a full reboot. Five seconds of /proc counters showed a hot retry loop that a single SIGTERM could end.
- Three alerting outages from one root: a name where an ID belongedSomeone renamed a Slack channel and alert delivery for a production system died silently. It was the third failure on the same identifier — every earlier fix changed the transport and kept the name.
- Kafka at 94 % disk with 17 hours left, and retention that kept twice what we askedTraffic grew 150x and nobody resized the volumes. Worse, with default segment rolling a 7-day retention really meant 7 to 14 days. How we bought time and fixed the cause.
- Basic auth vanished on port 80 after an Ingress to Gateway API migrationEvery status was green and my own curl returned 401. Yet the WordPress login was open to the world: Gateway API binds a policy to one listener, while Ingress annotations covered every port.
- NotReady for six hours, and the node was alive the whole timeEvery signal said the node was dead. It was thrashing on memory, and the obvious fix — delete the node and force-detach the volume — was one command away from corrupting a database.
- A star in the middle of a Vault path is not a wildcardA dead token took down a secrets operator. Reissuing it exposed an older bug: 27 Vault policies used * mid-path, silently denied what they declared, and hid an over-privilege.
- A Postgres replica stuck in a pg_rewind loop for 23 hours: rebuild, do not repairAfter a failover a CloudNativePG replica could not rewind: its WAL segments were zero bytes and its data directory a tenth of the primary. Nobody noticed for 23 hours because nothing scraped the operator metrics.
- An error budget in VictoriaMetrics: recording rules, burn-rate alerts and one honest panelHow we built an error budget for tier-1 services from metrics we already had: per-service SLO targets, multi-window burn-rate alerts, a Grafana row — and why a negative budget right after launch was not an incident.
- One p99 alert, five slow routes, five different causesA service-level latency alert named no endpoint. Splitting it per route showed an external gateway, a captcha check, a provider call — and a set of measurement traps that would have sent us the wrong way.
- Moving production Postgres from an RDS Multi-AZ instance to a Multi-AZ DB clusterA planned cutover with near-zero data loss — and three traps: "maintenance" that did not stop writes, passwords RDS will not let you copy, and a dump that took 25 minutes over VPN and 31 seconds in the VPC.
- The lag went to zero and nobody was fixed: a Kafka consumer stall hidden by a group.id renameConsumer lag grew for five days, then dropped to zero and stayed there. It looked self-healed. In fact the consumer was stuck, the metric had vanished, and the alert had resolved itself.
- Profiling a Python memory leak in production with memray — and the outage we caused on the wayA backend OOMed every 10–15 hours. Hourly memray snapshots on the live pod found the leak in library middleware, not the app — after a first attempt took production down for seven minutes.
- Moving a production cluster from Longhorn to LINSTOR — and the expectation that did not come trueWhy we swapped Longhorn for LINSTOR/DRBD on a self-managed cluster, how each workload was moved with a different method, and what went wrong on the way.
- From ingress-nginx to Envoy Gateway with zero downtime: 27 Ingresses, 39 HTTPRoutesingress-nginx is retired and blocks Kubernetes upgrades. How we moved 16 public hosts to Envoy Gateway by building the new entry point next to the old one and swapping only the address at the end.
- Moving CloudNativePG to the Barman Cloud plugin without splitting the archiveCNPG is removing native barmanObjectStore support. The switch has no valid intermediate state, the database had no base backups at all, and one path decides whether point-in-time recovery still works.
- Four checks that lied while the system was fineIn one day of platform migrations, four verification steps reported failure while everything was healthy. When one signal disagrees with all the others, suspect the signal first.
- Alerts that cannot fire: a blind ServiceMonitor and a dead delivery channelA Vault peer stayed sealed for 32 hours and secrets stopped syncing for a week, with a VaultSealed alert in the repo and Alertmanager healthy. Two independent failures hid it.
- "Kernel drift" was uptime drift: patches landed, nothing rebootedThree different kernels across seven nodes looked like unfinished patch waves and sent an investigation down a false trail. Every node had the same patches; only uptime differed.
- kubectl patch --type merge wiped CoreDNS, and every health check stayed greenA one-line image bump replaced the whole container spec. CoreDNS lost its config, quietly fell back to the whoami plugin and answered every query with NOERROR and no answer.
- Replacing MetalLB with Cilium LB IPAM without dropping a single requestThe cluster had exactly one LoadBalancer IP and every route depended on it. How I handed it from MetalLB to Cilium with zero failed requests — by letting both controllers run at once.
- Moving Vault from Shamir to transit auto-unseal, and the four traps on the wayEvery Vault pod restart came back sealed and froze secret sync behind it. How we added a transit auto-unseal provider, migrated a three-node raft cluster, and what bit us.
- Vault kept sealing itself. The cause was a 15-minute VM freeze below the clusterFour Vault seals in three weeks, each treated as a separate incident. A journald gap analysis showed the hypervisor stunning the VM during backups, and Longhorn turning that into a cascade.
- A GitLab upgrade and two signals that liedA routine minor upgrade of self-hosted GitLab in Docker. A pre-flight check claimed 32 pending migrations, and the container reported healthy while serving nothing.
- Updating the OS on seven Kubernetes nodes without anyone noticingA rolling Ubuntu point update across a cluster that holds all its own state. The pre-flight checks turned a scary job into a boring one, and Longhorn decided the pace.
- One stale DHCP option took down a whole Kubernetes clusterAfter a host reboot about fifty pods crashlooped while every node was Ready. The cause was a forgotten DHCP classless-route option that silently removed the default gateway.
- Merged, synced, mounted — and still not alerting: vmalert without hot reloadNew Kafka alert rules went through Git and Argo CD into the ConfigMap, yet vmalert never evaluated them. The GitOps pipeline broke on its very last step.
- A Grafana panel that tells you when a pod will OOMTurning a memory trend into a time-to-OOM and time-to-full ETA in PromQL, the two bugs that made it lie, and the leak it caught on day one.
- ONLINE_UNCERTAIN: when the MySQL operator and the database disagreeA router flapping, an operator stuck in a lock for two days, a CRD frozen at 2/3 — while all three InnoDB Cluster members were healthy. The fix was one metadata rescan.
- Vault Secrets Operator went quiet after a seal — and a restart did not wake it38 of 43 VaultStaticSecrets stuck for two days with a frozen error. Unsealing Vault and restarting the operator did nothing; a one-field spec change did.
- A WordPress login loop that only happened sometimeswp-admin bounced back to the login page with reauth=1, but not always. The cause was random auth salts generated per pod behind a non-sticky load balancer.
- Three incidents that were one: a database IP baked into a configLogin 500s, backends in CrashLoopBackOff and a stuck rollout looked like three problems. All of them were one hardcoded database IP in a pgbouncer config.
- The "disk errors" were an OOM: how Longhorn volumes lost their backendEIO, read-only ext4, a corrupted MySQL datadir and a sealed Vault looked like a failing disk. Host dmesg showed a node-wide OOM that killed the Longhorn instance-manager first.
- "Vault keeps sealing itself" — no, it just never unsealsA Vault pod sat at Ready 0/1 for days while raft quorum quietly held on two peers. Nothing was sealing it: Shamir seal without auto-unseal leaves every restart sealed.
- A Longhorn volume that would not grow: an attachment ticket pinned to a bad nodeA PVC sat in Resizing forever while Longhorn retried the expansion every five seconds without an error. The volume was locked to one unhealthy node by a stale attachment ticket.
- Changing an immutable StatefulSet field without restarting a single podGit said 20 GiB, the live StatefulSet ran 6 GiB templates and Argo CD said Synced. How orphan deletion plus OnDelete rebuilt the template with zero downtime.
- Five days at 1/3 with every dashboard green: when a CRD status liesA MySQL InnoDB Cluster ran without HA for five days while its status, the pod phase and the readiness gate all said healthy. All three came from one stalled operator.
- Helm says "another operation is in progress" — and nothing is runningA cancelled CI job left two Helm releases stuck in pending-upgrade and blocked every deploy. How to find the lock, release it safely and check a whole namespace at once.
- How I think about the platformPlatform as product, GitOps, CI/CD, DevSecOps — predictable releases and security in the pipeline, not after the fact.