https://stebas.com/ Notes from production and the home lab: upgrades, migrations, incidents and the reasoning behind them. en https://stebas.com/posts/rds-connection-ceiling-npgsql-pools/ https://stebas.com/posts/rds-connection-ceiling-npgsql-pools/ Mon, 12 Oct 2026 00:00:00 GMT Load tests hit a flat plateau of ~420 database connections and the request was for a much bigger database. The real ceiling was max_connections versus Npgsql default pools. https://stebas.com/posts/calico-xdp-resync-new-pod-blackhole/ https://stebas.com/posts/calico-xdp-resync-new-pod-blackhole/ Sat, 10 Oct 2026 00:00:00 GMT After a Calico upgrade, fresh pods were blackholed for 15–42 seconds. Felix was stuck in an XDP resync loop that only a cross-node, fresh-pod test could see. https://stebas.com/posts/kubernetes-five-hop-upgrade-without-kubespray/ https://stebas.com/posts/kubernetes-five-hop-upgrade-without-kubespray/ Sat, 10 Oct 2026 00:00:00 GMT An end-of-life bare-metal cluster, a lost kubespray inventory and binaries outside any package manager. How five kubeadm hops went in one day, and the traps no plan listed. https://stebas.com/posts/metallb-l2-ipvs-strictarp/ https://stebas.com/posts/metallb-l2-ipvs-strictarp/ Sat, 10 Oct 2026 00:00:00 GMT Moving hosts to a new LoadBalancer IP worked briefly, then an external load balancer saw Layer 4 timeouts. Every node was answering ARP for the same IP. https://stebas.com/posts/kubeadm-certs-expired-alert-into-null/ https://stebas.com/posts/kubeadm-certs-expired-alert-into-null/ Fri, 09 Oct 2026 00:00:00 GMT A kubeadm cluster lost its control plane exactly one year after install. The expiry alert had been firing for seven days, but Alertmanager only had a null receiver. https://stebas.com/posts/alert-lost-cluster-label/ https://stebas.com/posts/alert-lost-cluster-label/ Wed, 07 Oct 2026 00:00:00 GMT A node ran at 90-98% CPU for four hours and ended in an OOM kill. The CPU alert did fire, but avg by(instance) dropped the label every route and dashboard selected on. https://stebas.com/posts/k8s-136-cilium-in-one-day/ https://stebas.com/posts/k8s-136-cilium-in-one-day/ Wed, 30 Sep 2026 00:00:00 GMT A home-lab rehearsal: three Kubernetes hops, a CNI swap to Cilium with kube-proxy replacement, and ingress-nginx retired in favour of Gateway API — why a bounded big-bang beat live migration, and the traps worth knowing. https://stebas.com/posts/expired-tls-hidden-by-front/ https://stebas.com/posts/expired-tls-hidden-by-front/ Thu, 17 Sep 2026 00:00:00 GMT Every public host returned 200 while the cluster served an expired wildcard. A proxy with its own TLS hid the outage; the real finding was the missing expiry alert. https://stebas.com/posts/multipathd-hijacks-longhorn/ https://stebas.com/posts/multipathd-hijacks-longhorn/ Thu, 17 Sep 2026 00:00:00 GMT A pod sat in Init for five days, 2413 mount retries, and a full power cycle did not help. multipathd had built a map over the Longhorn iSCSI device and held it open. https://stebas.com/posts/rebalance-storm-deploy-signature/ https://stebas.com/posts/rebalance-storm-deploy-signature/ Wed, 16 Sep 2026 00:00:00 GMT A Kafka rebalance alert fired at 60 per hour on every release. The metric was summed over 40 consumers and averaged over a sliding hour, so the alert described the past. https://stebas.com/posts/argocd-synced-but-field-pruned/ https://stebas.com/posts/argocd-synced-but-field-pruned/ Tue, 15 Sep 2026 00:00:00 GMT Months of missing metrics, a fix already in git and a green Synced status. The field did not exist in the CRD, so the API server pruned it and nobody complained. https://stebas.com/posts/liveness-probe-external-dependency/ https://stebas.com/posts/liveness-probe-external-dependency/ Tue, 15 Sep 2026 00:00:00 GMT Two identical cloudflared replicas, 330 and 13 restarts. The liveness probe was checking the tunnel to the edge, so every uplink hiccup turned into a SIGKILL. https://stebas.com/posts/retry-loop-not-a-hang/ https://stebas.com/posts/retry-loop-not-a-hang/ Mon, 14 Sep 2026 00:00:00 GMT A game on my Linux workstation froze on its loading screen, and the known fix was a full reboot. Five seconds of /proc counters showed a hot retry loop that a single SIGTERM could end. https://stebas.com/posts/mutable-name-vs-contract/ https://stebas.com/posts/mutable-name-vs-contract/ Mon, 07 Sep 2026 00:00:00 GMT Someone renamed a Slack channel and alert delivery for a production system died silently. It was the third failure on the same identifier — every earlier fix changed the transport and kept the name. https://stebas.com/posts/kafka-retention-segment-roll/ https://stebas.com/posts/kafka-retention-segment-roll/ Sun, 06 Sep 2026 00:00:00 GMT Traffic grew 150x and nobody resized the volumes. Worse, with default segment rolling a 7-day retention really meant 7 to 14 days. How we bought time and fixed the cause. https://stebas.com/posts/gateway-api-policy-per-listener/ https://stebas.com/posts/gateway-api-policy-per-listener/ Tue, 01 Sep 2026 00:00:00 GMT Every status was green and my own curl returned 401. Yet the WordPress login was open to the world: Gateway API binds a policy to one listener, while Ingress annotations covered every port. https://stebas.com/posts/notready-was-memory-thrashing/ https://stebas.com/posts/notready-was-memory-thrashing/ Mon, 31 Aug 2026 00:00:00 GMT Every signal said the node was dead. It was thrashing on memory, and the obvious fix — delete the node and force-detach the volume — was one command away from corrupting a database. https://stebas.com/posts/vault-acl-mid-path-glob/ https://stebas.com/posts/vault-acl-mid-path-glob/ Mon, 31 Aug 2026 00:00:00 GMT A dead token took down a secrets operator. Reissuing it exposed an older bug: 27 Vault policies used * mid-path, silently denied what they declared, and hid an over-privilege. https://stebas.com/posts/cnpg-replica-pg-rewind-loop/ https://stebas.com/posts/cnpg-replica-pg-rewind-loop/ Fri, 28 Aug 2026 00:00:00 GMT After a failover a CloudNativePG replica could not rewind: its WAL segments were zero bytes and its data directory a tenth of the primary. Nobody noticed for 23 hours because nothing scraped the operator metrics. https://stebas.com/posts/error-budget-burn-rate-alerts/ https://stebas.com/posts/error-budget-burn-rate-alerts/ Wed, 26 Aug 2026 00:00:00 GMT How we built an error budget for tier-1 services from metrics we already had: per-service SLO targets, multi-window burn-rate alerts, a Grafana row — and why a negative budget right after launch was not an incident. https://stebas.com/posts/latency-triage-per-route/ https://stebas.com/posts/latency-triage-per-route/ Wed, 26 Aug 2026 00:00:00 GMT A service-level latency alert named no endpoint. Splitting it per route showed an external gateway, a captcha check, a provider call — and a set of measurement traps that would have sent us the wrong way. https://stebas.com/posts/rds-instance-to-multi-az-cluster-cutover/ https://stebas.com/posts/rds-instance-to-multi-az-cluster-cutover/ Mon, 24 Aug 2026 00:00:00 GMT A planned cutover with near-zero data loss — and three traps: "maintenance" that did not stop writes, passwords RDS will not let you copy, and a dump that took 25 minutes over VPN and 31 seconds in the VPC. https://stebas.com/posts/kafka-consumer-stall-hidden-by-group-rename/ https://stebas.com/posts/kafka-consumer-stall-hidden-by-group-rename/ Fri, 21 Aug 2026 00:00:00 GMT Consumer lag grew for five days, then dropped to zero and stayed there. It looked self-healed. In fact the consumer was stuck, the metric had vanished, and the alert had resolved itself. https://stebas.com/posts/python-memory-leak-memray-in-production/ https://stebas.com/posts/python-memory-leak-memray-in-production/ Mon, 17 Aug 2026 00:00:00 GMT A backend OOMed every 10–15 hours. Hourly memray snapshots on the live pod found the leak in library middleware, not the app — after a first attempt took production down for seven minutes. https://stebas.com/posts/longhorn-to-linstor-migration/ https://stebas.com/posts/longhorn-to-linstor-migration/ Sun, 16 Aug 2026 00:00:00 GMT Why we swapped Longhorn for LINSTOR/DRBD on a self-managed cluster, how each workload was moved with a different method, and what went wrong on the way. https://stebas.com/posts/ingress-nginx-to-envoy-gateway/ https://stebas.com/posts/ingress-nginx-to-envoy-gateway/ Sat, 15 Aug 2026 00:00:00 GMT ingress-nginx is retired and blocks Kubernetes upgrades. How we moved 16 public hosts to Envoy Gateway by building the new entry point next to the old one and swapping only the address at the end. https://stebas.com/posts/cnpg-barman-cloud-plugin-switchover/ https://stebas.com/posts/cnpg-barman-cloud-plugin-switchover/ Fri, 14 Aug 2026 00:00:00 GMT CNPG is removing native barmanObjectStore support. The switch has no valid intermediate state, the database had no base backups at all, and one path decides whether point-in-time recovery still works. https://stebas.com/posts/four-checks-that-lied/ https://stebas.com/posts/four-checks-that-lied/ Fri, 14 Aug 2026 00:00:00 GMT In one day of platform migrations, four verification steps reported failure while everything was healthy. When one signal disagrees with all the others, suspect the signal first. https://stebas.com/posts/alerts-that-cannot-fire/ https://stebas.com/posts/alerts-that-cannot-fire/ Thu, 13 Aug 2026 00:00:00 GMT A Vault peer stayed sealed for 32 hours and secrets stopped syncing for a week, with a VaultSealed alert in the repo and Alertmanager healthy. Two independent failures hid it. https://stebas.com/posts/kernel-drift-was-uptime-drift/ https://stebas.com/posts/kernel-drift-was-uptime-drift/ Thu, 13 Aug 2026 00:00:00 GMT Three different kernels across seven nodes looked like unfinished patch waves and sent an investigation down a false trail. Every node had the same patches; only uptime differed. https://stebas.com/posts/kubectl-merge-patch-wiped-coredns/ https://stebas.com/posts/kubectl-merge-patch-wiped-coredns/ Thu, 13 Aug 2026 00:00:00 GMT A one-line image bump replaced the whole container spec. CoreDNS lost its config, quietly fell back to the whoami plugin and answered every query with NOERROR and no answer. https://stebas.com/posts/metallb-to-cilium-lb-ipam/ https://stebas.com/posts/metallb-to-cilium-lb-ipam/ Thu, 13 Aug 2026 00:00:00 GMT The cluster had exactly one LoadBalancer IP and every route depended on it. How I handed it from MetalLB to Cilium with zero failed requests — by letting both controllers run at once. https://stebas.com/posts/vault-transit-auto-unseal/ https://stebas.com/posts/vault-transit-auto-unseal/ Thu, 13 Aug 2026 00:00:00 GMT Every Vault pod restart came back sealed and froze secret sync behind it. How we added a transit auto-unseal provider, migrated a three-node raft cluster, and what bit us. https://stebas.com/posts/vm-backup-freeze-cascade/ https://stebas.com/posts/vm-backup-freeze-cascade/ Thu, 13 Aug 2026 00:00:00 GMT Four Vault seals in three weeks, each treated as a separate incident. A journald gap analysis showed the hypervisor stunning the VM during backups, and Longhorn turning that into a cascade. https://stebas.com/posts/gitlab-upgrade-signals-that-lie/ https://stebas.com/posts/gitlab-upgrade-signals-that-lie/ Wed, 12 Aug 2026 00:00:00 GMT A routine minor upgrade of self-hosted GitLab in Docker. A pre-flight check claimed 32 pending migrations, and the container reported healthy while serving nothing. https://stebas.com/posts/rolling-os-update-seven-nodes/ https://stebas.com/posts/rolling-os-update-seven-nodes/ Wed, 12 Aug 2026 00:00:00 GMT A rolling Ubuntu point update across a cluster that holds all its own state. The pre-flight checks turned a scary job into a boring one, and Longhorn decided the pace. https://stebas.com/posts/stale-dhcp-route-killed-cluster/ https://stebas.com/posts/stale-dhcp-route-killed-cluster/ Mon, 10 Aug 2026 00:00:00 GMT After a host reboot about fifty pods crashlooped while every node was Ready. The cause was a forgotten DHCP classless-route option that silently removed the default gateway. https://stebas.com/posts/vmalert-rules-no-hot-reload/ https://stebas.com/posts/vmalert-rules-no-hot-reload/ Fri, 07 Aug 2026 00:00:00 GMT New Kafka alert rules went through Git and Argo CD into the ConfigMap, yet vmalert never evaluated them. The GitOps pipeline broke on its very last step. https://stebas.com/posts/grafana-capacity-forecast-dashboard/ https://stebas.com/posts/grafana-capacity-forecast-dashboard/ Thu, 06 Aug 2026 00:00:00 GMT Turning a memory trend into a time-to-OOM and time-to-full ETA in PromQL, the two bugs that made it lie, and the leak it caught on day one. https://stebas.com/posts/innodb-online-uncertain-rescan/ https://stebas.com/posts/innodb-online-uncertain-rescan/ Thu, 06 Aug 2026 00:00:00 GMT A router flapping, an operator stuck in a lock for two days, a CRD frozen at 2/3 — while all three InnoDB Cluster members were healthy. The fix was one metadata rescan. https://stebas.com/posts/vso-stuck-after-vault-seal/ https://stebas.com/posts/vso-stuck-after-vault-seal/ Thu, 06 Aug 2026 00:00:00 GMT 38 of 43 VaultStaticSecrets stuck for two days with a frozen error. Unsealing Vault and restarting the operator did nothing; a one-field spec change did. https://stebas.com/posts/wordpress-redirect-loop-per-pod-salts/ https://stebas.com/posts/wordpress-redirect-loop-per-pod-salts/ Thu, 06 Aug 2026 00:00:00 GMT wp-admin bounced back to the login page with reauth=1, but not always. The cause was random auth salts generated per pod behind a non-sticky load balancer. https://stebas.com/posts/dbaas-master-ip-change/ https://stebas.com/posts/dbaas-master-ip-change/ Fri, 31 Jul 2026 00:00:00 GMT Login 500s, backends in CrashLoopBackOff and a stuck rollout looked like three problems. All of them were one hardcoded database IP in a pgbouncer config. https://stebas.com/posts/longhorn-eio-was-node-oom/ https://stebas.com/posts/longhorn-eio-was-node-oom/ Fri, 31 Jul 2026 00:00:00 GMT EIO, read-only ext4, a corrupted MySQL datadir and a sealed Vault looked like a failing disk. Host dmesg showed a node-wide OOM that killed the Longhorn instance-manager first. https://stebas.com/posts/vault-shamir-reseals-on-restart/ https://stebas.com/posts/vault-shamir-reseals-on-restart/ Fri, 31 Jul 2026 00:00:00 GMT A Vault pod sat at Ready 0/1 for days while raft quorum quietly held on two peers. Nothing was sealing it: Shamir seal without auto-unseal leaves every restart sealed. https://stebas.com/posts/longhorn-online-expansion-stuck-on-bad-node/ https://stebas.com/posts/longhorn-online-expansion-stuck-on-bad-node/ Thu, 30 Jul 2026 00:00:00 GMT A PVC sat in Resizing forever while Longhorn retried the expansion every five seconds without an error. The volume was locked to one unhealthy node by a stale attachment ticket. https://stebas.com/posts/resize-immutable-sts-field/ https://stebas.com/posts/resize-immutable-sts-field/ Thu, 30 Jul 2026 00:00:00 GMT Git said 20 GiB, the live StatefulSet ran 6 GiB templates and Argo CD said Synced. How orphan deletion plus OnDelete rebuilt the template with zero downtime. https://stebas.com/posts/crd-status-lies-stalled-controller/ https://stebas.com/posts/crd-status-lies-stalled-controller/ Tue, 28 Jul 2026 00:00:00 GMT A MySQL InnoDB Cluster ran without HA for five days while its status, the pod phase and the readiness gate all said healthy. All three came from one stalled operator. https://stebas.com/posts/helm-pending-upgrade-lock/ https://stebas.com/posts/helm-pending-upgrade-lock/ Mon, 13 Jul 2026 00:00:00 GMT A cancelled CI job left two Helm releases stuck in pending-upgrade and blocked every deploy. How to find the lock, release it safely and check a whole namespace at once. https://stebas.com/posts/hello/ https://stebas.com/posts/hello/ Thu, 16 Apr 2026 00:00:00 GMT Platform as product, GitOps, CI/CD, DevSecOps — predictable releases and security in the pipeline, not after the fact.