GitOps and CI/CD
Argo CD, Helm and GitLab CI: a safe path from commit to production and the cases where Synced does not mean deployed.
8 posts
- Argo CD said Synced, and the API server had quietly thrown my fix awayMonths of missing metrics, a fix already in git and a green Synced status. The field did not exist in the CRD, so the API server pruned it and nobody complained.
- A GitLab upgrade and two signals that liedA routine minor upgrade of self-hosted GitLab in Docker. A pre-flight check claimed 32 pending migrations, and the container reported healthy while serving nothing.
- Merged, synced, mounted — and still not alerting: vmalert without hot reloadNew Kafka alert rules went through Git and Argo CD into the ConfigMap, yet vmalert never evaluated them. The GitOps pipeline broke on its very last step.
- A WordPress login loop that only happened sometimeswp-admin bounced back to the login page with reauth=1, but not always. The cause was random auth salts generated per pod behind a non-sticky load balancer.
- Three incidents that were one: a database IP baked into a configLogin 500s, backends in CrashLoopBackOff and a stuck rollout looked like three problems. All of them were one hardcoded database IP in a pgbouncer config.
- Changing an immutable StatefulSet field without restarting a single podGit said 20 GiB, the live StatefulSet ran 6 GiB templates and Argo CD said Synced. How orphan deletion plus OnDelete rebuilt the template with zero downtime.
- Helm says "another operation is in progress" — and nothing is runningA cancelled CI job left two Helm releases stuck in pending-upgrade and blocked every deploy. How to find the lock, release it safely and check a whole namespace at once.
- How I think about the platformPlatform as product, GitOps, CI/CD, DevSecOps — predictable releases and security in the pipeline, not after the fact.