Databases
PostgreSQL, CloudNativePG, Amazon RDS and MySQL InnoDB Cluster: replication, cutovers, connection pools and recovery.
8 posts
- RDS refused connections at 10% CPU: default connection pools times replicasLoad tests hit a flat plateau of ~420 database connections and the request was for a much bigger database. The real ceiling was max_connections versus Npgsql default pools.
- Argo CD said Synced, and the API server had quietly thrown my fix awayMonths of missing metrics, a fix already in git and a green Synced status. The field did not exist in the CRD, so the API server pruned it and nobody complained.
- A Postgres replica stuck in a pg_rewind loop for 23 hours: rebuild, do not repairAfter a failover a CloudNativePG replica could not rewind: its WAL segments were zero bytes and its data directory a tenth of the primary. Nobody noticed for 23 hours because nothing scraped the operator metrics.
- Moving production Postgres from an RDS Multi-AZ instance to a Multi-AZ DB clusterA planned cutover with near-zero data loss — and three traps: "maintenance" that did not stop writes, passwords RDS will not let you copy, and a dump that took 25 minutes over VPN and 31 seconds in the VPC.
- Moving CloudNativePG to the Barman Cloud plugin without splitting the archiveCNPG is removing native barmanObjectStore support. The switch has no valid intermediate state, the database had no base backups at all, and one path decides whether point-in-time recovery still works.
- ONLINE_UNCERTAIN: when the MySQL operator and the database disagreeA router flapping, an operator stuck in a lock for two days, a CRD frozen at 2/3 — while all three InnoDB Cluster members were healthy. The fix was one metadata rescan.
- Three incidents that were one: a database IP baked into a configLogin 500s, backends in CrashLoopBackOff and a stuck rollout looked like three problems. All of them were one hardcoded database IP in a pgbouncer config.
- Five days at 1/3 with every dashboard green: when a CRD status liesA MySQL InnoDB Cluster ran without HA for five days while its status, the pod phase and the readiness gate all said healthy. All three came from one stalled operator.