Three incidents that were one: a database IP baked into a config

The report was the usual one: “the stand is down”. It was a load-testing environment at Gambchamp, and on the surface it had three separate problems at once.

• Logging into the admin panel returned 500.
• Several backend pods sat in CrashLoopBackOff with hundreds of restarts.
• A deploy hung halfway: new pods never passed readiness, old ones kept serving traffic.

It is tempting to open three tickets here. That would have been three wrong investigations.

The first wrong guess

The application logs were full of this:

asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation

That reads like an application bug or a flaky network inside the cluster. But every backend talked to PostgreSQL through a shared pgbouncer, and its log said something more precise:

pooler error: server login has been failing, cached error: connect failed (server_login_retry)

connect failed is easy to misread as an authentication problem. It is not. pgbouncer could not even open a TCP connection to the database server. The application error was only a downstream echo of that.

Proving it from inside the cluster

Two read-only commands from inside the pgbouncer pod settled it: resolve the provider’s stable master hostname, and try to open port 5432 on the address from our manifest. The hostname resolved to a new address. The old address no longer listened.

The managed PostgreSQL node had been recreated by the provider and got a new IP. In the GitOps repository this one environment pointed pgbouncer and the postgres exporter at the database by IP address. Every other environment used the provider’s master FQDN, which follows the current master through recreation and failover.

The history made it worse: there was an older commit “point exporter at correct DB IP”. It had already happened once and had been fixed the same fragile way.

The fix

1. Replace the IP with the master FQDN in the pgbouncer config and in the exporter, render the chart locally, merge.
2. Argo CD synced the Secret, because self-heal was on.
3. Restart pgbouncer by hand. This step is easy to forget: pgbouncer.ini is mounted as a file, pgbouncer does not re-read it, and there was no reloader annotation on that Deployment. The same applied to the exporter.

After the restart the new pod logged normal login attempt lines and zero connect failed. The backends came out of the crash loop on their own, and the stuck rollout finished by itself.

Why it looked like three problems

The backends’ startup and liveness probes touched the database. When the database was unreachable, Kubernetes saw broken pods and kept restarting them, and the new ReplicaSet could never become ready. One missing dependency produced three very different-looking symptoms.

Takeaways

• connect failed (server_login_retry) in pgbouncer means a dead endpoint, not a wrong password. Look at the address before the credentials.
• A crash loop, a hung deploy and a 500 on login can be a single cause one layer down. Find the shared dependency first.
• Point clients at the managed database’s stable FQDN, never at a node IP. An IP in a manifest is a mine, not a setting.
• If a config is mounted as a file and the process does not watch it, a GitOps sync is only half of the change. Either add a reloader or make the restart part of the procedure.