RDS refused connections at 10% CPU: default connection pools times replicas

During load tests on a staging environment at Gambchamp, .NET services started failing to open database connections. The request that reached me was simple: “we need a much bigger database”. CPU on the RDS instance had spiked to 97% in the same windows, so the request looked reasonable.

The shape of the graph said otherwise.

A plateau, not a curve

CloudWatch DatabaseConnections showed a flat line at 412–424 for tens of minutes. The seven-day average was about 7. Capacity problems usually bend a curve; a flat top is a limit being hit.

The instance was a db.t4g.medium with the default max_connections formula, which works out to 399. Subtract superuser_reserved_connections=3 and reserved_connections=2, and ordinary roles get 394 slots.

Where 400 connections came from

None of the connection strings set Max Pool Size, so Npgsql used its default: 100 connections per process, per connection string. There was no PgBouncer and no RDS Proxy; every pod connected directly.

The PostgreSQL log in the peak window showed three client IPs on one database — the three replicas of an integration service — and one more on another database. Three times 100 plus 100 is roughly 400: the application ceiling, almost to the connection.

The proof was a specific FATAL line, 18 times in one test and 47 in the next:

remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections" role

Two readings that mislead

• CloudWatch peaked above max_connections (424 > 399). The metric also counts backends that were forked and then rejected with FATAL, so it overshoots during a storm.
• too many clients already never appeared. That message means even the reserved slots are gone. The “remaining connection slots are reserved” message fires earlier, when only the ordinary slots run out — which is exactly what happened here.

Logs Insights limited to the peak window was the only practical way to see this: a plain filter over a log group with infinite retention simply never finished.

A bigger instance moves the wall

A teammate resized the instance to db.m7g.large, which raises the limit to 832 (827 for applications). That buys headroom, but it does not remove the wall: 17 pods mount database connection strings, so the theoretical demand is at least 17 × 100 = 1,700.

The real fix is on the application side: an explicit Max Pool Size per service so that the sum of replicas × pool size stays at or below about 80% of the application slots. I handed that to the service owners with the numbers. If replica counts become dynamic with HPA, the next step is RDS Proxy or PgBouncer. I also added a node dashboard so the next load test shows node saturation too — 19 of 20 deployments in that namespace had no resource limits.

Takeaways

• A flat plateau in connections is a limit, not a capacity curve. Check the FATAL text before resizing anything.
• Npgsql pools are per process and per connection string: a pod talking to three databases can hold 300 connections.
• Budget connections explicitly: Σ(replicas × pool size) ≤ ~80% of max_connections minus reserved slots.
• CloudWatch DatabaseConnections can exceed max_connections during a rejection storm. That is not a bug in the limit.