AWS
AWS in production: Amazon EKS and RDS, Multi-AZ cutovers and the limits you meet before the CPU graph does.
4 posts
- RDS refused connections at 10% CPU: default connection pools times replicasLoad tests hit a flat plateau of ~420 database connections and the request was for a much bigger database. The real ceiling was max_connections versus Npgsql default pools.
- Kafka at 94 % disk with 17 hours left, and retention that kept twice what we askedTraffic grew 150x and nobody resized the volumes. Worse, with default segment rolling a 7-day retention really meant 7 to 14 days. How we bought time and fixed the cause.
- NotReady for six hours, and the node was alive the whole timeEvery signal said the node was dead. It was thrashing on memory, and the obvious fix — delete the node and force-detach the volume — was one command away from corrupting a database.
- Moving production Postgres from an RDS Multi-AZ instance to a Multi-AZ DB clusterA planned cutover with near-zero data loss — and three traps: "maintenance" that did not stop writes, passwords RDS will not let you copy, and a dump that took 25 minutes over VPN and 31 seconds in the VPC.