The MySQL operator missed one event, and the config in Git never reached mysqld
Too many connections on every site, a cluster at 2/3 for forty hours, and Argo CD saying Synced. One storage stall triggered it; two hidden operator behaviours turned it into an outage in waiting.
The request was vague: “something is wrong with the database”. By the time I looked, the sites returned 200. But the history was worse than the current state:
- the evening before, for about ninety minutes, every WordPress site on that MySQL InnoDB Cluster logged
Too many connectionsandLock wait timeout, hundreds per hour; - a critical “cluster degraded” alert and a
KubePodNotReadyfor one member had been firing for forty hours. The cluster wasONLINE_PARTIAL, 2 of 3, with zero fault tolerance, and nobody had acted.
Three layers, one trigger.
The trigger: storage on one hypervisor
Both VMs on one hypervisor host, a control-plane node and the worker carrying the primary, showed 85–108 % disk busy and 18–23 seconds average write latency. VMs on the other hosts were flat. InnoDB on the primary piled up semaphore waits, running threads went from 4 to 144, and the connection limit ran out. The etcd member on the same host slowed the API server, and leader-elected controllers restarted across the cluster.
Layer two: the operator lost an event
During that stall one member pod was recreated. The operator adds a member to Group Replication only on the pod create or restart event, not from its periodic probe. With the API server timing out, that handler never ran. mysqld started cleanly, the datadir was fine, but the member sat OFFLINE and its readiness gate was never set. The lastProbeTime in the custom resource stayed at the moment of the lost event.
What did not help: restarting the operator made its status honest again, but did not rejoin the member. Deleting the member pod made it worse: the operator treated the delete as “remove this instance from the cluster metadata”, and then logged ignored pod event for the new pod. Don’t do this.
What worked was a GTID check first: the member’s gtid_executed was a subset of the primary’s, there were no errant transactions, and the primary had not purged anything the member still needed. So an incremental recovery was possible:
dba.getCluster().addInstance(member, {recoveryMethod: 'incremental'})
RECOVERING → ONLINE in about nine minutes. No clone, no volume rebuild.
Layer three: the config in Git never arrived
Why was the connection limit 151 at all? Git declared max_connections=500, a one-day binlog retention and binlog compression in the cluster’s spec.mycnf, and Argo CD showed Synced. But the operator renders spec.mycnf into its init ConfigMap once, when the cluster is created. These settings had been added later, and that file had stayed empty for three and a half months.
The ground truth is in the database, not in Git:
SELECT VARIABLE_NAME, VARIABLE_SOURCE
FROM performance_schema.variables_info
WHERE VARIABLE_NAME = 'max_connections';
-- COMPILED <- a value you set should never say COMPILED
A side effect: with the default 30-day binlog retention, binlogs used 12 of 15 GB on the 20 GiB data volumes. The next outage was already on its way.
Only after the cluster was back at 3/3: SET PERSIST of the three variables on every member (no restart needed, and it survives restarts), a binlog flush that freed the volumes from 76 % to 22 %, and the ConfigMap patched so a rebuilt member starts with the right config.
The order matters. Never purge binlogs on the donor while a member is recovering incrementally: it is reading exactly those files, and the recovery would fall back to a full clone.
Takeaways
- An operator that reacts to events can lose an event when the API server is slow. If the status is frozen at one moment, check what it missed.
- Deleting a pod is not a harmless nudge with every operator. Read what its delete handler does first.
Syncedonly proves the custom resource equals Git. For config that an operator renders, check what the process actually runs.- Raising the connection limit does not fix a storage stall; it only turns errors into longer waits. The root cause was below the cluster, in hypervisor storage.
- A critical alert firing for forty hours is a routing and ownership gap, not a monitoring success.