ONLINE_UNCERTAIN: when the MySQL operator and the database disagree

At Optimacros a three-node MySQL InnoDB Cluster, run by the Oracle MySQL Operator, backs the company’s marketing sites. One morning the trail of symptoms was four layers deep, and every layer pointed somewhere else.

It started as “the router is OOMing”. Then the MySQL Router pods were restarting in Error. Then “the database is degraded”. The cluster’s custom resource said ONLINE_UNCERTAIN, two of three instances online, and its lastProbeTime had not moved for two days. The operator logged not ONLINE for one member every two seconds, plus Cluster status results inconsistent.

Ask the database, not the operator

The obvious next step would have been to rebuild the “dead” member. Instead I asked Group Replication directly, read-only, on the primary:

SELECT member_host, member_state
FROM performance_schema.replication_group_members;

All three members were ONLINE. The supposedly broken one had rejoined through incremental recovery two days earlier. The database was healthy. The operator’s view was not.

Why the operator lied

That member’s data directory had been rebuilt a while before, and a rebuilt or cloned MySQL instance always gets a new server_uuid. Group Replication knew it by the new UUID. The cluster metadata schema still held the old one.

The operator matches Group Replication members against the metadata by UUID. It looked for the old UUID, never found it online, declared the member down and reported an inconsistent status. Worse, it held a cluster busy-lock while doing so: its status handler had retried more than 27,000 times with the lock owned by that member, which is why the custom resource froze. MySQL Shell’s own cluster.status() said it plainly: OK_PARTIAL, with an instance error suggesting cluster.rescan().

The router flapped because the topology it was shown looked unstable. Every symptom upstream was real; none of them was the cause.

The fix

First a backup of the metadata schema only, with mysqldump --single-transaction. Then, from the primary:

dba.getCluster().rescan()

rescan() updates the metadata to match reality: it recorded the member’s new UUID and dropped an orphaned recovery account. It touches metadata, not application data, and it is idempotent. The operator immediately got a consistent view, released the lock and set the custom resource back to ONLINE with three instances. If the lock had lingered, restarting the operator Deployment would have been the next step.

What not to do

A few days earlier I had rebuilt members of this cluster for real, recreating their volumes, because their data was broken. That memory made a rebuild the tempting move here too. It would have been wrong: the member was already back in the group, and only the bookkeeping was stale. Rebuilding a healthy member adds risk and fixes nothing.

Takeaways

• For an InnoDB Cluster, replication_group_members on the primary is the ground truth. An operator’s .status is an opinion that can freeze.
• A rebuilt or cloned member always gets a new server_uuid. Make cluster.rescan() the standard last step of every member recovery.
• A frozen lastProbeTime and a retry counter in the tens of thousands mean a stuck controller, not a broken database.
• Back up the metadata schema before touching it, even when the operation is documented as safe.