Five days at 1/3 with every dashboard green: when a CRD status lies

At Optimacros a MySQL InnoDB Cluster backs the cluster that serves the company’s marketing sites. It is meant to run three members, so it survives the loss of one. For about five days it ran on one member out of three — no HA at all — and every dashboard was green.

Three signals, one lie

When I finally looked closely, three independent-looking signals all said “healthy”:

• the InnoDBCluster custom resource reported ONLINE with onlineInstances: 3;
• the pods were Running — the broken secondaries still had a live sidecar container;
• the readiness gate set by the operator was True.

The cluster UI rendered that .status as if it were current, so anyone glancing at it saw three healthy members while two of them were dead.

Why all three agreed

They were not independent at all. All three were written by the same MySQL operator, and its reconciler had stalled. The giveaway was .status.cluster.lastProbeTime: it was frozen days in the past.

A status field written by a controller is only as fresh as the controller. When the controller hangs, the last value stays in place, and it does not look stale — it looks authoritative. That is worse than no data: it is active misinformation. Our monitoring was wired to exactly those signals, the ones that depend on the component that failed, so nothing ever contradicted the lie.

Reading ground truth

The fix was to read signals that do not pass through the operator:

• the StatefulSet’s readyReplicas, which kube-state-metrics reads straight from the API server;
• the Group Replication member count from mysqld-exporter;
• the MySQL Router’s own view of live members, visible in its logs.

All three agreed with each other — and disagreed with the custom resource. A rollout restart of the operator un-froze it, and within about 30 seconds the status showed a fresh probe time and the real picture. Restarting the operator is safe: it only orchestrates, Group Replication keeps running on its own. It does not repair a dead member, though; that was a separate rebuild.

The trust chain

I now think of these signals as a chain from fact to hearsay:

containerStatuses[].ready → StatefulSet.status.readyReplicas → pod.status.phase → CustomResource.status

Page on the leftmost signal that is cheap to collect, never on the rightmost. We added a PrometheusRule on kube_statefulset_status_replicas_ready < 3 for five minutes. It does not try to detect a stalled operator — that is hard from Prometheus, since a CR status is not a metric. It simply stops depending on the signal that can stall.

Takeaways

• A controller-written .status is a hint, not a fact. If its probe or heartbeat timestamp is old, everything else in it is old too.
• When several “independent” health signals come from one component, you have one signal, not several.
• Alert on ground truth that does not share a failure domain with the thing it watches.
• The same applies to every operator-managed resource: databases, Vault, cert-manager certificates, Argo CD application health.