Changing an immutable StatefulSet field without restarting a single pod

At Optimacros the GitOps repository declared a MySQL InnoDB Cluster with a 20 GiB data volume template and a tuned MySQL config. The live StatefulSet had 6 GiB templates and MySQL binlog defaults, including a 30-day retention. Argo CD reported Synced and Healthy the whole time.

Why the drift was invisible

Both fields only take effect when the StatefulSet is built.

• spec.volumeClaimTemplates on a StatefulSet is immutable. It cannot be patched.
• The operator renders the MySQL config into the pod configuration when it creates the StatefulSet. A later sync does not re-render it.

selfHeal gave false confidence. It reconciled the custom resource — and the custom resource did say 20 GiB. But the operator does not, and cannot, mutate an existing StatefulSet template, and it does not rebuild the StatefulSet on its own. Existing PVCs also keep their size. So both the template and the live volumes were stale while every GitOps indicator looked perfect.

Two separate problems

It helped to split this in two, because they have different fixes.

The existing PVCs. These can simply be expanded live. A patch to storage: 20Gi and Longhorn’s online expansion did the rest (one of them needed extra work, which is a story of its own).

The StatefulSet template. This needs the StatefulSet to be recreated so it is rebuilt from the custom resource. The question was how to do that to a running database without a restart.

Orphan, recreate, adopt

kubectl delete sts <name> --cascade=orphan deletes the StatefulSet object but leaves its pods running. A new StatefulSet with the same selector then adopts them.

The key to zero downtime was the update strategy. With updateStrategy: OnDelete, an adopted pod is not restarted just because the template changed; pods are only updated when someone deletes them. With RollingUpdate, adoption would see a template-hash difference and roll every member — something you do in a maintenance window, not on a Tuesday afternoon.

The sequence:

1. Back up the StatefulSet spec and confirm updateStrategy is OnDelete. Record each pod’s controller-revision-hash and age.
2. Delete the StatefulSet with --cascade=orphan. The pods keep serving.
3. The operator did not recreate the StatefulSet, so I applied it myself from the backup: storage bumped to 20 GiB, runtime fields stripped (status, uid, resourceVersion, managedFields and the like).
4. Verify: the template says 20 GiB, ready is 3/3, and every pod kept its revision hash and creation time — nothing restarted.

Then I committed the same change in Git, so the next operator-driven rebuild would not diverge again.

The risky window

Between the orphan delete and the recreate, nothing controls those pods. If one dies in that window, nothing brings it back. Keep the window short and have the manifest ready before you delete anything.

Takeaways

• Synced means Git matches the objects Argo CD manages. It says nothing about fields that only apply at creation time.
• Fields that are immutable or rendered once can drift silently forever. Know which ones they are for every operator you run.
• --cascade=orphan plus updateStrategy: OnDelete lets you rebuild a StatefulSet template without touching the data plane.
• With RollingUpdate, the same trick restarts every member. Check the strategy before you start, not after.