← All posts

Synced, Healthy and changed nothing: a values key the chart never reads

The merge request was approved, Argo CD was green and the behaviour did not change. Helm silently ignores keys it does not know, so the render stayed the same and the diff was honestly empty.

A change to a RabbitMQ deployment went through review and was merged. Argo CD showed Synced and Healthy on the new commit. The broker behaved exactly as before.

I have written before about Argo CD reporting Synced while the API server pruned a field it did not know. This is a quieter relative of that bug. Nothing was pruned and nothing was rejected. Argo CD did not lie at all.

What Synced actually answers

Synced answers one question: does what I rendered match what is in the cluster? It does not answer “did my commit arrive”, and it does not answer “did the behaviour change”. At least four different causes give you the same green status:

  1. A child application is always green. Its status says nothing about whether the parent has picked up your commit. Compare the parent’s status.sync.revision with git.
  2. Synced to an older commit. reconciledAt can be fresh while operationState.finishedAt is a week old. Reconciliation is a comparison, not an apply: if the diff is empty, no operation runs.
  3. The render did not change. This post.
  4. The object changed, the process did not re-read it. A ConfigMap is updated, but the application reads it only at startup.

A key in the wrong place

The change added memoryHighWatermark at the top level of the values file. The chart reads it as config.memoryHighWatermark. Helm does not validate values against anything unless the chart ships a JSON schema, and most do not. Unknown keys are silently ignored.

So the render was byte-for-byte the same as before, the diff against the cluster was empty, and Argo CD correctly reported that everything matched. Every check was green because, from every tool’s point of view, nothing had changed. And that was true.

Check the render, not the status

The fastest way to catch this is to render both versions and compare:

helm template <release> <chart> --version <ver> -f values-old.yaml > /tmp/old.yaml
helm template <release> <chart> --version <ver> -f values-new.yaml > /tmp/new.yaml
diff /tmp/old.yaml /tmp/new.yaml

An empty diff for a non-empty change means the key is wrong. Then look for the real name in helm show values <chart> --version <ver>, and pay attention to nesting. Chart values.yaml files often put a section title in a comment, while the key itself sits one level deeper. It is easy to copy the title instead of the path.

The same check works well in CI: render the chart for the merge request and fail, or at least warn, when a values change produces no manifest change.

A short checklist for “it’s green but it didn’t work”

  1. Is the parent’s revision the commit from git?
  2. Is operationState.finishedAt recent?
  3. Did the chart’s render actually change?
  4. Did the process re-read its configuration, or is only the file on disk new?

A green status is not an answer to any of these four questions. The only honest check is observable behaviour: the service’s response, the output of a diagnostic command, the value at runtime.

Takeaways

  • Helm ignores unknown values keys without a word. A typo or wrong nesting looks exactly like a no-op.
  • An empty render diff for a real change is a bug in the change.
  • Synced compares two renders. It never checks that you meant what you wrote.
  • Verify a configuration change by its effect, not by the status of the pipeline that delivered it.