Argo CD said Synced, and the API server had quietly thrown my fix away

A side-project app in my home lab had zero application metrics for months. Its Postgres runs on CloudNativePG, the operator had created a PodMonitor, and the GitOps repo already carried a fix for the missing scrape. Argo CD showed Synced / Healthy on exactly the commit with that fix. Meanwhile a replica failure went unnoticed for 23 hours, because nothing was watching.

Measure the blind spot first

Before touching anything I counted. Prometheus had 46 active targets and none in that namespace. Not “fewer than expected” — none. That number turned a vague feeling into a concrete question: why is a PodMonitor that clearly exists scraped by nobody?

Two defects, each harmless-looking on its own

Label mismatch. kube-prometheus-stack, with chart defaults, only selects monitors that carry its release label. The PodMonitor the CNPG operator creates has app.kubernetes.io/* and cnpg.io/cluster labels and nothing else. So the object exists, looks like coverage when you inspect it, and no Prometheus ever reads it. The same default had silently dropped cert-manager’s ServiceMonitor too.

A field that does not exist. The fix in git tried to solve this from the CNPG side, adding labels via spec.monitoring.podMonitorMetadata.labels. That field is not in the CRD schema of the CNPG version I run. With structural schemas the API server prunes unknown fields on apply. No error, no warning in Argo CD. Argo compared the pruned live object with the pruned desired state, found them equal and reported Synced.

The key step was asking the CRD directly whether the field exists:

kubectl get crd clusters.postgresql.cnpg.io -o json \
  | jq -r '.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.monitoring.properties | keys[]'

podMonitorMetadata was not in the list. Case closed.

Fix the consumer, not every producer

Instead of chasing labels on every operator-created monitor, I changed Prometheus: set the chart’s *SelectorNilUsesHelmValues options to false, so the selectors are nil and Prometheus picks up every ServiceMonitor and PodMonitor in the cluster. One change fixed CNPG, cert-manager and every monitor a future chart will create. Right after the sync cnpg_collector_up appeared for every Postgres pod.

I deliberately left ruleSelector strict. An extra scraped metric is harmless; an extra firing alert is noise. The price is that every PrometheusRule must carry the release label, or it loads into nothing — the same trap, one layer over, so I check it whenever I add rules.

Rollback is trivial: remove the two lines and Prometheus goes back to the strict selector without losing data.

An alternative was CNPG’s spec.inheritedMetadata.labels, but it labels every object of the cluster including the Postgres pods, and changing pod labels means a rolling restart of the database. Fixing the consumer avoided that.

Takeaways

• A green Synced means “what the API server kept matches git after it kept it”. It says nothing about fields that were pruned on the way.
• When a GitOps change has no effect, ask the CRD schema whether the field exists before debugging the controller.
• An object that exists is not coverage. Count active targets, do not eyeball monitors.
• Prefer fixing the selector on the consumer side once over relabelling every producer forever.