Four checks that lied while the system was fine

During one working day of platform migrations in my home lab, four separate verification steps reported failure or absence while the thing they checked was healthy the whole time. Each looked like a real defect. Each cost me between a few minutes and an hour of misdirected work.

The individual bugs are trivia. The pattern is not: when a system is green everywhere except one signal, the most likely broken component is that signal.

1. A field that does not exist in this version

I waited four minutes on .status.valid of a VaultStaticSecret. This version of the operator reports SecretSynced, Healthy and Ready conditions instead. A jsonpath on a missing field returns an empty string, which is indistinguishable from “not ready yet”.

2. A label selector that matched nothing

I polled a plugin pod by app.kubernetes.io/name for five minutes while the pod was Running under different labels. An empty list reads as “not created yet” and never times out into an error.

3. An ambiguous short name

With Longhorn installed, kubectl get backup resolves to backups.longhorn.io, not to the CloudNativePG backups.postgresql.cnpg.io I wanted. A completed database backup looked statusless on every poll, and delete failed with backups.longhorn.io ... not found.

4. The wrong apply semantics

kubectl apply --dry-run=server rejected a correct manifest. Client-side three-way merge cannot know that a field missing from the file should be deleted when the live object has no last-applied-configuration annotation. That object had been created by Argo CD with server-side apply. Validating with --server-side --field-manager=argocd-controller accepted it immediately.

The same family, pointing the other way

Two more checks that week lied in the opposite direction:

• curl -w '%{http_code}' returns 000 for every transport failure. The real answer was the exit code, 35, a TLS error, which means the connection was actually established. Reading 000 as “no connectivity” sent me to the network layer instead of to SNI.
• A bad patch removed a container’s readiness probe, and the pod reported Ready forever, because a container with no probe is vacuously ready. That one caused a two-hour DNS outage.

The second direction is the dangerous one. A check that falsely reports failure costs time. A check that falsely reports success hides an outage, and nothing prompts you to look at it again.

The habit that resolves all of them

• Suspect the check first when everything else is green. If nodes, pods, controllers and traffic are healthy and exactly one probe disagrees, the probe leads the suspect list.
• Pair every check with a control. A positive result means nothing without a known-good comparison, a negative one nothing without a known-bad one.
• Prefer the protocol’s answer over a wrapper’s summary. pg_stat_archiver over a ContinuousArchiving condition, a real login over “the Secret exists”.
• Check the probe’s own vocabulary. Does the field exist in this version, does the selector match, is the resource name unambiguous, is this the same apply mechanism the real actor uses?

A control can be destructive too

Testing a wrong password against a real Grafana account five times tripped its brute-force protection, which then rejected the correct password for five minutes. My passing check turned into a failing one and invented a bug that did not exist. Run negative controls against an identity that does not exist, and never keep polling a locked account: every attempt extends the block.

Takeaways

• One red signal among many green ones most likely means a broken signal.
• Empty output is not success. Make a probe tell “not yet” from “cannot answer”.
• Use fully qualified resource names whenever two CRDs share a kind.
• Validate with the same apply mechanism the real controller uses.
• Be as suspicious of green as of red.