kubectl patch --type merge wiped CoreDNS, and every health check stayed green

In my home lab, cluster-wide DNS died for new lookups while every health signal stayed green. Pods failed with bad address for every name, but connecting by IP worked. CoreDNS pods were 1/1 Running, Ready, zero restarts. Argo CD was Synced, Postgres healthy, every public endpoint returned 200. CoreDNS logs looked busy and healthy: a steady stream of NOERROR. And dig said status: NOERROR with ANSWER: 0.

Everything already connected kept working. Only fresh lookups failed, which is why it took a while to even see.

The change that caused it

I had bumped the CoreDNS image with what looked like a harmless one-liner:

kubectl -n kube-system patch deploy coredns --type merge \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"coredns","image":"…"}]}}}}'

--type merge is a JSON Merge Patch (RFC 7386), and a merge patch replaces arrays wholesale. It does not merge list items by name — that is what a strategic merge patch does. So the entire containers array was replaced with my one-element object, and everything I had not repeated was deleted: the -conf /etc/coredns/Corefile argument, the volume mounts, ports, both probes and resources.

Why it looked healthy

Without -conf, CoreDNS never reads the mounted Corefile. It starts on its built-in default, which is a single server block with the whoami plugin.

whoami answers every query with NOERROR, puts the client’s own address in the ADDITIONAL section, adds an SRV record with the client’s port, and leaves ANSWER empty. So resolvers saw no answer and failed; dig +short printed nothing, which reads like “no response”; logs looked like normal NOERROR traffic. And the pods were Ready because the readiness probe had been deleted by the same patch — a container with no probe is always ready.

How I found it

I lost time on plausible false leads: the Cilium eBPF service map briefly held one of two CoreDNS backends after the rollout, endpoints sat in a restoring state after an agent restart, and I even toggled a Cilium DNS proxy setting. None of it was the cause.

What worked was a raw DNS query sent from a node, with a control: the same query to a known-good resolver returned one answer, while CoreDNS returned zero answers and two additional records. Decoding the bytes showed an A record holding my own address and an SRV record with TTL 0 — the whoami fingerprint. A quick check confirmed it: a real CoreDNS also serves /ready on 8181 and metrics on 9153, and this one served only port 53.

The fix

I restored the Deployment from the dump I had taken before the change — minutes of work. Reconstructing the container spec from memory would not have been. Later the same day I redid the bump with kubectl set image, and the new version turned out to be completely uneventful.

Takeaways

• --type merge on anything containing a list is destructive. Containers, volumes, env, ports, tolerations are all arrays. Use kubectl set image, a strategic merge patch (the default kubectl patch), or apply a full manifest.
• Ready is only as meaningful as the probe behind it. If a component is “Ready but wrong”, check that the probe still exists.
• dig +short hides ADDITIONAL-only answers. Read the full response before concluding there was no reply.
• Diagnose with a control. Comparing against a known-good resolver separates “my tool lies” from “the server is broken”.
• Dump an object before hand-editing a platform component, and verify the spec survived before trusting any rollout.