A production certificate expired and nobody noticed: the front proxy covered for it

At Optimacros the task arrived as routine: “new certificates for the cluster that serves the marketing sites”, an archive with the certificate, the key and the CA bundle. During the inventory I found that the certificate in the cluster had expired 49 minutes earlier — and nothing had told anyone. No alert, no user, no monitoring. All five public hosts returned 200.

Why the outage did not show

The topology is hybrid. A front proxy on a separate VM terminates TLS itself and talks to the cluster origin without verifying its certificate. Someone had already deployed the new certificate on that proxy a week earlier: the serial of the live :443 matched the file in the archive byte for byte. So from outside, everything looked fine.

The broken path was the other one: the direct entrance to the cluster gateway by SNI. Health checks, monitoring and anything that bypasses the front use it. For them the service was down.

“The site returns 200” proves nothing about TLS when a proxy with its own termination sits in front. Check the path you are fixing, not the path that is convenient to check:

echo | openssl s_client -connect <gateway-ip>:443 -servername <host> \
  | openssl x509 -noout -serial -enddate

Fix it at the source

The Kubernetes secrets were managed by the Vault Secrets Operator, so a kubectl apply would have been overwritten within five minutes. The certificate had to go into Vault, and the operator would roll it out.

Before writing anything I checked the material: the key modulus matched the certificate, and openssl verify passed against the CA bundle. Then a backup of the current Vault versions, then the write — of the full chain, not just the leaf. A local openssl says OK for a leaf-only file because the intermediate is already in the workstation’s trust store, while mobile clients and minimal images fail with unable to get local issuer certificate. The verification that matters is curl -w '%{ssl_verify_result}' from a clean client.

There were five copies of the secret; only one was referenced by the gateway listener, the rest were leftovers of an earlier migration. I updated all of them so stale objects would not mislead the next investigation. Total time: about 15 minutes, no downtime.

The traps on the way

• The issuing CA had changed. Nothing broke only because nothing verified the origin certificate. Any future backend TLS policy would turn the same swap into an outage.
• My KUBECONFIG was pinned to another cluster. The first kubectl get nodes showed the wrong nodes, and a corporate wildcard almost went to the wrong place. Before any mutation, confirm which cluster you are in.

Takeaways

• The real root cause was not the late renewal but the absence of a certificate-expiry alert. A blackbox probe on probe_ssl_earliest_cert_expiry covers the whole fleet at once.
• A proxy that terminates TLS can hide a dead certificate behind it. Probe every entrance separately.
• Fix secrets where their source of truth lives, or the operator will undo your fix.
• Always ship the full chain and verify with a client that has no extra intermediates.