The certificates expired on schedule. The alert fired for a week — into nowhere

One morning every kubectl call against a bare-metal cluster at Optimacros failed with the same line:

x509: certificate has expired or is not yet valid: current time ... is after 2026-10-09T04:52:15Z

Running pods kept serving traffic, so from the outside nothing looked broken. Inside, the controller-manager and the scheduler were crash-looping with 28–31 restarts each. Nothing new could be scheduled and nothing was being reconciled. CI jobs that needed fresh pods simply failed.

Exactly one year

The date gave it away. The cluster had been installed by kubespray on 2025-10-09 and never upgraded since. kubeadm issues its leaf certificates — the API server, the kubelet client, front-proxy, and the kubeconfigs for admin, controller-manager and scheduler — for one year. A kubeadm upgrade renews them as a side effect, so clusters that get upgraded regularly never notice. This one had not been touched, and kubespray’s auto_renew_certificates was off. All of them expired in the same second.

What was not affected matters just as much for the fix: the CA is valid for ten years, kubespray’s etcd certificates for a hundred, and kubelet client certificates rotate on their own. So this was a renewal, not a rebuild.

The alert did fire

My first assumption was “we had no alert for this”. That turned out to be wrong, and the truth was worse. kube-prometheus-stack ships KubeClientCertificateExpiration out of the box. I queried Prometheus for its history:

count_over_time(ALERTS{alertname="KubeClientCertificateExpiration",alertstate="firing"}[10d])

It had been firing as a warning for about seven days and as critical for the last day. Then I looked at the generated Alertmanager config. The only receiver was "null" — the chart default, because the Argo CD Application had never set alertmanager.config. The alert worked perfectly. It just had nowhere to go.

The fix, one master at a time

The cluster has three control-plane nodes, so I renewed them one by one and verified each before touching the next:

1. Back up /etc/kubernetes to a tarball.
2. kubeadm certs renew all with the cluster’s kubeadm config. It keeps the existing SANs.
3. Move the static pod manifests out, wait until the API server, controller-manager and scheduler are gone, move them back.
4. Check readyz and, more importantly, the certificate that is actually being served:

echo | openssl s_client -connect <master>:6443 2>/dev/null | openssl x509 -noout -enddate

Step 3 is the one people skip. kubeadm certs renew only writes files. The running processes keep the old certificate in memory until the static pods restart, so “the files are new” proves nothing.

The laptop kubeconfig also had to be replaced with the new admin.conf. One more trap: kubespray points server: at 127.0.0.1:6443, so a copied file gives you connection refused until you rewrite the address.

Making sure it does not happen again

Prevention: a monthly systemd timer that renews the certificates, a clone of what kubespray would install with auto_renew_certificates: true, staggered by master so they never restart together. I ran it once by hand and checked the served certificate again — a timer you have never seen succeed is a hope, not a control.

Detection is the real gap, and it is wider than this cluster: an alert without a receiver is decoration. Wiring Alertmanager delivery through GitOps was recorded as a separate decision with its own trigger.

Takeaways

• kubeadm clusters that are not upgraded at least once a year will lose their control plane on the anniversary of the install.
• After renewing certificates, verify what the endpoint serves, not what is on disk.
• “We had no alert” and “the alert went nowhere” look the same during an incident. Check receivers on every cluster — a lone null means nobody is listening.
• Keep a separate admin kubeconfig on another master: when the first one restarts, you still have a way in.