← All posts

The certificate was valid. The CI image just did not know its new root

Overnight a CI job started failing with "self signed certificate in certificate chain", while the browser opened the same URL fine. The server had switched to a newer CA root, and the job image sat on an end-of-life Debian whose trust store predated it.

One morning a release job in GitLab CI failed after 3.3 seconds. The build never started: the first step asks the issue tracker for the ticket title, and curl stopped right there:

curl: (60) SSL certificate problem: self signed certificate in certificate chain

The developer who reported it added a reasonable-looking argument: “the tracker opens fine in the browser”. That sounds like proof the certificate is fine. In fact it confirms a certificate problem: the browser ships its own trust store, and the job does not use it.

The wording sends you the wrong way

“Self signed” sounds like someone slipped in a fake certificate, a TLS-inspecting proxy, maybe a MITM. Every root certificate is self-signed by definition. The message means exactly one thing: the chain ended at a root that is not in my trust store.

A control job beats any reasoning

Three hypotheses, in order:

  1. The tracker broke. From a workstation the chain was complete and valid: leaf, intermediate, root, verify=0, HTTP 200. Ruled out.
  2. The network broke, or a proxy appeared. In the same pipeline another job called the same URL with the same curl, and it had passed after the failures started. That one green job ruled out the network, DNS, the tracker and the credentials in one go.
  3. The image. The only difference: the green job used an Alpine image from a shared template, while the red one inherited the project default — a Node image on Debian 11 “bullseye”.

When the pipeline has a job doing the same thing and staying green, everything they share is cleared, and only the differences remain.

What changed overnight

The job history via the GitLab API gave the boundary: last success in the afternoon, first failure the next morning, with no commits in between. Something outside the repository had changed.

It was the tracker’s wildcard certificate. It had been reissued, and the chain of trust changed with it: the old issuer chained to GlobalSign Root CA R3 (2009, present practically everywhere), the new one to GlobalSign Root R46 (2019, only in recent CA bundles). The image carried ca-certificates from January 2021. It had R3 and did not have R46.

Proof by reproduction

The base image is public, so the hypothesis could be tested locally in a minute:

docker run --rm node:22.16-bullseye \
  curl -fsS https://tracker.example.com/
# curl: (60) SSL certificate problem: self signed certificate in certificate chain

docker run --rm alpine:3 sh -c \
  'apk add -q ca-certificates curl; curl -so /dev/null -w "verify=%{ssl_verify_result}\n" https://tracker.example.com/'
# verify=0

docker run --rm node:22.16-bullseye sh -c \
  'ls /usr/share/ca-certificates/mozilla/ | grep -i "R46\|Root_CA_-_R3"'
# GlobalSign_Root_CA_-_R3.crt   <- R46 is missing

The fix is one line, and three traps

FROM node:22-bookworm   # was: node:22.16-bullseye

Getting that line to production had three traps:

  • Same tag, same failure. The runner pulls with if-not-present. It already had the old image locally and would never fetch a rebuilt one under the same tag. The fix “doesn’t help”. The image needs a new tag.
  • A fix in main does not reach the pipeline. The project included the CI template by a pinned tag, not by branch. A commit changes nothing until a new tag is cut and the consumer bumps its ref.
  • You cannot patch the trust store with apt. Debian 11 is end-of-life and its packages are gone from the mirrors. apt-get update still succeeds (the metadata is served), which makes it more confusing: only the install returns 404. Only a new base helps.

And --insecure is not an option here, even though the template already used it elsewhere: this curl carried a service account’s credentials. If you really need a workaround, pass --cacert with the right root, and keep verification on.

Takeaways

  • A CA root change is a breaking change for every client with an old trust store, even when the certificate itself is perfect. Inventory images on EOL bases before you swap certificates.
  • “Self signed certificate in certificate chain” from a client means a missing root on the client side, not a forged certificate.
  • Find a green sibling job that does the same thing. It removes every shared suspect at once.
  • Reproduce on the public base image locally. It takes a minute and turns a guess into proof.
  • Fixes in CI images need a new tag, and fixes in versioned CI templates need a new release. Otherwise you “fix” it and nothing changes.