← All posts

CI was red, and the CI code was fine: two silent expiries behind one Vault

A pipeline failed at its first stage, and every later stage was skipped. The CI code never ran: Vault was crashlooping on an expired auto-unseal token, and once it was back up, its serving certificate had expired too.

A pipeline failed at the very first stage. Its only gate, a Vault smoke test that logs in with AppRole, timed out, and every later stage (build, workspaces, tests, maintenance) was marked skipped:

curl: (28) Failed to connect to vault.example.com port 8200 after 133008 ms

It looked like “CI broke”. It was not CI: the job never got to run anything of its own.

Layer one: Vault was not listening

Read-only, one step at a time:

getent hosts vault.example.com                      # resolves: not DNS
ping -c2 <vault-host>                               # host is up
: < /dev/tcp/<vault-host>/8200 && echo open || echo refused
# refused: nothing listens on the port
docker ps -a | grep vault                           # Restarting (1)
docker logs --tail 20 vault                         # 403 invalid token on transit/encrypt

This Vault runs as a plain Docker container and auto-unseals through the transit engine of another Vault. Its token in the seal stanza had reached its TTL. The upstream Vault answered 403 invalid token, the container could not unseal, exited and restarted, forever. The CI runner saw a 133-second timeout, my laptop saw Connection refused. Different firewall paths, same fact: Vault was down.

The fix was a new token, with the same narrow policy (update on the transit encrypt and decrypt paths only), made periodic and orphaned so it would not expire again. I tested it before trusting it: an encrypt → decrypt roundtrip on the transit key with the new token. Then I backed up the config, swapped the token and restarted the container. Vault auto-unsealed.

Layer two: the retry failed differently

The retried pipeline failed again, now with:

curl: (60) SSL certificate problem: certificate has expired

The listener on 8200 still served a wildcard certificate that had expired two weeks earlier. All hosts had moved to a new certificate in a planned swap, but this one was missed because it had never been inventoried. While Vault was down nobody could notice. As soon as it came up, strict TLS in CI noticed in seconds.

Before installing the new certificate I checked the SANs and dates, and that the key modulus matched the certificate:

diff <(openssl x509 -noout -modulus -in fullchain.pem) \
     <(openssl rsa  -noout -modulus -in privkey.pem)

Backup, swap, restart. Then a check the way CI connects, with strict TLS and no -k: sealed: false. The pipeline went green.

A health check with -k lies

The usual quick check is curl -sk .../v1/sys/health. It would have said “Vault is up” while every real client rejected the certificate. Check the way your real consumer connects.

What we pay

A non-expiring token is a trade-off: a stolen bearer token lives forever. It is limited to transit unseal by policy and is an orphan, and it can be revoked by its accessor, which I saved. The better long-term option is AppRole with a short secret_id TTL and a bound CIDR instead of a raw token on disk.

Takeaways

  • CI red ≠ CI broken. If the first gate fails and everything else is skipped, the problem is in a dependency, not in the code.
  • Two expiries were stacked: one hid the other. After fixing the first, expect the second to show up immediately.
  • Tokens and certificates that expire silently are the same class of problem. The real root cause was no expiry alerting: no alert on the token TTL, no alert on the certificate date. Fixing the incident does not fix the class.
  • An auto-unseal through another Vault is a hidden dependency: this CI’s health now depends on a different environment’s Vault being reachable and unsealed.
  • Verify every credential before using it, and back up every file before you change it. Both changes were reversible by design.