A GitLab upgrade and two signals that lied

I run GitLab in Docker in my home lab. It is the source of truth for all my GitOps, so a minor upgrade (19.1.2 to 19.2.2) is not something I do casually. It was a planned job, not an incident, yet two things nearly derailed it — and both looked like the opposite of what they were.

Signal one: 32 unfinished migrations

GitLab will not let you upgrade while batched background migrations are still running. My pre-flight query reported 32 unfinished migrations, which would have been a hard blocker.

It was a measurement error. The status column has two terminal values: 3 means finished and 6 means finalized. Filtering on status != 3 counts every finalized migration as pending. The documented criterion is status NOT IN (3, 6), and by that criterion the real number was zero. A side note: querying Postgres directly answered instantly, while the equivalent gitlab-rails runner call took more than two minutes.

Signal two: healthy in 60 seconds

After the restart Docker reported the container as healthy within about a minute. It was serving nothing.

The image’s health check covers only a narrow part of the stack. Chef starts Postgres, Redis, Gitaly and a few others first, runs the schema migrations, and only then starts the web tier. So healthy fired while Puma, Nginx, Workhorse and Sidekiq did not exist yet. When Nginx did start, it returned 502 for two full minutes — so even a TCP port probe would have reported success in the middle of the outage.

The procedure

I ran a fixed order: verify → back up → pull → swap → apply → verify.

• A logical backup with gitlab-backup create, plus a tarball of the config directory, which the backup deliberately omits (gitlab-secrets.json lives there).
• An off-host copy of the backup, checked with sha256sum on both ends.
• docker pull of the new 3.9 GB image before stopping the old container, which kept downtime to 5 minutes 59 seconds against a forecast of 10–25 minutes.
• Swap the pinned tag in the compose file, docker compose down && up -d.
• Wait for the real completion marker in the logs, not “Chef Client finished”.

And I accepted the result only on user-visible signals: the sign-in page returning 200 and ssh -T git@… greeting me. Not on the vendor’s health flag.

Why the backup goes off-host first

Rollback is asymmetric. Before the new container runs migrations, rollback takes seconds: restore the old compose file and start the old image. After the migrations there is no tag revert, because 19.2 rewrites the schema and 19.1 cannot read it. The only way back is a full restore inside the old image — hours. That asymmetry is why the backup leaves the host before the pull.

Takeaways

• Know the terminal states of whatever you are counting. != finished is not the same as not in (finished, finalized).
• A container health check is the vendor’s opinion. Accept an upgrade on what users see: HTTP 200 and a working git-over-SSH.
• Pull before you stop. Downtime is the restart, not the download.
• Respect the upgrade path. GitLab has required stops; a stop is a ceiling, not a target — any version below the next stop is reachable in one hop.
• Never run such upgrades unattended. A cron job that applies database migrations with nobody watching is a quiet way to lose a weekend.