← All posts

A sleep(3) race, proven with a dose-response test

About a third of workspace provisioning runs failed with two unrelated-looking errors, and the CI retry made it worse. Both came from one fixed timer racing systemd inside an LXC container. Changing the timer from 3 to 5, 7 and 10 seconds turned a correlation into proof.

A CI farm builds test workspaces: a vendor installer creates an LXC container, starts it and runs a setup script inside. About a third of the runs failed. Retries were a lottery, and on one host they almost never helped. Two errors appeared, and they looked unrelated:

  • on the first attempt: getcwd() failed: No such file or directory;
  • on the retries: chown: cannot access '/run/mysqld'.

One had been blamed on a downgrade, the other on that host being “special”. Both explanations were wrong.

Read the code, not the logs

The installer starts the container and then waits a fixed sleep(3) before running the setup script. It does not wait for systemd inside the container to finish early boot. One early-boot unit matters here: systemd-tmpfiles-setup. It wipes /tmp and it creates /run/mysqld. Its start floats behind systemd-journal-flush, which took anywhere from 0.7 to 5 seconds.

That explained both errors as one race:

  • First attempt. The setup script first spends a second or two unpacking empty databases. Then tmpfiles runs late, wipes the script’s working directory in /tmp, and a child shell prints getcwd() failed to stderr. The installer treats any stderr output as failure.
  • Retry. The data already exists, so the script reaches chown /run/mysqld in under a second, before tmpfiles has created the directory.

The CI retry made it worse: it only ran stop → sleep 90 → start, without the clean reinstall. Every retry took the fast path straight into the race.

Correlation is not proof

Log timings matched the theory, but timings that match are still a correlation. To prove cause, I changed the dose: one free host, the installer’s cached code edited to use a different sleep value, one run per value on the retry path, and the original restored and checksummed at the end.

Times from container start; margin = setup script start minus tmpfiles done:

sleep journal flush margin result
3, fresh 0.68 s +3.14 s ✅
3, retry 4.85 s ≈ −1.1 s ❌ chown /run/mysqld
5, retry 4.79 s +0.98 s ✅ borderline
7, retry 2.10 s +5.71 s ✅
10, retry 5.01 s +5.78 s ✅ slowest flush seen

The failure appeared on demand at 3 and disappeared at 5, 7 and 10. The setup script always started at sleep(N) + 1.3 s, so the timer was the only gate. The variable was the journal flush. With one run per value this is a demonstration, not a failure rate, but it is causal evidence.

A second, independent stall

The same runs showed another problem: a network init unit stalled every container boot for three minutes. It is ordered before basic.target and runs netplan apply. networkctl reload needs D-Bus, D-Bus waits for basic.target, and basic.target waits for the network unit. A deadlock, broken only by 90-second timeouts. The network itself was configured 13.5 seconds after start; only the redundant reload hung. It was not related to the sleep: the stall was the same at every value.

The fix

The workaround, sleep(10), is valid by margin, but it is still a timer. The cached code it patches is also lost on a reboot of the host, and then the race returns silently. The real fix is to wait for the event instead of a duration: systemctl is-system-running --wait (or waiting for sysinit.target) inside the container before running the setup. Also: cd / and mkdir -p /run/mysqld in the script, and do not treat stderr output as failure.

Takeaways

  • A fixed sleep before using something another process creates is a race with a margin. It works until that margin disappears.
  • Two different errors can be one race, depending on which path the code takes to the same timing window.
  • A retry that skips the clean-up step can hit a bug more reliably than the first attempt.
  • To prove cause, change the dose and watch the effect come and go. That needs one free host and an hour, and it ends the discussion.
  • Measure “why this host fails most” (here a slower journal flush) separately from the root cause. They are different questions.