A sleep(3) race, proven with a dose-response test
About a third of workspace provisioning runs failed with two unrelated-looking errors, and the CI retry made it worse. Both came from one fixed timer racing systemd inside an LXC container. Changing the timer from 3 to 5, 7 and 10 seconds turned a correlation into proof.
A CI farm builds test workspaces: a vendor installer creates an LXC container, starts it and runs a setup script inside. About a third of the runs failed. Retries were a lottery, and on one host they almost never helped. Two errors appeared, and they looked unrelated:
- on the first attempt:
getcwd() failed: No such file or directory; - on the retries:
chown: cannot access '/run/mysqld'.
One had been blamed on a downgrade, the other on that host being “special”. Both explanations were wrong.
Read the code, not the logs
The installer starts the container and then waits a fixed sleep(3) before running the setup script. It does not wait for systemd inside the container to finish early boot. One early-boot unit matters here: systemd-tmpfiles-setup. It wipes /tmp and it creates /run/mysqld. Its start floats behind systemd-journal-flush, which took anywhere from 0.7 to 5 seconds.
That explained both errors as one race:
- First attempt. The setup script first spends a second or two unpacking empty databases. Then tmpfiles runs late, wipes the script’s working directory in
/tmp, and a child shell printsgetcwd() failedto stderr. The installer treats any stderr output as failure. - Retry. The data already exists, so the script reaches
chown /run/mysqldin under a second, before tmpfiles has created the directory.
The CI retry made it worse: it only ran stop → sleep 90 → start, without the clean reinstall. Every retry took the fast path straight into the race.
Correlation is not proof
Log timings matched the theory, but timings that match are still a correlation. To prove cause, I changed the dose: one free host, the installer’s cached code edited to use a different sleep value, one run per value on the retry path, and the original restored and checksummed at the end.
Times from container start; margin = setup script start minus tmpfiles done:
| sleep | journal flush | margin | result |
|---|---|---|---|
| 3, fresh | 0.68 s | +3.14 s | ✅ |
| 3, retry | 4.85 s | ≈ −1.1 s | ❌ chown /run/mysqld |
| 5, retry | 4.79 s | +0.98 s | ✅ borderline |
| 7, retry | 2.10 s | +5.71 s | ✅ |
| 10, retry | 5.01 s | +5.78 s | ✅ slowest flush seen |
The failure appeared on demand at 3 and disappeared at 5, 7 and 10. The setup script always started at sleep(N) + 1.3 s, so the timer was the only gate. The variable was the journal flush. With one run per value this is a demonstration, not a failure rate, but it is causal evidence.
A second, independent stall
The same runs showed another problem: a network init unit stalled every container boot for three minutes. It is ordered before basic.target and runs netplan apply. networkctl reload needs D-Bus, D-Bus waits for basic.target, and basic.target waits for the network unit. A deadlock, broken only by 90-second timeouts. The network itself was configured 13.5 seconds after start; only the redundant reload hung. It was not related to the sleep: the stall was the same at every value.
The fix
The workaround, sleep(10), is valid by margin, but it is still a timer. The cached code it patches is also lost on a reboot of the host, and then the race returns silently. The real fix is to wait for the event instead of a duration: systemctl is-system-running --wait (or waiting for sysinit.target) inside the container before running the setup. Also: cd / and mkdir -p /run/mysqld in the script, and do not treat stderr output as failure.
Takeaways
- A fixed
sleepbefore using something another process creates is a race with a margin. It works until that margin disappears. - Two different errors can be one race, depending on which path the code takes to the same timing window.
- A retry that skips the clean-up step can hit a bug more reliably than the first attempt.
- To prove cause, change the dose and watch the effect come and go. That needs one free host and an hour, and it ends the discussion.
- Measure “why this host fails most” (here a slower journal flush) separately from the root cause. They are different questions.