The loading screen was a retry loop, not a hang — and the reboot was never needed

This one is from my own workstation, not from a cluster, but the lesson transfers directly to production.

Crimson Desert, running under Proton Experimental on Ubuntu, sometimes got stuck on the save-loading screen forever. The known workaround was a full machine reboot. It happened “sometimes”, with no obvious trigger.

Hung or spinning?

The first question decides everything: is the process waiting on something, or burning CPU? A process in D state is blocked in the kernel, usually on I/O, and cannot be killed. A process in R state is running and will die from a signal.

ps showed R. Per thread, three loader threads were each pegged at 100 % CPU, and their wchan was 0 — spinning in user space, not blocked in the kernel.

The decisive measurement

High CPU alone does not prove a loop; slow loading also burns CPU. Progress does. I read /proc/<pid>/io twice, five seconds apart:

Signal	Delta over 5 s
read_bytes	0
syscr	+38 468

About 7 700 read syscalls per second, and not a single byte actually read. The game log named the error: ReadFile failing with Windows error 6, ERROR_INVALID_HANDLE, retried in an unbounded loop with no backoff and no attempt limit. The loading screen was not waiting for anything — it was hammering reads that could never succeed.

Ruled out by measurement, not by guessing

• stuck disk I/O — no processes in D state, zero bytes read, plenty of free space;
• network or cloud-save stall — no outgoing connections at all;
• memory pressure — tens of gigabytes free, no swap in use;
• a corrupt save — the log showed the save loading successfully before the loop began; the failure started about eight seconds later, during asset streaming;
• a GPU hang — the GPU load was the loading-screen animation, and it dropped instantly when the process went away.

The fix

Because the process was in R state, a signal was enough:

pkill -TERM -f CrimsonDesert.exe

It exited cleanly on SIGTERM. GPU load fell from 99 % to 2 %. The reboot had always been overkill.

What triggers the invalid handle is still unknown — most likely a race in the game’s asset streaming or in the experimental Proton build. Next time I will have PROTON_LOG=1 on, so the trace names the file.

Takeaways

• R versus D decides whether you need a reboot at all. A spinning process is not hung.
• CPU high + I/O counters flat = retry loop. Measure the delta over a few seconds before reaching for the big hammer.
• Rule hypotheses out with numbers. Each “maybe it’s the disk / network / GPU” took one command to disprove.
• The same fingerprint shows up everywhere: an init container retrying a failing mount, a sidecar retrying a dead endpoint, a CI step retrying a 404.