← All posts

A node reboot froze dozens of test environments: their readiness gate was a pod nobody owned

After an in-place worker reboot, every test environment with data on that node hung in Init for hours. The node, the volumes and the platform were fine. The setup steps ran as plain Pods without a controller, the reboot deleted them, and the apps waited for a pod that no longer existed.

During an OS upgrade on a work cluster I rebooted the workers in place, one at a time. The node came back Ready, platform pods were healthy, local volumes were intact. And yet every test environment whose data lived on that node stayed down: about 43 of roughly 60 pods per environment sat in Init:0/3 or Init:CrashLoopBackOff, for hours.

One environment had been broken in exactly the same way since a node disruption a day earlier. Nobody had connected it to the node.

What the init containers were waiting for

The init container logs of an app pod gave the answer in one line:

kubectl -n <env-ns> logs <app-pod> -c migrate-completed
# Error from server (NotFound): pods "migrate" not found

Each environment is set up by the CI pipeline as a chain of steps: init, a couple of database init steps, migrate, ready. Each step is a plain Pod without a controller. The application pods do not start until those pods have completed: their init containers poll for the pod with a small wait script, roughly wait until pod "migrate" is Completed.

That works on a fresh deploy. It falls apart the first time a node goes away.

Root cause: nothing recreates an ownerless pod

A node reboot with kubelet graceful shutdown, an eviction, or a drain deletes the pods on that node. Pods that belong to a Deployment, StatefulSet or Job come back, because their controller recreates them. A pod with no ownerReferences is simply gone.

So the completed migrate pod disappeared with the reboot. The wait loop in every app pod started polling a pod that no longer existed, and the whole environment hung in Init until someone ran the CI deploy again.

A quick way to see which environments are blocked:

kubectl get pods -A -o json | jq -r '
  [.items[] | select(.metadata.namespace | test("^test-[0-9]+$"))]
  | group_by(.metadata.namespace)[]
  | (.[0].metadata.namespace) as $ns
  | ([.[] | select(.metadata.name == "migrate")][0].status.phase // "MISSING") as $m
  | "\($ns) migrate=\($m)"'

Getting the environments back without CI

For the next nodes I changed the procedure: save the ownerless pods before the reboot, recreate them after it.

# before the reboot: save specs of pods without an owner
kubectl get pods -A -o json \
  | jq -c '.items[] | select(.metadata.namespace | test("^test-")) | select(.metadata.ownerReferences == null)' \
  | while read -r p; do
      ns=$(jq -r .metadata.namespace <<<"$p"); n=$(jq -r .metadata.name <<<"$p")
      jq '{apiVersion:"v1", kind:"Pod",
           metadata:{name:.metadata.name, namespace:.metadata.namespace, labels:.metadata.labels},
           spec:(.spec | del(.nodeName))}' <<<"$p" > "bare-pods/$ns--$n.json"
    done

# after the node is back: recreate missing or Failed ones
for f in bare-pods/*.json; do
  ns=$(jq -r .metadata.namespace "$f"); n=$(jq -r .metadata.name "$f")
  ph=$(kubectl -n "$ns" get pod "$n" -o jsonpath='{.status.phase}' 2>/dev/null)
  [ "$ph" = Failed ] && kubectl -n "$ns" delete pod "$n" --wait=true && ph=""
  [ -z "$ph" ] && kubectl create -f "$f"
done

Removing nodeName matters: without it the scheduler places the pod again instead of pinning it to a node that may not be there. migrate runs again on the existing data. That is only safe because migrations for that app version are idempotent, so I checked that first.

Result: environments on two of the three rebooted nodes came back without CI. The ones on the first node, broken before I had saved anything, needed a redeploy.

Two practical notes:

  • The saved specs can contain environment values. Keep them out of any repository and chmod 600 them.
  • Arm the restore step only after the node is cordoned. A check like “wait until the node is Ready and schedulable” is already true before the reboot starts, and fires immediately.

The real fix belongs to the chart

The workaround makes reboots survivable. It does not fix the design. Setup steps should be Jobs or Helm hooks, and the apps should gate on Job completion, not on the existence of a pod. A Job keeps its completed status after its pod is gone. A bare pod’s status disappears with the pod.

Takeaways

  • A pod without ownerReferences is one node event away from disappearing for good. kubectl get pods -A -o json | jq '... ownerReferences == null' is worth running on any cluster you are about to drain.
  • “Wait for pod X to complete” is a fragile contract. Wait for a Job, or for a state that outlives the pod.
  • Any drain, eviction, node failure or graceful kubelet shutdown hits the same flaw, not only planned reboots. If one environment has been “randomly” broken for a day, look at what happened to its node.
  • A platform that is fully healthy while the apps are stuck usually means the coupling is in the app layer. Read the init container logs before you touch the node.