A GitHub Actions runner pod writes Acknowledging runner request '96a86da3-...', writes one more line carrying an AAD correlation ID, and then writes nothing for the rest of its life. The pod stays Running. No Runner.Worker process ever appears. Ten minutes later the service annotates the job with a lost-communication error, and the pod is still sitting in your scale set holding a slot. Three separate runner issues filed in the first two weeks of September 2026 describe that silence, and they are three different bugs.

The last log line marks a timeout boundary

The reason the log stops precisely there is visible in the runner's own source. BrokerMessageListener does not apply a global deadline to its work. It applies deadlines per call site, by hand, and only some call sites got one.

AcknowledgeRunnerRequestAsync runs under new CancellationTokenSource(TimeSpan.FromSeconds(5)). Session creation is bounded at roughly 100 seconds. The call that comes next, a POST to acquirejob on the run_service_url that arrived inside the RunnerJobRequest message, has no timeout of its own and inherits whatever token the caller passed down. Issue #4703, filed against runner 2.336.0 on 10 September, puts that contrast at the centre of its diagnosis and names GetJobMessageAsync as the unbounded call. So the last bounded operation completed and logged, the first unbounded one is still waiting, and the process has no mechanism to notice.

Listener: GET /message long poll, held up to ~50smessageType RunnerJobRequest?AcknowledgeRunnerRequestAsyncbounded by CTS 5sPOST acquirejob at run_service_urlno per-call deadlineSpawn Runner.Workerrenewjob every 60sBlocked body readpod alive, no Worker, no further log lineService annotates lost communication after ~10 minlistener never exits, slot stays occupied"202 Accepted""yes""body completes""connection half-open"

The block is a body read, not a connect and not a header read. The analysis in issue #4668 (runner 2.337.0, ARC 0.14.2, filed 2 September) places it after the status line and headers arrive but before the response body completes. If the peer's FIN or RST never reaches the runner, which is the ordinary result of a middlebox dropping state, a NAT table flush, or a load balancer losing a connection record, nothing wakes the read. There is no TCP keepalive on the client socket to probe a dead peer. .NET has a long-standing shape for this: dotnet/runtime #36822 documents a response stream that "is never closed even though cancelation token is set" when content is read outside the scope the timeout actually covers. The reporter in #4668 saw four instances on a single date, which reads as one transient network event rather than anything job-specific.

Version pinning does not save you here, and it is worth saying plainly because the reflex on this site has been to pin harder. If you already pin every action by SHA, the pin covers what runs inside the job. It cannot cover a listener that never got as far as starting one. The same asymmetry showed up in the Node 20 removal on hosted runners, where the failure lived in the runner's own launch path rather than in anything the workflow declared.

One hung pod is not one lost job

The blast radius is capacity, not the build. A hung listener still occupies a slot in the scale set, so the ARC controller counts it as busy and declines to schedule a replacement. Four hung pods against a maxRunners of eight cuts your fleet in half, and no ARC metric distinguishes "running a job" from "blocked on a socket read".

Every control a team reaches for in this situation is already defeated. A job-level timeout-minutes never arms, because the job never started. A liveness probe on the runner process passes, because the process is alive and healthy by any definition the kubelet can check. ss shows the connection to the Actions service as ESTABLISHED with Recv-Q and Send-Q both empty, which is indistinguishable from a healthy long-poll on a quiet queue. This is the same failure mode as the class of breakage that lands after CI has already gone green: all the signals report success and the thing you care about is gone anyway.

The mirror image: the job that never arrives

Issue #4728 looks like the same problem in the UI and is the opposite bug. The runner here is persistent, not ephemeral, it is idle and healthy, and it is the client that hangs up on the service.

The reported mechanism: when a job finishes, OnJobStatus sets _runnerStatus = Online and cancels _getMessagesTokenSource, which aborts the in-flight GET /message?status=Busy long poll. If the service had already written a job assignment onto that open poll, the client throws away a message the service considers delivered. Redelivery arrives roughly 61 seconds later, past the assignment's validity window, which produces a 409 MissingKey and then a cancellation. The job surfaces as "The job was not acquired by Runner of type self-hosted even after multiple attempts". Nothing is stuck. Deleting the runner accomplishes nothing, and restarting it accomplishes nothing, because the runner was never the thing holding state. The behaviour is reported from 2.336.0 onward.

Every thread and every vendor runbook I read on these four collapses them into the same advice: delete the stuck runner. That clears the symptom for two of the four, does nothing for #4728, and teaches you nothing about which one you are looking at, so it comes back next week.

What I read to sort this out: runner issues #4668, #4703 and #4728, actions-runner-controller #4307, and Depot's walkthrough of the listener's message loop, which is the clearest public description of where the long poll sits relative to job acquisition. All four issues are open as of 22 September 2026.

Four log signatures, four decisions

Splitting one symptom into its distinct causes is the same exercise as four separate causes behind one Node mark-compact error: the fix is only correct once you know which branch you are on.

  • Last line is the AAD correlation ID after Acknowledging runner request, pod Running, no Runner.Worker child: the unbounded acquirejob body read (#4668, #4703). Kill the pod and go look at the network path, not the runner config. Reported on both 2.336.0 and 2.337.0.
  • The job was not acquired by Runner of type self-hosted even after multiple attempts on a healthy idle persistent runner: the aborted Busy poll (#4728), 2.336.0 and later. Do not delete anything.
  • Skipping message Job. Job message not found '<guid>'. job was canceled, often next to a registration-not-found error: the ARC cancellation path (#4307, reported on ARC 0.13.0). The pod does need deleting, because ARC ships no cleanup for this state.
  • Pod gone or OOMKilled while the EphemeralRunner object still reads Running: a controller reconciliation gap with nothing to do with the listener.

The signal that separates the first case from a genuinely busy runner is a process check, not a log grep. A log line proves the listener reached a point; it cannot prove the listener is still moving. pgrep -f Runner.Worker returning nothing while pgrep -f Runner.Listener returns a PID, held for more than about 90 seconds after an acknowledgement, is the state that identifies the hang, and it answers offline with no API call.

What to arm this week, and the condition for ripping it out

  1. Add a watchdog that arms on the log line, not on pod age. Grep Acknowledging runner request out of /home/runner/_diag/Runner_*.log; once it appears, wait 120 seconds and require a Runner.Worker PID. If there is none, pkill -f Runner.Listener so the runner container exits and ARC reclaims the slot. As a sidecar this needs shareProcessNamespace: true on the pod spec, otherwise the sidecar cannot see or signal the runner's processes at all. Failing loudly beats any dashboard here, because the dashboard is already green.
  2. Accept the trade honestly. A 120-second threshold will occasionally kill a pod whose worker spawn was merely slow under node pressure, and that job gets retried. I would rather eat a rare retry than lose half the fleet to slots that nothing on the cluster reports as broken, but measure your own worker-spawn distribution before picking the number. If your p99 acknowledgement-to-worker gap is 40 seconds, 120 is generous; if it is 90, raise the threshold instead of narrowing the check.
  3. Alert on the gap between ARC's busy-runner count and the number of pods with a live Runner.Worker. A gap that persists across two scrape intervals is the capacity leak, and no existing ARC metric exposes it.
  4. Check the runner version before you file or escalate. If you are on 2.336.0 or later and the message is "not acquired", you are in #4728 and no amount of pod deletion applies. If the last line is the correlation ID, you are in the acquirejob stall on either 2.336.0 or 2.337.0.
  5. Delete the watchdog when the job-acquisition path grows its own deadline. The tripwire is a release note or commit adding a CancellationTokenSource around GetJobMessageAsync, matching the five-second bound already present on AcknowledgeRunnerRequestAsync. Until that lands, every ephemeral runner fleet is one dropped connection record away from silently shrinking.