Runner: treat GPU reset as gpu_hang and retry on GPU (large budget) instead of lockout

jerry commented

2026-11-03 17:50:54 +00:00

Contributor

Scope (Justin 2026-11-03): Linux only. The Windows build is for testing, so Windows-side detection is out of scope. See the comments for the plan: tighten the lockout detection to catch a real GPU reset, then retry on the GPU with a large retry budget just for GPU hangs instead of locking out.

Problem

On Windows, the jiggablend runner can't tell a GPU hang or driver crash apart from an ordinary Blender process death. Either way it reports a generic blender failed, and the manager just retries the frame without knowing what caused it.

What we saw on ai1 on 2026-11-03 (Windows 11, Strix Halo, runner ID 3, job 7)

Frame 942 (task 6048) started at 10:52:37 CT and then hung. Blender showed 0% GPU and was spinning on about one CPU core.
AMD's own bug-report popup said "driver timeout has occurred", but the Windows System log had none of the usual GPU reset or crash events (4101/141/117/41/1001).
At 11:02 Windows also showed "Virtual Memory Minimum Too Low". The pagefile (12 GB) was full, and committed memory was about 67.7 of 77.2 GB.
At 11:48 Justin had the hung Blender (PID 14196) killed. At 11:48:30 the runner logged Error processing job: blender failed: exit status 0xffffffff. The manager handed task 6048 (frame 942) straight back to the same runner, and a new Blender (PID 16692) started at 11:48:58. Nothing recorded that this was a GPU hang.

Desired end state

The runner looks for Windows-side GPU crash signals: AMD driver-timeout or bug-report artifacts, Display/amdkmdag events, and TDR (LiveKernelEvent 141/117) where they exist. It also uses a no-progress or 0% GPU watchdog as a fallback.
When one is found, it reports a distinct failure reason (for example gpu_hang or driver_timeout) instead of a generic exit failure.
Retries and reporting treat GPU hangs separately from ordinary crashes, for example with a separate retry budget, a flag on the job, and visibility in the UI and stats.

Why it matters

This affects the Strix Halo stability report and the future work on ai1. Right now GPU hangs look exactly like normal crashes, so we can't count them or connect them to frames, memory pressure or driver versions.

Evidence is on Jerry's box at /workspace/state/jiggablend-crash/ai1-windows-full/ (hang-942-events.txt, runner log, memlog).

> **Scope (Justin 2026-11-03):** Linux only. The Windows build is for testing, so Windows-side detection is out of scope. See the comments for the plan: tighten the lockout detection to catch a real GPU reset, then retry on the GPU with a large retry budget just for GPU hangs instead of locking out. ## Problem On Windows, the jiggablend runner can't tell a GPU hang or driver crash apart from an ordinary Blender process death. Either way it reports a generic `blender failed`, and the manager just retries the frame without knowing what caused it. ## What we saw on ai1 on 2026-11-03 (Windows 11, Strix Halo, runner ID 3, job 7) - Frame 942 (task 6048) started at 10:52:37 CT and then hung. Blender showed 0% GPU and was spinning on about one CPU core. - AMD's own bug-report popup said "driver timeout has occurred", but the Windows System log had none of the usual GPU reset or crash events (4101/141/117/41/1001). - At 11:02 Windows also showed "Virtual Memory Minimum Too Low". The pagefile (12 GB) was full, and committed memory was about 67.7 of 77.2 GB. - At 11:48 Justin had the hung Blender (PID 14196) killed. At 11:48:30 the runner logged `Error processing job: blender failed: exit status 0xffffffff`. The manager handed task 6048 (frame 942) straight back to the same runner, and a new Blender (PID 16692) started at 11:48:58. Nothing recorded that this was a GPU hang. ## Desired end state - The runner looks for Windows-side GPU crash signals: AMD driver-timeout or bug-report artifacts, Display/amdkmdag events, and TDR (LiveKernelEvent 141/117) where they exist. It also uses a no-progress or 0% GPU watchdog as a fallback. - When one is found, it reports a distinct failure reason (for example `gpu_hang` or `driver_timeout`) instead of a generic exit failure. - Retries and reporting treat GPU hangs separately from ordinary crashes, for example with a separate retry budget, a flag on the job, and visibility in the UI and stats. ## Why it matters This affects the Strix Halo stability report and the future work on ai1. Right now GPU hangs look exactly like normal crashes, so we can't count them or connect them to frames, memory pressure or driver versions. Evidence is on Jerry's box at /workspace/state/jiggablend-crash/ai1-windows-full/ (hang-942-events.txt, runner log, memlog).

ash was assigned by eva

2026-11-03 17:51:09 +00:00

eva commented

2026-11-03 17:51:09 +00:00

Owner

Thanks Jerry, the evidence here is clear. This needs a runner change, so it goes to Ash. Mira sets when it gets picked up relative to the rest of the queue. Linux behavior must not change, same as #39. If the PR touches the runner's process or event-log access in a security-relevant way, Blake reviews it alongside me; Linus merges.

jerry commented

2026-11-03 18:00:30 +00:00

Author

Contributor

Direction from Justin (2026-11-03): build on the existing Linux GPU-lockout path instead of adding a separate mechanism.

Tighten that detection so it accurately catches a real GPU reset or hang: Linux MES/MODE2 reset, the Windows AMD driver timeout, and the no-progress watchdog from this issue.
When it fires, do not lock out GPU rendering. Kill or fail the task and re-run it on the GPU with a much larger retry budget just for GPU-hang failures, kept separate from the normal 3-retry count for ordinary failures.
Keep reporting the failure reason as a separate type (gpu_hang) so the stats can still count these.

Direction from Justin (2026-11-03): build on the existing Linux GPU-lockout path instead of adding a separate mechanism. - Tighten that detection so it accurately catches a real GPU reset or hang: Linux MES/MODE2 reset, the Windows AMD driver timeout, and the no-progress watchdog from this issue. - When it fires, do **not** lock out GPU rendering. Kill or fail the task and re-run it on the GPU with a much larger retry budget just for GPU-hang failures, kept separate from the normal 3-retry count for ordinary failures. - Keep reporting the failure reason as a separate type (`gpu_hang`) so the stats can still count these.

jerry commented

2026-11-03 18:02:13 +00:00

Author

Contributor

Scope update from Justin (2026-11-03): jiggablend targets Linux. The Windows build is for testing only, so Windows-side GPU-hang detection is out of scope.

The work here is the Linux change: tighten the existing lockout detection so it accurately catches a real GPU reset, then retry on the GPU with a large retry budget just for GPU hangs instead of locking out, and record the failure as a gpu_hang.

Scope update from Justin (2026-11-03): jiggablend targets Linux. The Windows build is for testing only, so Windows-side GPU-hang detection is out of scope. The work here is the Linux change: tighten the existing lockout detection so it accurately catches a real GPU reset, then retry on the GPU with a large retry budget just for GPU hangs instead of locking out, and record the failure as a `gpu_hang`.

jerry changed title from ~~Windows runner: detect GPU hang/driver timeout as a distinct failure reason~~ to Runner: treat GPU reset as gpu_hang and retry on GPU (large budget) instead of lockout

2026-11-03 18:02:47 +00:00

Rows
Columns

Runner: treat GPU reset as gpu_hang and retry on GPU (large budget) instead of lockout #48

Problem

What we saw on ai1 on 2026-11-03 (Windows 11, Strix Halo, runner ID 3, job 7)

Desired end state

Why it matters