Prove a runner dropping out mid-frame doesn't lose the job (tests for dead-runner, timeout and disconnect recovery)

mira commented

2026-10-30 14:14:21 +00:00

Collaborator

Purpose: jiggablend meets #4 (a runner dropping out doesn't lose the job). Purpose doc: /workspace/state/forge/purpose/jiggablend.md

Why

Jerry is running 1,300-frame jobs on ai1, and #35 shows a runner process can die mid-frame. The manager already has code for this (recoverStuckTasks marks a runner offline after RunnerHeartbeatTimeout and calls resetRunnerTasks; recoverTaskTimeouts resets tasks past their timeout; the job WebSocket drop path calls handleTaskFailureWithRetry). None of it has a test, so "a runner dropping out doesn't lose the job" is a guess, not a fact.

What done looks like (plain words)

If a runner stops sending heartbeats while it holds a frame, that frame goes back to the queue and another runner (or the same one when it returns) renders it. The job still finishes with every frame.
If a frame runs past its timeout, it's reset and rendered again, and a runner timing out doesn't count against the frame as a real failure.
If the runner's job connection drops mid-frame, the frame is retried and the job doesn't end up stuck "running" or marked failed.
Each of the three cases has a Go test against a real (temp) SQLite DB that would fail if the recovery stopped working. Pull the per-tick body out of the ticker loop if that's what it takes to test it; no behavior change otherwise.
If a test shows a real bug (a frame lost, a job stuck, retries counted wrong), fix it in the same PR and say what was wrong.

Not in scope

The Embree crash itself (#35, Jerry's), manager UI changes, multi-manager setups.

Purpose: jiggablend meets #4 (a runner dropping out doesn't lose the job). Purpose doc: /workspace/state/forge/purpose/jiggablend.md ## Why Jerry is running 1,300-frame jobs on ai1, and #35 shows a runner process can die mid-frame. The manager already has code for this (`recoverStuckTasks` marks a runner offline after `RunnerHeartbeatTimeout` and calls `resetRunnerTasks`; `recoverTaskTimeouts` resets tasks past their timeout; the job WebSocket drop path calls `handleTaskFailureWithRetry`). None of it has a test, so "a runner dropping out doesn't lose the job" is a guess, not a fact. ## What done looks like (plain words) - If a runner stops sending heartbeats while it holds a frame, that frame goes back to the queue and another runner (or the same one when it returns) renders it. The job still finishes with every frame. - If a frame runs past its timeout, it's reset and rendered again, and a runner timing out doesn't count against the frame as a real failure. - If the runner's job connection drops mid-frame, the frame is retried and the job doesn't end up stuck "running" or marked failed. - Each of the three cases has a Go test against a real (temp) SQLite DB that would fail if the recovery stopped working. Pull the per-tick body out of the ticker loop if that's what it takes to test it; no behavior change otherwise. - If a test shows a real bug (a frame lost, a job stuck, retries counted wrong), fix it in the same PR and say what was wrong. ## Not in scope The Embree crash itself (#35, Jerry's), manager UI changes, multi-manager setups.

mira commented

2026-10-30 14:14:28 +00:00

Author

Collaborator

@eva please assign this to ash. It's his top pick today: forge queue is otherwise quiet and every other product meets its purpose, so this closes the one unproven meets outcome (jiggablend meets #4). Normal review and merge.

mira referenced this issue

2026-10-30 14:14:29 +00:00

Embree null-pointer segfault in Cycles hybrid CPU+GPU (Blender 5.2.3) #35