runner: re-registration (200 reused) treated as failure; runner can't restart

jerry commented

2026-10-24 17:21:37 +00:00

Contributor

Bug: runner treats re-registration (HTTP 200, `reused:true`) as failure → can never restart

Impact (prod, 2026-10-24): job 1 (KIRA v3, 1308 frames) stalled ~15.5 h at 236/1308. ai1 runner crash-looped 200+ times.

Root cause (all tags 0.0.10–0.0.13, not a version mismatch)

Manager internal/manager/runners.go ~L205–223: when a runner registers with a fingerprint already in runners (same API key), it updates the row and replies 200 OK {"id":1,"reused":true,"status":"online"}.
Runner internal/runner/api/manager.go L158: if resp.StatusCode != http.StatusCreated → error registration failed (status 200). After 10 tries it exits 1; systemd restarts; repeat forever.
So the first registration of a host works (201) and any later runner restart fails permanently.

Trigger

apt-daily-upgrade (unattended-upgrades) on ai1 at 01:51:09 CT restarted services (networkd, udevd, lemond, jiggablend-runner) at 01:51:42 CT → SIGTERM mid frame 525 (task 237).

Evidence

06:51:42Z systemd: Stopping jiggablend-runner.service ...   (apt-daily-upgrade restart wave)
06:51:42Z [WARN] Registration failed (attempt 1/10): registration failed (status 200): {"hostname":"ai1","id":1,"name":"ai1-0447e097","reused":true,"status":"online"}
06:54:13Z [ERROR] Failed to register runner after 10 attempts ... status=1/FAILURE

(Jerry's box: /workspace/state/jiggablend-stats/raw/ai1-runner-journal-20261024.log.gz)

Hotfix running on ai1 now

0.0.12-jerry1 = tag 0.0.12 + one-line change at manager.go:158:

if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK {

Registered first try 12:17 CT, got task 237 (frame 525), rendering on GPU. Old binary kept at ai1:/usr/local/bin/jiggablend.0.0.10.

Ask

Proper fix + regression test (runner accepts 200/reused), release, then ai1 moves back to an official build.
Consider: should the reused path return 201 for compat, and should a runner restart requeue its in-flight task immediately?

## Bug: runner treats re-registration (HTTP 200, `reused:true`) as failure → can never restart **Impact (prod, 2026-10-24):** job 1 (KIRA v3, 1308 frames) stalled ~15.5 h at 236/1308. ai1 runner crash-looped 200+ times. ### Root cause (all tags 0.0.10–0.0.13, not a version mismatch) - Manager `internal/manager/runners.go` ~L205–223: when a runner registers with a fingerprint already in `runners` (same API key), it updates the row and replies **`200 OK`** `{"id":1,"reused":true,"status":"online"}`. - Runner `internal/runner/api/manager.go` L158: `if resp.StatusCode != http.StatusCreated` → error `registration failed (status 200)`. After 10 tries it exits 1; systemd restarts; repeat forever. - So the first registration of a host works (201) and **any later runner restart fails permanently**. ### Trigger `apt-daily-upgrade` (unattended-upgrades) on ai1 at 01:51:09 CT restarted services (networkd, udevd, lemond, jiggablend-runner) at 01:51:42 CT → SIGTERM mid frame 525 (task 237). ### Evidence ``` 06:51:42Z systemd: Stopping jiggablend-runner.service ... (apt-daily-upgrade restart wave) 06:51:42Z [WARN] Registration failed (attempt 1/10): registration failed (status 200): {"hostname":"ai1","id":1,"name":"ai1-0447e097","reused":true,"status":"online"} 06:54:13Z [ERROR] Failed to register runner after 10 attempts ... status=1/FAILURE ``` (Jerry's box: /workspace/state/jiggablend-stats/raw/ai1-runner-journal-20261024.log.gz) ### Hotfix running on ai1 now `0.0.12-jerry1` = tag 0.0.12 + one-line change at manager.go:158: ```go if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK { ``` Registered first try 12:17 CT, got task 237 (frame 525), rendering on GPU. Old binary kept at ai1:/usr/local/bin/jiggablend.0.0.10. ### Ask - Proper fix + regression test (runner accepts 200/`reused`), release, then ai1 moves back to an official build. - Consider: should the reused path return 201 for compat, and should a runner restart requeue its in-flight task immediately?

ash was assigned by eva

2026-10-24 17:21:54 +00:00

eva commented

2026-10-24 17:21:55 +00:00

Owner

Ash, please take this one: make the runner accept a 200 with reused:true from /register as success, add a regression test for a runner restarting against an existing fingerprint, then PR to develop so a release can follow and ai1 can go back to an official build. Jerry's hotfix line in the issue is the shape of it. On the two open questions, my default is to keep the manager's 200 as is (the runner fix covers it) and to file requeue-on-restart as its own follow-up rather than fold it in here.

ash referenced this issue from a commit

2026-10-24 17:26:48 +00:00

runner: accept 200 reused re-registration from manager (closes #31)

ash referenced this issue from a pull request that will close it,

2026-10-24 17:26:56 +00:00

runner: accept 200 reused re-registration from manager (closes #31) #32