ops: Signal disk-full and eviction capacity pressure #46

2026-09-07T14:24:57-05:00

ash commented

2026-09-07 14:24:57 -05:00

Why

When the disk tier is full or thrashing eviction, operators see what look like random misses with no clear signal that the cache is dropping data. They need to distinguish a cold cache from capacity pressure.

What

Add /metrics counter capacity_pressure_events (keep existing evictions as-is)
Increment when soft eviction frees space under the memory or disk cap, and when disk Create/Write/Mkdir hits ENOSPC (or Windows disk-full)
Log tier (memory or disk) and reason (eviction or enospc); Info on eviction, Warn on ENOSPC
Portable ENOSPC detection (unix.ENOSPC / errors.Is on Unix; ERROR_DISK_FULL / ERROR_HANDLE_DISK_FULL on Windows)
Document next to the #33 disk_tier_ready notes in README Quick check and Large Cache Initialization

Does not add occupancy gauges (#41).

How tested

/usr/local/bin/go test -count=1 for ./steamcache/metrics ./vfs/memory ./vfs/disk ./steamcache
Metric increments on forced eviction pressure (memory + disk)
Simulated ENOSPC via unix.ENOSPC and wrapped os.PathError (no flaky full-disk filesystem tricks)
GOOS=windows GOARCH=amd64 go test -c ./vfs/disk compiles

Risks

Info logs fire on every eviction batch that frees bytes; at-cap caches will be chatty (that is the intended ops signal)
ENOSPC matching is errno-based; unusual wrapped errors might miss the counter (the write still fails)

Rollback

Revert this PR on develop.

Closes #36

## Why When the disk tier is full or thrashing eviction, operators see what look like random misses with no clear signal that the cache is dropping data. They need to distinguish a cold cache from capacity pressure. ## What - Add `/metrics` counter `capacity_pressure_events` (keep existing `evictions` as-is) - Increment when soft eviction frees space under the memory or disk cap, and when disk Create/Write/Mkdir hits ENOSPC (or Windows disk-full) - Log `tier` (`memory` or `disk`) and `reason` (`eviction` or `enospc`); Info on eviction, Warn on ENOSPC - Portable ENOSPC detection (`unix.ENOSPC` / `errors.Is` on Unix; `ERROR_DISK_FULL` / `ERROR_HANDLE_DISK_FULL` on Windows) - Document next to the `#33` `disk_tier_ready` notes in README Quick check and Large Cache Initialization Does not add occupancy gauges (#41). ## How tested - `/usr/local/bin/go test -count=1` for `./steamcache/metrics ./vfs/memory ./vfs/disk ./steamcache` - Metric increments on forced eviction pressure (memory + disk) - Simulated ENOSPC via `unix.ENOSPC` and wrapped `os.PathError` (no flaky full-disk filesystem tricks) - `GOOS=windows GOARCH=amd64 go test -c ./vfs/disk` compiles ## Risks - Info logs fire on every eviction batch that frees bytes; at-cap caches will be chatty (that is the intended ops signal) - ENOSPC matching is errno-based; unusual wrapped errors might miss the counter (the write still fails) ## Rollback Revert this PR on `develop`. Closes #36

ash requested review from linus 2026-09-07 14:24:57 -05:00

linus commented

2026-09-07 15:36:37 -05:00

Hold — do not merge. CI red on head d7af699.

check-and-test failed: TestDiskTierSignalMixedPendingReady (steamcache_test.go:1159) — heartbeat X-SteamCache-Disk-Tier="ready", want pending. Vulncheck green; build/vet/lint green. Your capacity-pressure diff does not touch that path; this looks like the #45 pending-window race biting under -race -shuffle.

Also: head parent is 8cebc1f (#45). develop tip is 8eb3114 (#48). Rebase onto current develop, get CI green (re-run or harden that pending/ready race if it keeps flaking), then ping for merge.

Scope/#36 look fine on a skim once green: capacity_pressure_events, keep evictions, portable ENOSPC, README next to disk_tier_ready. Not a Blake gate.

Hold — do not merge. CI red on head `d7af699`. `check-and-test` failed: `TestDiskTierSignalMixedPendingReady` (`steamcache_test.go:1159`) — heartbeat `X-SteamCache-Disk-Tier="ready"`, want `pending`. Vulncheck green; build/vet/lint green. Your capacity-pressure diff does not touch that path; this looks like the #45 pending-window race biting under `-race -shuffle`. Also: head parent is `8cebc1f` (#45). `develop` tip is `8eb3114` (#48). Rebase onto current `develop`, get CI green (re-run or harden that pending/ready race if it keeps flaking), then ping for merge. Scope/#36 look fine on a skim once green: `capacity_pressure_events`, keep `evictions`, portable ENOSPC, README next to `disk_tier_ready`. Not a Blake gate.

ash was assigned by linus

2026-09-07 15:36:47 -05:00

ash force-pushed ops-disk-full-eviction-signal from d7af699e84 to 5e22f1054b

2026-09-08 10:17:02 -05:00

Compare

eva commented

2026-09-08 13:14:12 -05:00

main tip moved under this head after #50 landed (a0e929c5). Compare shows this branch is behind develop by those 2 commits, so mergeable is false.

Default-branch CI run 1541 is red on TestDiskOnlyDelayedAttach (steamcache_test.go:1076 — during pending attach, DiskTierReady=1, want 0). This PR head already carries the DiskOnly init-hold fix and was green at run 1539 — landing it after rebase should unred main.

Please rebase ops-disk-full-eviction-signal onto current develop, push, and leave merge to Linus when CI is green again. Holding 1.0.26 until main is green.

main tip moved under this head after #50 landed (a0e929c5). Compare shows this branch is behind develop by those 2 commits, so mergeable is false. Default-branch CI run 1541 is red on `TestDiskOnlyDelayedAttach` (`steamcache_test.go:1076` — during pending attach, DiskTierReady=1, want 0). This PR head already carries the DiskOnly init-hold fix and was green at run 1539 — landing it after rebase should unred main. Please rebase `ops-disk-full-eviction-signal` onto current `develop`, push, and leave merge to Linus when CI is green again. Holding `1.0.26` until main is green.

ash added 3 commits 2026-09-08 13:17:05 -05:00

ops: Signal disk-full and eviction capacity pressure de43a71929

When the disk (or memory) tier is at cap or the volume returns ENOSPC,
ops currently look like random misses with no clear "we are dropping
data." Count those events as capacity_pressure_events on /metrics and
log tier plus reason so operators can tell capacity pressure from a
cold cache, without changing the existing evictions counter.

Link: #36

vfs/disk: Harden pending-window test against empty-dir race fe04fb9fbb

Empty-dir disk init closes initDone almost immediately, so
TestDiskTierSignalMixedPendingReady can observe DiskTierReady=0 then
a heartbeat that already saw ready. Register a per-root init hold so
the test can keep the pending window open without slowing production.

Link: #36

test: Hold disk init in DiskOnlyDelayedAttach too

CI / vulncheck (pull_request) Successful in 15s

Details

CI / check-and-test (pull_request) Successful in 42s

Details

e3b2b8de1e

Empty-dir attach can finish before the pending DiskTierReady
assertion under CI load, same race MixedPendingReady already
fixed with RegisterInitHold. Drop t.Parallel and hold the
barrier through the pending checks so disk-only stays green.

Link: #36

ash force-pushed ops-disk-full-eviction-signal from 7d8dd6ea6f to e3b2b8de1e

2026-09-08 13:17:05 -05:00

Compare

linus merged commit 0db9943436 into develop

2026-09-08 13:20:23 -05:00

linus deleted branch ops-disk-full-eviction-signal

2026-09-08 13:20:23 -05:00

pike referenced this pull request

2026-09-09 11:06:12 -05:00

ops: Disk tier occupancy on /metrics #53

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: s1d3sw1ped/steamcache2#46