139 lines
6.9 KiB
Markdown
139 lines
6.9 KiB
Markdown
# P0: Critical Hardening & Stability Fixes
|
|
|
|
**Priority**: P0 — Ship-blocking
|
|
**Theme**: Eliminate crashes, goroutine leaks, and silent misconfigurations that make the service unreliable as a long-running LAN cache appliance.
|
|
**Status**: Not started
|
|
**Target**: All items resolved before next production deployment or public release.
|
|
|
|
## Goal
|
|
|
|
Make `steamcache2` safe to run continuously with real traffic under normal and adverse conditions (bad config, unreachable upstream, frequent restarts, high load).
|
|
|
|
## Overview
|
|
|
|
The current implementation has several classes of defects that can cause:
|
|
|
|
- Immediate panics on startup or misconfiguration
|
|
- Deadlocks / hangs on graceful shutdown
|
|
- Silent failure to enforce documented configuration constraints
|
|
- Loss of error visibility in metrics and logs
|
|
|
|
These must be fixed before the project can be considered production-ready.
|
|
|
|
## Tasks
|
|
|
|
### P0-01: Make `New()` return an error instead of panicking on invalid sizes
|
|
|
|
- **Description**: `units.FromHumanSize()` failures in `New()` currently call `panic()`. This is hostile to callers (tests, embedding, future CLI refactoring) and prevents clean error handling.
|
|
- **Impact**: Any invalid memory/disk size string (including from generated default config in edge cases) crashes the entire process with a stack trace instead of a helpful message.
|
|
- **Affected Files**:
|
|
- `steamcache/steamcache.go` (New function, lines ~854-863)
|
|
- Call sites in `cmd/root.go`
|
|
- All tests that construct `SteamCache`
|
|
- **Approach**:
|
|
1. Change signature to `func New(...) (*SteamCache, error)`
|
|
2. Return wrapped error for parse failures.
|
|
3. Update `NewWithOptions` accordingly.
|
|
4. Update all internal construction paths and tests (use `t.Cleanup` + proper error checks).
|
|
- **Acceptance Criteria**:
|
|
- No more panics from `New()` for bad size strings.
|
|
- Clear error messages: `"invalid memory size: ..."`
|
|
- All existing tests still pass (updated for new signature).
|
|
- `go test -race -short ./...` is green.
|
|
- **Dependencies**: None
|
|
- **Effort**: Small (1-2 hours)
|
|
|
|
### P0-02: Fix upstream connectivity check nil pointer dereference
|
|
|
|
- **Description**: In `Run()`, the code does:
|
|
```go
|
|
resp, err := sc.client.Get(sc.upstream)
|
|
if err != nil || resp.StatusCode != http.StatusOK {
|
|
... use resp.StatusCode when err != nil ...
|
|
```
|
|
When the GET fails, `resp` is typically `nil`.
|
|
- **Impact**: Service crashes with panic on startup whenever the configured `upstream` is unreachable or returns non-200 (very common on first setup or network hiccup).
|
|
- **Affected Files**:
|
|
- `steamcache/steamcache.go` (Run method, ~1025-1032)
|
|
- **Approach**:
|
|
1. Reorder the check: handle `err != nil` first.
|
|
2. Only inspect `resp.StatusCode` when `resp != nil`.
|
|
3. Always close `resp.Body` when `resp != nil`.
|
|
4. Improve the error log message.
|
|
5. Consider making the upstream check optional or retrying (but keep current behavior of exiting for now).
|
|
- **Acceptance Criteria**:
|
|
- Starting with an unreachable upstream produces a clean error log + `os.Exit(1)` instead of a panic.
|
|
- Starting with a reachable but non-200 upstream behaves cleanly.
|
|
- No resource leaks in the error paths.
|
|
- **Dependencies**: None
|
|
- **Effort**: Trivial (< 30 min)
|
|
|
|
### P0-03: Call `config.Validate()` on startup and fail fast with actionable messages
|
|
|
|
- **Description**: A complete `Validate()` method exists in `config/config.go` but is **never invoked**.
|
|
- **Impact**:
|
|
- Users can start the service with invalid GC algorithm names, negative concurrency limits, or disk cache enabled with no path.
|
|
- The service runs in a broken or surprising state instead of failing early with a clear message.
|
|
- **Affected Files**:
|
|
- `config/config.go`
|
|
- `cmd/root.go` (after `LoadConfig`)
|
|
- **Approach**:
|
|
1. Call `cfg.Validate()` immediately after loading (and after applying CLI overrides).
|
|
2. On error, log the problem at ERROR level with the exact field and suggestion.
|
|
3. Exit with code 1 and print a user-friendly message to stderr.
|
|
4. Add unit tests for `Validate()` covering all error cases (currently untested).
|
|
- **Acceptance Criteria**:
|
|
- Invalid `gc_algorithm`, negative limits, or missing disk path cause immediate clean failure.
|
|
- Error messages are specific and actionable.
|
|
- `Validate()` has ≥90% statement coverage in its own test file.
|
|
- **Dependencies**: None
|
|
- **Effort**: Small (1-2 hours including tests)
|
|
|
|
### P0-04: Wire up error metrics and ensure all error paths are instrumented
|
|
|
|
- **Description**: `metrics.IncrementErrors()` exists and `Stats.Errors` is exposed, but the method is **never called** anywhere in the codebase. Many 5xx paths also fail to increment other relevant counters.
|
|
- **Impact**: Operators have no visibility into error rates via `/metrics`. The "errors" field is always zero.
|
|
- **Affected Files**:
|
|
- `steamcache/metrics/metrics.go`
|
|
- `steamcache/steamcache.go` (ServeHTTP and related methods — many locations)
|
|
- Possibly `vfs/*` error paths that bubble up
|
|
- **Approach**:
|
|
1. Audit every place that returns 5xx or logs an error in the request path.
|
|
2. Call `sc.metrics.IncrementErrors()` (and any other appropriate counters) in those paths.
|
|
3. Ensure coalesced request error paths also record errors.
|
|
4. Add a simple test that exercises error paths and asserts metric values.
|
|
- **Acceptance Criteria**:
|
|
- `/metrics` reports non-zero `errors` under induced failure conditions.
|
|
- All current 5xx response paths increment the counter exactly once per failed request.
|
|
- No double-counting on coalesced failures.
|
|
- **Dependencies**: P0-02 (partially)
|
|
- **Effort**: Medium (2-4 hours)
|
|
|
|
## Definition of Done (for the whole P0 milestone)
|
|
|
|
- [ ] All four tasks above completed and merged.
|
|
- [ ] `go test -race -shuffle=on -timeout=5m ./...` passes cleanly.
|
|
- [ ] Manual verification:
|
|
- Start with bad memory size → clean error, no panic.
|
|
- Start with unreachable upstream → clean error + exit 1, no panic.
|
|
- Start with invalid `gc_algorithm` → fails fast with clear message.
|
|
- Induce upstream 500s and connection errors → `/metrics` shows increasing errors count.
|
|
- [ ] Shutdown no longer hangs due to the client limiter goroutine (see P0-05 below if split out).
|
|
- [ ] Updated README or a new `docs/OPERATIONS.md` section documents the new strict startup validation behavior.
|
|
|
|
## Notes for Implementers
|
|
|
|
- These fixes are intentionally small and localized so they can be landed quickly.
|
|
- Prefer adding new tests over modifying large amounts of existing test code.
|
|
- Keep backward compatibility for the public `New` constructor as much as possible (or provide a clear migration path in comments).
|
|
|
|
## References
|
|
|
|
- Full code review (see conversation history or `plans/` directory).
|
|
- Original locations identified in `steamcache/steamcache.go`, `config/config.go`, `cmd/root.go`.
|
|
- Related goroutine leak in `cleanupOldClientLimiters` (may be promoted to its own P0 item if it blocks shutdown testing).
|
|
|
|
---
|
|
|
|
**Next actions after P0**: Move on to P1 items once the service can start and stop reliably without crashing.
|