s1d3sw1ped/steamcache2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Enhance Makefile and README for improved validation and cache management

Browse Source 
- Updated the Makefile to include a new `clean-disk` target for removing disk cache, and modified the `run-validation` target to clean the disk cache before starting.
- Enhanced the `check-review-labels` target to include additional file types in the search for temporary review labels, improving code hygiene checks.
- Refined the README.md to clarify the hardening section and improve the description of the `prefill` command.
- Removed the obsolete `test_cache/.gitkeep` file to clean up the repository.
This commit is contained in:
Justin Harms
2026-05-28 20:26:23 -05:00
parent b7e3a0da86
commit 099e5347d5
5 changed files with 34 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/test-pr.yaml
+1 -1
View File
@@ -21,4 +21,4 @@ jobs:
- run: go install golang.org/x/vuln/cmd/govulncheck@latest
- run: govulncheck ./...
- run: go test -race -v -shuffle=on -coverprofile=coverage.out -timeout=5m ./...
- run: go tool cover -func=coverage.out | tail -10 # basic coverage report (P2-04)
- run: go tool cover -func=coverage.out | tail -10 # basic coverage report
Makefile
+13 -7
View File
@@ -18,7 +18,7 @@ lint: deps check-review-labels ## Run golangci-lint + review label hygiene check
@golangci-lint run ./...
check-review-labels: ## Fail if temporary review labels (P0-01, T1, I3, R2, etc.) are found in source
@! grep -rnE '\b[A-Z][0-9][^a-zA-Z]' --include='*.go' . 2>/dev/null | grep -v 'G[0-9]\{3\}' || (echo "Error: Found temporary review labels (P*, T*, I*, etc.) in source. See AGENTS.md for the rule." && exit 1)
@! grep -rnE '\b[A-Z][0-9][^a-zA-Z]' --include='*.go' --include='*.md' --include='*.yaml' --include='*.sh' --exclude='AGENTS.md' . 2>/dev/null | grep -v 'G[0-9]\{3\}' || (echo "Error: Found temporary review labels (P*, T*, I*, etc.) in source. See AGENTS.md for the rule." && exit 1)
deps: ## Download dependencies
@go mod tidy
@@ -26,6 +26,9 @@ deps: ## Download dependencies
clean: ## Remove build artifacts and test cache
@rm -rf bin/ dist/ *.test coverage.out steamcache2
clean-disk: ## Remove disk cache
@rm -rf validate-disk/
bench: deps ## Run all benchmarks (MemoryFS + DiskFS variants, including all eviction strategies)
@echo "Running MemoryFS benchmarks..."
@go test -bench=. -benchmem -run=^$ -benchtime=1s ./vfs/memory
@@ -38,7 +41,7 @@ setcap: build ## Explicitly set cap_net_bind_service on the (just-built) binary
@sudo setcap 'cap_net_bind_service=+ep' dist/default_linux_amd64_v1/steamcache2
@echo "Done. You should now be able to run 'make run-validation' as your normal user (no root)."
validate run-validation: build ## Start steamcache2 on :80 with small test caches (foreground)
validate run-validation: build clean-disk ## Start steamcache2 on :80 with small test caches (foreground)
@echo "=== Starting steamcache2 in validation mode ==="
@echo "Port 80 + small memory/disk caches (for exercising disk tier, GC, etc.)"
@echo "Press Ctrl-C to stop the server."
@@ -67,9 +70,6 @@ validate-check: ## Quick post-benchmark sanity check against a running steamcach
echo ""; \
echo "Tip: also inspect recent server logs for errors, coalesced hits, and disk activity."
prefill: ## Download latest SteamPrefill into bin/steam-prefill/SteamPrefill (gitignored)
@./scripts/download-prefill.sh
validate-kill: ## Kill leftover steamcache2 processes (safer, checks process name)
@echo "Looking for steamcache2 processes on common validation ports (80 is primary)..."
@for port in 80 8040 8080; do \
@@ -95,6 +95,11 @@ validate-kill: ## Kill leftover steamcache2 processes (safer, checks process nam
done
@echo "Validation server cleanup complete."
prefill: ## Download latest SteamPrefill into bin/steam-prefill/SteamPrefill (gitignored)
@./scripts/download-prefill.sh
help: ## Show this help message
@@ -109,9 +114,10 @@ help: ## Show this help message
@echo " check-review-labels Fail on temporary review labels (P*, T*, I*, R*, etc.)"
@echo " deps Download dependencies"
@echo " clean Remove build/test artifacts"
@echo " clean-disk Remove disk cache"
@echo " bench Run low-level VFS microbenchmarks"
@echo " validate / run-validation Start server on :80 (builds, auto-setcaps fresh binary, then runs as normal user)"
@echo " validate / run-validation Start server on :80 (builds, auto-setcaps fresh binary, then runs as normal user, cleans disk cache first)"
@echo " setcap Explicitly set cap on current build (for port 80 use outside validate)"
@echo " validate-check Quick /metrics report after running a workload"
@echo " validate-kill Kill leftover steamcache2 processes (safer)"
@echo " prefill Download latest SteamPrefill into bin/steam-prefill/ (for use with run-validation)"
@echo " prefill Download latest SteamPrefill into bin/steam-prefill/SteamPrefill (gitignored)"
README.md
+7 -7
View File
@@ -194,7 +194,7 @@ SteamCache2 uses a YAML configuration file (`config.yaml`) for all settings. Her
# Server configuration
listen_address: :80
# P1 hardening (see Security Hardening section)
# Hardening (see Security Hardening section)
max_object_size: "0" # 0=unlimited; set e.g. "256MB" for response size DoS protection
trusted_proxies: [] # empty = safe (ignore XFF for rate limit); set CIDRs for trusted proxies
@@ -222,7 +222,7 @@ upstream: "https://steam.cdn.com"
```
#### Startup Validation
As of P0, `steamcache2` performs strict validation on startup (after loading config + CLI overrides, before creating the cache). Invalid configs cause immediate clean failure (no default written, no panic):
`steamcache2` performs strict validation on startup (after loading config + CLI overrides, before creating the cache). Invalid configs cause immediate clean failure (no default written, no panic):
- Negative `max_concurrent_requests` / `max_requests_per_client`: "negative concurrency not allowed"
- Invalid `gc_algorithm` (memory): "invalid memory gc algorithm: badvalue"
@@ -236,12 +236,12 @@ Error: Invalid configuration: invalid memory gc algorithm: foo. Please fix the c
See `config.Validate()` and `steamcache.New` error paths. This ensures the LAN appliance fails fast on misconfig.
#### Security Hardening (P1)
- `max_object_size` (default "0" = unlimited): set e.g. "256MB" or "512MB" to reject oversized upstream responses with HTTP 413 before buffering/ReadAll. Prevents OOM DoS from large or malicious responses (P1-01). Large legitimate Steam files still served if under limit.
- `trusted_proxies`: CIDR list (default empty). When empty (safe default), X-Forwarded-For and client IP spoofing are ignored for rate limiting — always uses `r.RemoteAddr` only. When set (e.g. your reverse proxy CIDR), uses correct "rightmost untrusted" extraction. Prevents bypass of `max_requests_per_client` (P1-02). Documented for LAN proxy setups only.
- These + P0 validation make steamcache2 safe-by-default for LAN exposure.
#### Security Hardening
- `max_object_size` (default "0" = unlimited): set e.g. "256MB" or "512MB" to reject oversized upstream responses with HTTP 413 before buffering/ReadAll. Prevents OOM DoS from large or malicious responses. Large legitimate Steam files still served if under limit.
- `trusted_proxies`: CIDR list (default empty). When empty (safe default), X-Forwarded-For and client IP spoofing are ignored for rate limiting — always uses `r.RemoteAddr` only. When set (e.g. your reverse proxy CIDR), uses correct "rightmost untrusted" extraction. Prevents bypass of `max_requests_per_client`. Documented for LAN proxy setups only.
- These + the startup validation make steamcache2 safe-by-default for LAN exposure.
#### Migration / Breaking Changes (P1)
#### Migration / Breaking Changes
- `New()` public signature gained 2 required trailing params (`maxObjectSize`, `trustedProxies`). Direct callers (rare; most use config or NewWithOptions) must update.
- Recommended: migrate to `NewWithOptions(Options{...})` (non-breaking) or rely on YAML config + cmd/root.go.
- No behavior change for existing configs (defaults preserve prior semantics).
steamcache/metrics/metrics.go
+13 -16
View File
@@ -24,8 +24,6 @@ type Metrics struct {
TotalBytesServed int64
TotalBytesSaved int64 // bytes served from cache instead of being re-downloaded from upstream
// Cache metrics
MemoryCacheSize int64
DiskCacheSize int64
@@ -194,8 +192,8 @@ func (m *Metrics) GetStats() *Stats {
RateLimited: atomic.LoadInt64(&m.RateLimited),
HitRate: hitRate,
AvgResponseTime: avgResponseTime,
TotalBytesServed: atomic.LoadInt64(&m.TotalBytesServed),
TotalBytesSaved: atomic.LoadInt64(&m.TotalBytesSaved),
TotalBytesServed: atomic.LoadInt64(&m.TotalBytesServed),
TotalBytesSaved: atomic.LoadInt64(&m.TotalBytesSaved),
MemoryCacheSize: atomic.LoadInt64(&m.MemoryCacheSize),
DiskCacheSize: atomic.LoadInt64(&m.DiskCacheSize),
MemoryCacheHits: atomic.LoadInt64(&m.MemoryCacheHits),
@@ -242,18 +240,17 @@ func (m *Metrics) Reset() {
// Stats represents a snapshot of metrics
type Stats struct {
TotalRequests int64
CacheHits int64
CacheMisses int64
CacheCoalesced int64
Errors int64
RateLimited int64
HitRate float64
AvgResponseTime time.Duration
TotalBytesServed int64
TotalBytesSaved int64
MemoryCacheSize int64
TotalRequests int64
CacheHits int64
CacheMisses int64
CacheCoalesced int64
Errors int64
RateLimited int64
HitRate float64
AvgResponseTime time.Duration
TotalBytesServed int64
TotalBytesSaved int64
MemoryCacheSize int64
DiskCacheSize int64
MemoryCacheHits int64
steamcache/test_cache/.gitkeep
View File