Enhance Makefile and README for improved validation and cache management
- Updated the Makefile to include a new `clean-disk` target for removing disk cache, and modified the `run-validation` target to clean the disk cache before starting. - Enhanced the `check-review-labels` target to include additional file types in the search for temporary review labels, improving code hygiene checks. - Refined the README.md to clarify the hardening section and improve the description of the `prefill` command. - Removed the obsolete `test_cache/.gitkeep` file to clean up the repository.
This commit is contained in:
@@ -21,4 +21,4 @@ jobs:
|
||||
- run: go install golang.org/x/vuln/cmd/govulncheck@latest
|
||||
- run: govulncheck ./...
|
||||
- run: go test -race -v -shuffle=on -coverprofile=coverage.out -timeout=5m ./...
|
||||
- run: go tool cover -func=coverage.out | tail -10 # basic coverage report (P2-04)
|
||||
- run: go tool cover -func=coverage.out | tail -10 # basic coverage report
|
||||
@@ -18,7 +18,7 @@ lint: deps check-review-labels ## Run golangci-lint + review label hygiene check
|
||||
@golangci-lint run ./...
|
||||
|
||||
check-review-labels: ## Fail if temporary review labels (P0-01, T1, I3, R2, etc.) are found in source
|
||||
@! grep -rnE '\b[A-Z][0-9][^a-zA-Z]' --include='*.go' . 2>/dev/null | grep -v 'G[0-9]\{3\}' || (echo "Error: Found temporary review labels (P*, T*, I*, etc.) in source. See AGENTS.md for the rule." && exit 1)
|
||||
@! grep -rnE '\b[A-Z][0-9][^a-zA-Z]' --include='*.go' --include='*.md' --include='*.yaml' --include='*.sh' --exclude='AGENTS.md' . 2>/dev/null | grep -v 'G[0-9]\{3\}' || (echo "Error: Found temporary review labels (P*, T*, I*, etc.) in source. See AGENTS.md for the rule." && exit 1)
|
||||
|
||||
deps: ## Download dependencies
|
||||
@go mod tidy
|
||||
@@ -26,6 +26,9 @@ deps: ## Download dependencies
|
||||
clean: ## Remove build artifacts and test cache
|
||||
@rm -rf bin/ dist/ *.test coverage.out steamcache2
|
||||
|
||||
clean-disk: ## Remove disk cache
|
||||
@rm -rf validate-disk/
|
||||
|
||||
bench: deps ## Run all benchmarks (MemoryFS + DiskFS variants, including all eviction strategies)
|
||||
@echo "Running MemoryFS benchmarks..."
|
||||
@go test -bench=. -benchmem -run=^$ -benchtime=1s ./vfs/memory
|
||||
@@ -38,7 +41,7 @@ setcap: build ## Explicitly set cap_net_bind_service on the (just-built) binary
|
||||
@sudo setcap 'cap_net_bind_service=+ep' dist/default_linux_amd64_v1/steamcache2
|
||||
@echo "Done. You should now be able to run 'make run-validation' as your normal user (no root)."
|
||||
|
||||
validate run-validation: build ## Start steamcache2 on :80 with small test caches (foreground)
|
||||
validate run-validation: build clean-disk ## Start steamcache2 on :80 with small test caches (foreground)
|
||||
@echo "=== Starting steamcache2 in validation mode ==="
|
||||
@echo "Port 80 + small memory/disk caches (for exercising disk tier, GC, etc.)"
|
||||
@echo "Press Ctrl-C to stop the server."
|
||||
@@ -67,9 +70,6 @@ validate-check: ## Quick post-benchmark sanity check against a running steamcach
|
||||
echo ""; \
|
||||
echo "Tip: also inspect recent server logs for errors, coalesced hits, and disk activity."
|
||||
|
||||
prefill: ## Download latest SteamPrefill into bin/steam-prefill/SteamPrefill (gitignored)
|
||||
@./scripts/download-prefill.sh
|
||||
|
||||
validate-kill: ## Kill leftover steamcache2 processes (safer, checks process name)
|
||||
@echo "Looking for steamcache2 processes on common validation ports (80 is primary)..."
|
||||
@for port in 80 8040 8080; do \
|
||||
@@ -95,6 +95,11 @@ validate-kill: ## Kill leftover steamcache2 processes (safer, checks process nam
|
||||
done
|
||||
@echo "Validation server cleanup complete."
|
||||
|
||||
prefill: ## Download latest SteamPrefill into bin/steam-prefill/SteamPrefill (gitignored)
|
||||
@./scripts/download-prefill.sh
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
help: ## Show this help message
|
||||
@@ -109,9 +114,10 @@ help: ## Show this help message
|
||||
@echo " check-review-labels Fail on temporary review labels (P*, T*, I*, R*, etc.)"
|
||||
@echo " deps Download dependencies"
|
||||
@echo " clean Remove build/test artifacts"
|
||||
@echo " clean-disk Remove disk cache"
|
||||
@echo " bench Run low-level VFS microbenchmarks"
|
||||
@echo " validate / run-validation Start server on :80 (builds, auto-setcaps fresh binary, then runs as normal user)"
|
||||
@echo " validate / run-validation Start server on :80 (builds, auto-setcaps fresh binary, then runs as normal user, cleans disk cache first)"
|
||||
@echo " setcap Explicitly set cap on current build (for port 80 use outside validate)"
|
||||
@echo " validate-check Quick /metrics report after running a workload"
|
||||
@echo " validate-kill Kill leftover steamcache2 processes (safer)"
|
||||
@echo " prefill Download latest SteamPrefill into bin/steam-prefill/ (for use with run-validation)"
|
||||
@echo " prefill Download latest SteamPrefill into bin/steam-prefill/SteamPrefill (gitignored)"
|
||||
@@ -194,7 +194,7 @@ SteamCache2 uses a YAML configuration file (`config.yaml`) for all settings. Her
|
||||
# Server configuration
|
||||
listen_address: :80
|
||||
|
||||
# P1 hardening (see Security Hardening section)
|
||||
# Hardening (see Security Hardening section)
|
||||
max_object_size: "0" # 0=unlimited; set e.g. "256MB" for response size DoS protection
|
||||
trusted_proxies: [] # empty = safe (ignore XFF for rate limit); set CIDRs for trusted proxies
|
||||
|
||||
@@ -222,7 +222,7 @@ upstream: "https://steam.cdn.com"
|
||||
```
|
||||
|
||||
#### Startup Validation
|
||||
As of P0, `steamcache2` performs strict validation on startup (after loading config + CLI overrides, before creating the cache). Invalid configs cause immediate clean failure (no default written, no panic):
|
||||
`steamcache2` performs strict validation on startup (after loading config + CLI overrides, before creating the cache). Invalid configs cause immediate clean failure (no default written, no panic):
|
||||
|
||||
- Negative `max_concurrent_requests` / `max_requests_per_client`: "negative concurrency not allowed"
|
||||
- Invalid `gc_algorithm` (memory): "invalid memory gc algorithm: badvalue"
|
||||
@@ -236,12 +236,12 @@ Error: Invalid configuration: invalid memory gc algorithm: foo. Please fix the c
|
||||
|
||||
See `config.Validate()` and `steamcache.New` error paths. This ensures the LAN appliance fails fast on misconfig.
|
||||
|
||||
#### Security Hardening (P1)
|
||||
- `max_object_size` (default "0" = unlimited): set e.g. "256MB" or "512MB" to reject oversized upstream responses with HTTP 413 before buffering/ReadAll. Prevents OOM DoS from large or malicious responses (P1-01). Large legitimate Steam files still served if under limit.
|
||||
- `trusted_proxies`: CIDR list (default empty). When empty (safe default), X-Forwarded-For and client IP spoofing are ignored for rate limiting — always uses `r.RemoteAddr` only. When set (e.g. your reverse proxy CIDR), uses correct "rightmost untrusted" extraction. Prevents bypass of `max_requests_per_client` (P1-02). Documented for LAN proxy setups only.
|
||||
- These + P0 validation make steamcache2 safe-by-default for LAN exposure.
|
||||
#### Security Hardening
|
||||
- `max_object_size` (default "0" = unlimited): set e.g. "256MB" or "512MB" to reject oversized upstream responses with HTTP 413 before buffering/ReadAll. Prevents OOM DoS from large or malicious responses. Large legitimate Steam files still served if under limit.
|
||||
- `trusted_proxies`: CIDR list (default empty). When empty (safe default), X-Forwarded-For and client IP spoofing are ignored for rate limiting — always uses `r.RemoteAddr` only. When set (e.g. your reverse proxy CIDR), uses correct "rightmost untrusted" extraction. Prevents bypass of `max_requests_per_client`. Documented for LAN proxy setups only.
|
||||
- These + the startup validation make steamcache2 safe-by-default for LAN exposure.
|
||||
|
||||
#### Migration / Breaking Changes (P1)
|
||||
#### Migration / Breaking Changes
|
||||
- `New()` public signature gained 2 required trailing params (`maxObjectSize`, `trustedProxies`). Direct callers (rare; most use config or NewWithOptions) must update.
|
||||
- Recommended: migrate to `NewWithOptions(Options{...})` (non-breaking) or rely on YAML config + cmd/root.go.
|
||||
- No behavior change for existing configs (defaults preserve prior semantics).
|
||||
|
||||
@@ -24,8 +24,6 @@ type Metrics struct {
|
||||
TotalBytesServed int64
|
||||
TotalBytesSaved int64 // bytes served from cache instead of being re-downloaded from upstream
|
||||
|
||||
|
||||
|
||||
// Cache metrics
|
||||
MemoryCacheSize int64
|
||||
DiskCacheSize int64
|
||||
@@ -254,7 +252,6 @@ type Stats struct {
|
||||
TotalBytesSaved int64
|
||||
MemoryCacheSize int64
|
||||
|
||||
|
||||
DiskCacheSize int64
|
||||
MemoryCacheHits int64
|
||||
DiskCacheHits int64
|
||||
|
||||
Reference in New Issue
Block a user