Operators could see attach-ready and capacity-pressure events but not how
full the configured disk (or memory) tier was without reading filesystems.
Expose size/capacity gauges and disk_cache_full_ratio next to disk_tier_ready.
Capacity is a config read, so it stays available while attach is pending.
In-flight coalescing already dedups identical misses, but acceptance
still lacked a gated test (without a hold, waiters can become sequential
HITs after the first fill) and the Quick check metrics table omitted
cache_coalesced.
Add steamcache/coalesce_test.go: N concurrent identical GETs share one
upstream fill (leader MISS, waiters HIT-COALESCED) and a 5xx sibling.
Document cache_coalesced next to the other hit/miss fields.
Fixes#35
/metrics was Prometheus-ish name/value text with a custom banner and
text/plain Content-Type, so strict scrapers could fail.
Emit Prometheus text 0.0.4 (# HELP, # TYPE, counter|gauge) with stable
metric names, and set Content-Type to text/plain; version=0.0.4; charset=utf-8.
Empty-dir attach can finish before the pending DiskTierReady
assertion under CI load, same race MixedPendingReady already
fixed with RegisterInitHold. Drop t.Parallel and hold the
barrier through the pending checks so disk-only stays green.
Link: #36
Empty-dir disk init closes initDone almost immediately, so
TestDiskTierSignalMixedPendingReady can observe DiskTierReady=0 then
a heartbeat that already saw ready. Register a per-root init hold so
the test can keep the pending window open without slowing production.
Link: #36
When the disk (or memory) tier is at cap or the volume returns ENOSPC,
ops currently look like random misses with no clear "we are dropping
data." Count those events as capacity_pressure_events on /metrics and
log tier plus reason so operators can tell capacity pressure from a
cold cache, without changing the existing evictions counter.
Link: #36
Large disk caches can look memory-only/broken during async DiskFS attach.
Expose disk_tier_ready in /metrics, pending/ready logs for disk-only and
mixed modes, and X-SteamCache-Disk-Tier on /lancache-heartbeat. GetMetrics
skips blocking disk.Size() while attach is pending so /metrics stays usable.
Closes#33
Steam clients lean on Range requests. Hit/miss looked fine while downloads
still felt cold when a Range miss returned the full upstream body as 200.
Add range_cache / range_upstream metrics. HIT path already sliced 206 from
the cached full object; count range_cache there. On MISS, keep stripping
Range for the upstream fetch (full object still cached) but serve the
client's requested slice as 206 and count range_upstream.
Tests in steamcache/range_test.go cover Range HIT (local, no upstream),
Range MISS (206 + full object cached), and /metrics emission.
When upstream is empty the cache used the client Host as the fetch URL, so any LAN client with a spoofed Steam User-Agent could proxy to literal IPs or arbitrary names. Reject those hosts, stop following upstream redirects, and keep path-only cache keys so real Steam CDNs still share entries.
- Updated the Makefile to include a new `clean-disk` target for removing disk cache, and modified the `run-validation` target to clean the disk cache before starting.
- Enhanced the `check-review-labels` target to include additional file types in the search for temporary review labels, improving code hygiene checks.
- Refined the README.md to clarify the hardening section and improve the description of the `prefill` command.
- Removed the obsolete `test_cache/.gitkeep` file to clean up the repository.
- Added metrics for bytes saved from cache to improve performance insights.
- Updated cache eviction strategies in MemoryFS and DiskFS to include metrics tracking for hits and evictions.
- Enhanced README.md with updated garbage collection algorithm descriptions and recommendations for cache usage.
- Introduced new madviseSequential functionality for improved memory access hints on Unix systems.
- Adjusted validation configuration in examples to better reflect realistic usage scenarios.
- Introduced coalescing logic in `coalescing.go` to handle concurrent identical upstream fetches, including a state machine and response buffering for improved performance.
- Implemented a new cache file format in `format.go`, supporting serialization and deserialization of HTTP responses, along with range request handling.
- Developed an HTTP handler in `handler.go` to manage requests, including special endpoint handling and metrics reporting.
- Added rate limiting functionality in `ratelimit.go` to control per-client and global request rates, enhancing security and performance.
- Created service management capabilities in `service.go` to define and manage cacheable services, including user-agent detection.
- Updated tests in `steamcache_test.go` to cover new functionalities, ensuring robustness and reliability across the codebase.
- Updated .golangci.yml to enable default linters and refine suppression rules, enhancing code quality visibility.
- Improved error handling in cmd/root.go by explicitly discarding low-value error messages during fatal exits for consistency with errcheck posture.
- Added best-effort error handling in various locations across the codebase, ensuring that non-critical errors are logged without affecting overall functionality.
- Introduced a new writeMetricsText function to streamline metrics output, improving code clarity and maintainability.
- Updated `disk.New` to support asynchronous initialization for large caches, improving responsiveness during startup.
- Introduced an eviction function parameter to `disk.New`, ensuring proper handling of over-capacity scenarios.
- Enhanced error handling in various components, including memory and disk tests, to ensure robustness and clarity.
- Refactored tests to validate new behaviors, including checks for delayed attachment and proper error propagation.
- Removed obsolete error handling code and tests related to the now-deleted errors package, streamlining the codebase.
- Implemented a more robust shutdown mechanism using sync.Once to prevent multiple shutdown calls and ensure all background managers are stopped properly.
- Refactored coalesced request handling to utilize atomic operations for waiting counts, improving thread safety and performance.
- Introduced a done channel for coalesced requests to signal completion, enhancing the handling of concurrent requests and reducing potential deadlocks.
- Updated logging to provide better insights into cache request processing and error handling.
- C4: Propagate request context through semaphores and upstream calls
- C5: Make client rate limiter cleanup goroutine idempotent via sync.Once
- P1: Streaming response path using io.Copy instead of full materialization
- P2: Gate promotion goroutines to files >= 64KiB
- P3: Reduce global lock contention during eviction
- P4: Demote hot-path logging and compact metrics output
- R1: Add upstream URL scheme/host validation
- R4: Add cache file format version + tolerant deserialization
- Makefile: Make build-snapshot-single practical by using -short tests
- Config: Add GetDefaultConfig + Validate for better testability and R1 coverage
All changes follow the "smallest safe diff" principle from the review.
Safe test suite now builds and runs cleanly via make build-snapshot-single.
Ref: docs/reviews/steamcache2-production-hardening-review-2026-05-26.md
- Introduced a new error handling system with custom error types for better context and clarity in error reporting.
- Implemented URL validation to prevent invalid requests and enhance security.
- Updated cache key generation functions to return errors, improving robustness in handling invalid inputs.
- Added comprehensive metrics tracking for requests, cache hits, misses, and performance metrics, allowing for better monitoring and analysis of the caching system.
- Enhanced logging to include detailed metrics and error information for improved debugging and operational insights.
- Updated the caching logic to utilize a predictive cache warmer, enhancing content prefetching based on access patterns.
- Replaced the legacy warming system with a more efficient predictive approach, allowing for better performance and resource management.
- Refactored memory management to integrate dynamic cache size adjustments based on system memory usage, improving overall efficiency.
- Simplified the VFS interface and improved concurrency handling with sharded locks for better performance in multi-threaded environments.
- Enhanced tests to validate the new caching and memory management behaviors, ensuring reliability and performance improvements.
- Changed module name from `s1d3sw1ped/SteamCache2` to `s1d3sw1ped/steamcache2` for consistency.
- Updated all import paths and references throughout the codebase to reflect the new module name.
- Adjusted README and Makefile to use the updated module name, ensuring clarity in usage instructions.
- Updated the `downloadThroughCache` function to remove the upstream URL parameter, streamlining the caching process.
- Modified the `serializeRawResponse` function to eliminate unnecessary parameters, enhancing clarity and usability.
- Adjusted integration tests to align with the new function signatures, ensuring consistent testing of caching behavior.
- Updated caching logic to support size-based promotion filtering, ensuring that not all files may be promoted based on size constraints.
- Implemented adaptive caching strategies with a new AdaptiveCacheManager to analyze access patterns and adjust caching strategies dynamically.
- Introduced predictive caching features with a PredictiveCacheManager to prefetch content based on access patterns.
- Added a CacheWarmer to preload popular content into the cache, improving access times for frequently requested files.
- Refactored memory management with a DynamicCacheManager to adjust cache sizes based on system memory usage.
- Enhanced VFS interface and file metadata handling to support new features and improve performance.
- Updated tests to validate new caching behaviors and ensure reliability of the caching system.
- Introduced integration tests for SteamCache to validate caching behavior with real Steam URLs.
- Implemented a ServiceManager to manage service configurations, allowing for dynamic detection of services based on User-Agent.
- Updated cache key generation to include service prefixes, enhancing cache organization and retrieval.
- Enhanced the caching logic to support multiple services, starting with Steam and Epic Games.
- Improved .gitignore to exclude test cache files while retaining necessary structure.
- Introduced maxConcurrentRequests and maxRequestsPerClient fields in the Config struct to manage request limits.
- Updated the SteamCache implementation to utilize these new configuration options for controlling concurrent requests.
- Enhanced the ServeHTTP method to enforce global and per-client rate limiting using semaphores.
- Modified the root command to accept new flags for configuring concurrency limits via command-line arguments.
- Updated tests to reflect changes in the SteamCache initialization and request handling logic.
- Changed expected Content-Type from "application/octet-stream" to "application/x-steam-chunk" to align with Steam's file specifications.
- Enhanced warning message for unexpected content types to provide clearer context for debugging.
- Replaced SHA1 hash calculations with SHA256 for improved security and consistency in cache key generation.
- Introduced a new TestURLHashing function to validate the new cache key generation logic.
- Removed outdated hash calculation tests and streamlined the caching process to focus on URL-based hashing.
- Implemented lightweight validation methods in ServeHTTP to enhance performance and reliability of cached responses.
- Added batched time updates in VFS implementations for better performance during access time tracking.
- Removed the min function and the verifyResponseHash function to streamline the codebase.
- Updated extractHashFromSteamPath to use strings.TrimPrefix for cleaner path handling.
- Retained comments regarding removed Prometheus metrics for future reference.
- Introduced a YAML-based configuration system, allowing for automatic generation of a default `config.yaml` file.
- Updated the application to load configuration settings from the YAML file, improving flexibility and ease of use.
- Added a Makefile to streamline development tasks, including running the application, testing, and managing dependencies.
- Enhanced `.gitignore` to include build artifacts and configuration files.
- Removed unused Prometheus metrics and related code to simplify the codebase.
- Updated dependencies in `go.mod` and `go.sum` for improved functionality and performance.
- Introduced a map for hop-by-hop headers to be removed from responses.
- Enhanced cache serving logic to read and filter HTTP responses, ensuring only relevant headers are forwarded.
- Updated cache writing to handle full HTTP responses, improving cache integrity and performance.
- Added upstream server configuration to launch.json for improved connectivity.
- Increased HTTP client timeout from 60s to 120s for better handling of slow responses.
- Updated server timeouts in steamcache.go: increased ReadTimeout to 30s and WriteTimeout to 60s.
- Introduced ReadHeaderTimeout to mitigate header attacks and set MaxHeaderBytes to 1MB.
- Improved error logging in the Run method to include HTTP status codes for better debugging.
- Adjusted ServeHTTP method to handle root path and metrics endpoint correctly.
- Removed copyright footer from .goreleaser.yaml.
- Increased HTTP client connection settings in steamcache.go for improved performance:
- MaxIdleConns from 100 to 200
- MaxIdleConnsPerHost from 10 to 50
- IdleConnTimeout from 90s to 120s
- TLSHandshakeTimeout from 10s to 15s
- ResponseHeaderTimeout from 10s to 30s
- ExpectContinueTimeout from 1s to 5s
- Added DisableCompression and ForceAttemptHTTP2 options.
- Removed debug logging for manifest files in ServeHTTP method.
- Updated .gitignore to include all .exe files and ensure .smashignore is tracked.
- Expanded README.md with advanced configuration options for garbage collection algorithms, detailing available algorithms and use cases.
- Modified launch.json to include memory and disk garbage collection flags for better configuration.
- Refactored root.go to introduce memoryGC and diskGC flags for garbage collection algorithms.
- Implemented hash extraction and verification in steamcache.go to ensure data integrity during caching.
- Added new tests in steamcache_test.go for hash extraction and verification, ensuring correctness of caching behavior.
- Enhanced garbage collection strategies in gc.go, introducing LFU, FIFO, Largest, Smallest, and Hybrid algorithms with corresponding metrics.
- Updated caching logic to conditionally cache responses based on hash verification results.
- Updated disk_test.go to replace Set and Get with Create and Open methods for better clarity and functionality.
- Modified fileinfo.go to include package comment.
- Refactored gc.go to streamline garbage collection handling and removed unused statistics.
- Updated gc_test.go to comment out large random tests for future implementation.
- Enhanced memory.go to implement LRU caching and metrics for memory usage.
- Updated memory_test.go to replace Set and Get with Create and Open methods.
- Removed sync.go as it was redundant and not utilized.
- Updated vfs.go to reflect changes in the VFS interface, replacing Set and Get with Create and Open.
- Added package comments to vfserror.go for consistency.
- Added Prometheus client library for metrics collection.
- Refactored garbage collection strategy from random deletion to LRU (Least Recently Used) deletion.
- Introduced per-key locking in cache to prevent race conditions.
- Enhanced logging with structured log messages for cache hits and misses.
- Implemented a retry mechanism for upstream requests with exponential backoff.
- Updated Go modules and indirect dependencies for better compatibility and performance.
- Removed unused sync filesystem implementation.
- Added version initialization to ensure a default version string.
pike
ash
moss
linus
s1d3sw1ped_bot
s1d3sw1ped