- Replaced the `validate-with-prefill.sh` script with a streamlined `make validate` command for improved usability.
- Updated `validate-config.yaml` to clarify cache management instructions and garbage collection algorithms.
- Enhanced comments to provide better guidance on upstream configurations and their implications for caching setups.
- Removed outdated quick start section to streamline the validation process.
- Updated the validation server description for better clarity and accessibility.
- Enhanced the explanation of the validation configuration file to emphasize its importance and usage.
- Added `lint` as a prerequisite for the `build`, `test`, and `test-race` targets to ensure code quality checks are performed before executing tests and builds.
- This enhancement promotes better code hygiene and consistency across the development workflow.
- Updated the Makefile to include a new `clean-disk` target for removing disk cache, and modified the `run-validation` target to clean the disk cache before starting.
- Enhanced the `check-review-labels` target to include additional file types in the search for temporary review labels, improving code hygiene checks.
- Refined the README.md to clarify the hardening section and improve the description of the `prefill` command.
- Removed the obsolete `test_cache/.gitkeep` file to clean up the repository.
- Added metrics for bytes saved from cache to improve performance insights.
- Updated cache eviction strategies in MemoryFS and DiskFS to include metrics tracking for hits and evictions.
- Enhanced README.md with updated garbage collection algorithm descriptions and recommendations for cache usage.
- Introduced new madviseSequential functionality for improved memory access hints on Unix systems.
- Adjusted validation configuration in examples to better reflect realistic usage scenarios.
- Added new targets in the Makefile for validation, including `run-validation`, `validate-check`, and `validate-kill`, to streamline the testing process with external tools like SteamPrefill.
- Introduced a `setcap` target to manage necessary capabilities for running the server on port 80 without root access.
- Updated README.md to include detailed instructions for validating functionality, including quick start guides and troubleshooting tips.
- Improved .gitignore to exclude validation artifacts and logs, ensuring a cleaner repository.
- Introduced coalescing logic in `coalescing.go` to handle concurrent identical upstream fetches, including a state machine and response buffering for improved performance.
- Implemented a new cache file format in `format.go`, supporting serialization and deserialization of HTTP responses, along with range request handling.
- Developed an HTTP handler in `handler.go` to manage requests, including special endpoint handling and metrics reporting.
- Added rate limiting functionality in `ratelimit.go` to control per-client and global request rates, enhancing security and performance.
- Created service management capabilities in `service.go` to define and manage cacheable services, including user-agent detection.
- Updated tests in `steamcache_test.go` to cover new functionalities, ensuring robustness and reliability across the codebase.
- Updated .golangci.yml to enable default linters and refine suppression rules, enhancing code quality visibility.
- Improved error handling in cmd/root.go by explicitly discarding low-value error messages during fatal exits for consistency with errcheck posture.
- Added best-effort error handling in various locations across the codebase, ensuring that non-critical errors are logged without affecting overall functionality.
- Introduced a new writeMetricsText function to streamline metrics output, improving code clarity and maintainability.
- Updated `disk.New` to support asynchronous initialization for large caches, improving responsiveness during startup.
- Introduced an eviction function parameter to `disk.New`, ensuring proper handling of over-capacity scenarios.
- Enhanced error handling in various components, including memory and disk tests, to ensure robustness and clarity.
- Refactored tests to validate new behaviors, including checks for delayed attachment and proper error propagation.
- Removed obsolete error handling code and tests related to the now-deleted errors package, streamlining the codebase.
- Added a new section in AGENTS.md outlining the importance of not leaving temporary review labels in source code or comments.
- Updated the error message in the Makefile's check-review-labels target to reference AGENTS.md for review hygiene rules instead of plans/README.md, ensuring consistency in documentation.
- Updated the 'bench' target in the Makefile to run all benchmarks for MemoryFS and DiskFS, improving clarity and coverage.
- Added explicit post-eviction consistency checks in DiskFS tests to ensure on-disk files are removed after eviction.
- Introduced new benchmarks for memory eviction strategies under pressure, enhancing test coverage for memory management.
- Improved error handling in benchmark tests for both disk and memory file systems, ensuring robustness during performance evaluations.
- Refactored key generation in tests for consistency and clarity.
- Implemented a more robust shutdown mechanism using sync.Once to prevent multiple shutdown calls and ensure all background managers are stopped properly.
- Refactored coalesced request handling to utilize atomic operations for waiting counts, improving thread safety and performance.
- Introduced a done channel for coalesced requests to signal completion, enhancing the handling of concurrent requests and reducing potential deadlocks.
- Updated logging to provide better insights into cache request processing and error handling.
- Consolidated build commands by replacing 'build-snapshot-single' with a generic 'build' target.
- Enhanced test commands to include shuffling and a race detector for improved reliability.
- Updated help output to reflect new build and test targets, improving usability for developers.
The implementation work based on the review is complete and the
resulting code changes are already merged. The review document
itself (with all the round-by-round notes) is no longer needed in
the repository.
Include the completed review (with all Status/Response updates and
Completion Note) as the authoritative record of the work done.
This closes out the post-implement merge cleanup.
This test file was added back temporarily during the post-implement
cleanup so that its removal could be recorded explicitly as part of
the production hardening merge.
It was originally scaffolding from the implement session and is no
longer needed (the minimal Validate + GetDefaultConfig support was
added directly to config.go instead).
This file was previously ignored as stray. Adding it now so that
its removal can be part of the upcoming production hardening merge
instead of being a silent untracked file.
It will be deleted in a follow-up commit.
- C4: Propagate request context through semaphores and upstream calls
- C5: Make client rate limiter cleanup goroutine idempotent via sync.Once
- P1: Streaming response path using io.Copy instead of full materialization
- P2: Gate promotion goroutines to files >= 64KiB
- P3: Reduce global lock contention during eviction
- P4: Demote hot-path logging and compact metrics output
- R1: Add upstream URL scheme/host validation
- R4: Add cache file format version + tolerant deserialization
- Makefile: Make build-snapshot-single practical by using -short tests
- Config: Add GetDefaultConfig + Validate for better testability and R1 coverage
All changes follow the "smallest safe diff" principle from the review.
Safe test suite now builds and runs cleanly via make build-snapshot-single.
Ref: docs/reviews/steamcache2-production-hardening-review-2026-05-26.md
- Introduced a new error handling system with custom error types for better context and clarity in error reporting.
- Implemented URL validation to prevent invalid requests and enhance security.
- Updated cache key generation functions to return errors, improving robustness in handling invalid inputs.
- Added comprehensive metrics tracking for requests, cache hits, misses, and performance metrics, allowing for better monitoring and analysis of the caching system.
- Enhanced logging to include detailed metrics and error information for improved debugging and operational insights.
- Introduced multiple new markdown files detailing caching patterns, configuration management, development workflows, Go language conventions, HTTP proxy patterns, logging and monitoring practices, performance optimization guidelines, project structure, security validation, and VFS architecture.
- Each document outlines best practices, patterns, and guidelines to enhance the understanding and implementation of various components within the SteamCache2 project.
- This documentation aims to improve maintainability, facilitate onboarding for new contributors, and ensure consistent application of coding and architectural standards across the codebase.
- Updated the caching logic to utilize a predictive cache warmer, enhancing content prefetching based on access patterns.
- Replaced the legacy warming system with a more efficient predictive approach, allowing for better performance and resource management.
- Refactored memory management to integrate dynamic cache size adjustments based on system memory usage, improving overall efficiency.
- Simplified the VFS interface and improved concurrency handling with sharded locks for better performance in multi-threaded environments.
- Enhanced tests to validate the new caching and memory management behaviors, ensuring reliability and performance improvements.
- Replaced legacy depot file migration logic with concurrent directory scanning for improved performance.
- Introduced batch processing of files to minimize lock contention during initialization.
- Simplified the init function by removing unnecessary complexity and focusing on efficient file handling.
- Enhanced logging to provide better insights into directory scan progress and completion.
- Changed module name from `s1d3sw1ped/SteamCache2` to `s1d3sw1ped/steamcache2` for consistency.
- Updated all import paths and references throughout the codebase to reflect the new module name.
- Adjusted README and Makefile to use the updated module name, ensuring clarity in usage instructions.
- Updated the `downloadThroughCache` function to remove the upstream URL parameter, streamlining the caching process.
- Modified the `serializeRawResponse` function to eliminate unnecessary parameters, enhancing clarity and usability.
- Adjusted integration tests to align with the new function signatures, ensuring consistent testing of caching behavior.
- Updated caching logic to support size-based promotion filtering, ensuring that not all files may be promoted based on size constraints.
- Implemented adaptive caching strategies with a new AdaptiveCacheManager to analyze access patterns and adjust caching strategies dynamically.
- Introduced predictive caching features with a PredictiveCacheManager to prefetch content based on access patterns.
- Added a CacheWarmer to preload popular content into the cache, improving access times for frequently requested files.
- Refactored memory management with a DynamicCacheManager to adjust cache sizes based on system memory usage.
- Enhanced VFS interface and file metadata handling to support new features and improve performance.
- Updated tests to validate new caching behaviors and ensure reliability of the caching system.
- Updated the run command to execute the application from a built snapshot instead of using `go run`.
- Added a new run-debug command for running the application with debug logging.
- Consolidated the build process into a single target snapshot build command.
- Enhanced help output to reflect the new command structure.
- Introduced integration tests for SteamCache to validate caching behavior with real Steam URLs.
- Implemented a ServiceManager to manage service configurations, allowing for dynamic detection of services based on User-Agent.
- Updated cache key generation to include service prefixes, enhancing cache organization and retrieval.
- Enhanced the caching logic to support multiple services, starting with Steam and Epic Games.
- Improved .gitignore to exclude test cache files while retaining necessary structure.
- Introduced maxConcurrentRequests and maxRequestsPerClient fields in the Config struct to manage request limits.
- Updated the SteamCache implementation to utilize these new configuration options for controlling concurrent requests.
- Enhanced the ServeHTTP method to enforce global and per-client rate limiting using semaphores.
- Modified the root command to accept new flags for configuring concurrency limits via command-line arguments.
- Updated tests to reflect changes in the SteamCache initialization and request handling logic.
- Eliminated the threads variable and its associated logic for setting maximum processing threads.
- Simplified the command initialization by removing unnecessary flags related to thread management.
- Changed expected Content-Type from "application/octet-stream" to "application/x-steam-chunk" to align with Steam's file specifications.
- Enhanced warning message for unexpected content types to provide clearer context for debugging.
- Replaced SHA1 hash calculations with SHA256 for improved security and consistency in cache key generation.
- Introduced a new TestURLHashing function to validate the new cache key generation logic.
- Removed outdated hash calculation tests and streamlined the caching process to focus on URL-based hashing.
- Implemented lightweight validation methods in ServeHTTP to enhance performance and reliability of cached responses.
- Added batched time updates in VFS implementations for better performance during access time tracking.
- Removed the min function and the verifyResponseHash function to streamline the codebase.
- Updated extractHashFromSteamPath to use strings.TrimPrefix for cleaner path handling.
- Retained comments regarding removed Prometheus metrics for future reference.
- Introduced a YAML-based configuration system, allowing for automatic generation of a default `config.yaml` file.
- Updated the application to load configuration settings from the YAML file, improving flexibility and ease of use.
- Added a Makefile to streamline development tasks, including running the application, testing, and managing dependencies.
- Enhanced `.gitignore` to include build artifacts and configuration files.
- Removed unused Prometheus metrics and related code to simplify the codebase.
- Updated dependencies in `go.mod` and `go.sum` for improved functionality and performance.
- Added AccessCount field to FileInfo struct for improved tracking of file access frequency.
- Updated NewFileInfo and NewFileInfoFromOS functions to initialize AccessCount.
- Modified DiskFS and MemoryFS to preserve and increment AccessCount during file operations.
- Enhanced garbage collection methods (LRU, LFU, FIFO, Largest, Smallest, Hybrid) to utilize AccessCount for more effective space reclamation.
revert Enhance FileInfo structure and DiskFS functionality
- Added CTime (creation time) and AccessCount fields to FileInfo struct for better file metadata tracking.
- Updated NewFileInfo and NewFileInfoFromOS functions to initialize new fields.
- Enhanced DiskFS to maintain access counts and file metadata, including flushing to JSON files.
- Modified Open and Create methods to increment access counts and set creation times appropriately.
- Updated garbage collection logic to utilize real access counts for files.
- Changed memory-gc strategy from 'lfu' to 'lru' for improved cache management.
- Commented out the upstream server configuration to prevent potential connectivity issues during development.
- Added CTime (creation time) and AccessCount fields to FileInfo struct for better file metadata tracking.
- Updated NewFileInfo and NewFileInfoFromOS functions to initialize new fields.
- Enhanced DiskFS to maintain access counts and file metadata, including flushing to JSON files.
- Modified Open and Create methods to increment access counts and set creation times appropriately.
- Updated garbage collection logic to utilize real access counts for files.
- Introduced a map for hop-by-hop headers to be removed from responses.
- Enhanced cache serving logic to read and filter HTTP responses, ensuring only relevant headers are forwarded.
- Updated cache writing to handle full HTTP responses, improving cache integrity and performance.
- Added upstream server configuration to launch.json for improved connectivity.
- Increased HTTP client timeout from 60s to 120s for better handling of slow responses.
- Updated server timeouts in steamcache.go: increased ReadTimeout to 30s and WriteTimeout to 60s.
- Introduced ReadHeaderTimeout to mitigate header attacks and set MaxHeaderBytes to 1MB.
- Improved error logging in the Run method to include HTTP status codes for better debugging.
- Adjusted ServeHTTP method to handle root path and metrics endpoint correctly.
- Removed copyright footer from .goreleaser.yaml.
- Increased HTTP client connection settings in steamcache.go for improved performance:
- MaxIdleConns from 100 to 200
- MaxIdleConnsPerHost from 10 to 50
- IdleConnTimeout from 90s to 120s
- TLSHandshakeTimeout from 10s to 15s
- ResponseHeaderTimeout from 10s to 30s
- ExpectContinueTimeout from 1s to 5s
- Added DisableCompression and ForceAttemptHTTP2 options.
- Removed debug logging for manifest files in ServeHTTP method.
s1d3sw1ped