Justin Harms
3703e40442
- Introduced multiple new markdown files detailing caching patterns, configuration management, development workflows, Go language conventions, HTTP proxy patterns, logging and monitoring practices, performance optimization guidelines, project structure, security validation, and VFS architecture. - Each document outlines best practices, patterns, and guidelines to enhance the understanding and implementation of various components within the SteamCache2 project. - This documentation aims to improve maintainability, facilitate onboarding for new contributors, and ensure consistent application of coding and architectural standards across the codebase.
64 lines
2.7 KiB
Plaintext
64 lines
2.7 KiB
Plaintext
---
|
|
description: Caching system patterns and best practices
|
|
---
|
|
|
|
# Caching System Patterns
|
|
|
|
## Cache Key Generation
|
|
- Use SHA256 hashing for cache keys to ensure uniform distribution
|
|
- Include service prefix (e.g., "steam/", "epic/") based on User-Agent detection
|
|
- Never include query parameters in cache keys - strip them before hashing
|
|
- Cache keys should be deterministic and consistent
|
|
|
|
## Cache File Format
|
|
The cache uses a custom format with:
|
|
- Magic number: "SC2C" (SteamCache2 Cache)
|
|
- Content hash: SHA256 of response body
|
|
- Response size: Total HTTP response size
|
|
- Raw HTTP response: Complete response as received from upstream
|
|
- Header line format: "SC2C <hash> <size>\n"
|
|
- Integrity verification on read operations
|
|
- Automatic corruption detection and cleanup
|
|
|
|
## Garbage Collection Algorithms
|
|
Available algorithms and their use cases:
|
|
- **LRU**: Best for general gaming patterns, keeps recently accessed content
|
|
- **LFU**: Good for gaming cafes with popular games
|
|
- **FIFO**: Predictable behavior, good for testing
|
|
- **Largest**: Maximizes number of cached files
|
|
- **Smallest**: Maximizes cache hit rate
|
|
- **Hybrid**: Combines access time and file size for optimal performance
|
|
|
|
## Cache Validation
|
|
- Always verify Content-Length matches received data
|
|
- Use SHA256 hashing for content integrity
|
|
- Don't cache chunked transfer encoding (no Content-Length)
|
|
- Reject files with invalid or missing Content-Length
|
|
|
|
## Request Coalescing
|
|
- Multiple clients requesting the same file should share the download
|
|
- Use channels and mutexes to coordinate concurrent requests
|
|
- Buffer response data for coalesced clients
|
|
- Clean up coalesced request structures after completion
|
|
|
|
## Range Request Support
|
|
- Always cache the full file, regardless of Range headers
|
|
- Support serving partial content from cached full files
|
|
- Parse Range headers correctly (bytes=start-end, bytes=start-, bytes=-suffix)
|
|
- Return appropriate HTTP status codes (206 for partial content, 416 for invalid ranges)
|
|
|
|
## Service Detection
|
|
- Use regex patterns to match User-Agent strings
|
|
- Support multiple services (Steam, Epic Games, etc.)
|
|
- Cache keys include service prefix for isolation
|
|
- Default to Steam service configuration
|
|
|
|
## Memory vs Disk Caching
|
|
- Memory cache: Fast access, limited size, use LRU or LFU
|
|
- Disk cache: Slower access, large size, use Hybrid or Largest
|
|
- Tiered caching: Memory as L1, disk as L2
|
|
- Dynamic memory management with configurable thresholds
|
|
- Cache promotion: Move frequently accessed files from disk to memory
|
|
- Sharded storage: Use directory sharding for Steam keys to reduce inode pressure
|
|
- Memory-mapped files: Use mmap for large disk operations
|
|
- Batched operations: Group operations for better performance |