diff --git a/.cursor/rules/caching-patterns.mdc b/.cursor/rules/caching-patterns.mdc new file mode 100644 index 0000000..1d218f5 --- /dev/null +++ b/.cursor/rules/caching-patterns.mdc @@ -0,0 +1,64 @@ +--- +description: Caching system patterns and best practices +--- + +# Caching System Patterns + +## Cache Key Generation +- Use SHA256 hashing for cache keys to ensure uniform distribution +- Include service prefix (e.g., "steam/", "epic/") based on User-Agent detection +- Never include query parameters in cache keys - strip them before hashing +- Cache keys should be deterministic and consistent + +## Cache File Format +The cache uses a custom format with: +- Magic number: "SC2C" (SteamCache2 Cache) +- Content hash: SHA256 of response body +- Response size: Total HTTP response size +- Raw HTTP response: Complete response as received from upstream +- Header line format: "SC2C \n" +- Integrity verification on read operations +- Automatic corruption detection and cleanup + +## Garbage Collection Algorithms +Available algorithms and their use cases: +- **LRU**: Best for general gaming patterns, keeps recently accessed content +- **LFU**: Good for gaming cafes with popular games +- **FIFO**: Predictable behavior, good for testing +- **Largest**: Maximizes number of cached files +- **Smallest**: Maximizes cache hit rate +- **Hybrid**: Combines access time and file size for optimal performance + +## Cache Validation +- Always verify Content-Length matches received data +- Use SHA256 hashing for content integrity +- Don't cache chunked transfer encoding (no Content-Length) +- Reject files with invalid or missing Content-Length + +## Request Coalescing +- Multiple clients requesting the same file should share the download +- Use channels and mutexes to coordinate concurrent requests +- Buffer response data for coalesced clients +- Clean up coalesced request structures after completion + +## Range Request Support +- Always cache the full file, regardless of Range headers +- Support serving partial content from cached full files +- Parse Range headers correctly (bytes=start-end, bytes=start-, bytes=-suffix) +- Return appropriate HTTP status codes (206 for partial content, 416 for invalid ranges) + +## Service Detection +- Use regex patterns to match User-Agent strings +- Support multiple services (Steam, Epic Games, etc.) +- Cache keys include service prefix for isolation +- Default to Steam service configuration + +## Memory vs Disk Caching +- Memory cache: Fast access, limited size, use LRU or LFU +- Disk cache: Slower access, large size, use Hybrid or Largest +- Tiered caching: Memory as L1, disk as L2 +- Dynamic memory management with configurable thresholds +- Cache promotion: Move frequently accessed files from disk to memory +- Sharded storage: Use directory sharding for Steam keys to reduce inode pressure +- Memory-mapped files: Use mmap for large disk operations +- Batched operations: Group operations for better performance \ No newline at end of file diff --git a/.cursor/rules/configuration-patterns.mdc b/.cursor/rules/configuration-patterns.mdc new file mode 100644 index 0000000..0856236 --- /dev/null +++ b/.cursor/rules/configuration-patterns.mdc @@ -0,0 +1,65 @@ +--- +description: Configuration management patterns +--- + +# Configuration Management Patterns + +## YAML Configuration +- Use YAML format for human-readable configuration +- Provide sensible defaults for all configuration options +- Validate configuration on startup +- Generate default configuration file on first run + +## Configuration Structure +- Group related settings in nested structures +- Use descriptive field names with YAML tags +- Provide default values in struct tags where possible +- Use appropriate data types (strings for sizes, ints for limits) + +## Size Configuration +- Use human-readable size strings (e.g., "1GB", "512MB") +- Parse sizes using `github.com/docker/go-units` +- Support "0" to disable cache layers +- Validate size limits are reasonable + +## Garbage Collection Configuration +- Support multiple GC algorithms per cache layer +- Provide algorithm-specific configuration options +- Allow different algorithms for memory vs disk caches +- Document algorithm characteristics and use cases + +## Server Configuration +- Configure listen address and port +- Set concurrency limits (global and per-client) +- Configure upstream server URL +- Support both absolute and relative upstream URLs + +## Runtime Configuration +- Allow command-line overrides for critical settings +- Support configuration file path specification +- Provide help and version information +- Validate configuration before starting services + +## Default Configuration +- Generate appropriate defaults for different use cases +- Consider system resources when setting defaults +- Provide conservative defaults for home users +- Document configuration options in comments + +## Configuration Validation +- Validate required fields are present +- Check that size limits are reasonable +- Verify file paths are accessible +- Test upstream server connectivity + +## Configuration Updates +- Support configuration reloading (if needed) +- Handle configuration changes gracefully +- Log configuration changes +- Maintain backward compatibility + +## Environment-Specific Configuration +- Support different configurations for development/production +- Allow environment variable overrides +- Provide configuration templates for common scenarios +- Document configuration best practices \ No newline at end of file diff --git a/.cursor/rules/development-workflow.mdc b/.cursor/rules/development-workflow.mdc new file mode 100644 index 0000000..53fe6bb --- /dev/null +++ b/.cursor/rules/development-workflow.mdc @@ -0,0 +1,77 @@ +--- +description: Development workflow and best practices +--- + +# Development Workflow for SteamCache2 + +## Build System +- Use the provided [Makefile](mdc:Makefile) for all build operations +- Prefer `make` commands over direct `go` commands +- Use `make test` to run all tests before committing +- Use `make run-debug` for development with debug logging + +## Code Organization +- Keep related functionality in the same package +- Use clear package boundaries and interfaces +- Minimize dependencies between packages +- Follow the existing project structure + +## Git Workflow +- Use descriptive commit messages +- Keep commits focused and atomic +- Test changes thoroughly before committing +- Use meaningful branch names + +## Code Review +- Review code for correctness and performance +- Check for proper error handling +- Verify test coverage for new functionality +- Ensure code follows project conventions + +## Documentation +- Update README.md for user-facing changes +- Add comments for complex algorithms +- Document configuration options +- Keep API documentation current + +## Testing Strategy +- Write tests for new functionality +- Maintain high test coverage +- Test edge cases and error conditions +- Run integration tests before major releases + +## Performance Testing +- Test with realistic data sizes +- Measure performance impact of changes +- Profile the application under load +- Monitor memory usage and leaks + +## Configuration Management +- Test configuration changes thoroughly +- Validate configuration on startup +- Provide sensible defaults +- Document configuration options + +## Error Handling +- Implement proper error handling +- Use structured logging for errors +- Provide meaningful error messages +- Handle edge cases gracefully + +## Security Considerations +- Validate all inputs +- Implement proper rate limiting +- Log security-relevant events +- Follow security best practices + +## Release Process +- Test thoroughly before releasing +- Update version information +- Create release notes +- Tag releases appropriately + +## Maintenance +- Monitor application performance +- Update dependencies regularly +- Fix bugs promptly +- Refactor code when needed \ No newline at end of file diff --git a/.cursor/rules/golang-conventions.mdc b/.cursor/rules/golang-conventions.mdc new file mode 100644 index 0000000..2a66535 --- /dev/null +++ b/.cursor/rules/golang-conventions.mdc @@ -0,0 +1,62 @@ +--- +globs: *.go +--- + +# Go Language Conventions for SteamCache2 + +## Code Style +- Use `gofmt` and `goimports` for formatting +- Follow standard Go naming conventions (camelCase for private, PascalCase for public) +- Use meaningful variable names that reflect their purpose +- Prefer explicit error handling over panic (except in constructors where configuration is invalid) + +## Package Organization +- Keep packages focused and cohesive +- Use internal packages for implementation details that shouldn't be exported +- Group related functionality together (e.g., all VFS implementations in `vfs/`) +- Use interface implementation verification: `var _ Interface = (*Implementation)(nil)` +- Create type aliases for backward compatibility when refactoring +- Use separate packages for different concerns (e.g., `vfserror`, `types`, `locks`) + +## Error Handling +- Always handle errors explicitly - never ignore them with `_` +- Use `fmt.Errorf` with `%w` verb for error wrapping +- Log errors with context using structured logging (zerolog) +- Return meaningful error messages that help with debugging +- Create custom error types for domain-specific errors (see `vfs/vfserror/`) +- Use `errors.New()` for simple error constants +- Include relevant context in error messages (file paths, operation names) + +## Testing +- All tests should run with a timeout (as per user rules) +- Use table-driven tests for multiple test cases +- Use `t.Helper()` in test helper functions +- Test both success and failure cases +- Use `t.TempDir()` for temporary files in tests + +## Concurrency +- Use `sync.RWMutex` for read-heavy operations +- Prefer channels over shared memory when possible +- Use `context.Context` for cancellation and timeouts +- Be explicit about goroutine lifecycle management +- Use sharded locks for high-concurrency scenarios (see `vfs/locks/sharding.go`) +- Use `atomic.Value` for lock-free data structure updates +- Use `sync.Map` for concurrent map operations when appropriate + +## Performance +- Use `io.ReadAll` sparingly - prefer streaming for large data +- Use connection pooling for HTTP clients +- Implement proper resource cleanup (defer statements) +- Use buffered channels when appropriate + +## Logging +- Use structured logging with zerolog +- Include relevant context in log messages (keys, URLs, client IPs) +- Use appropriate log levels (Debug, Info, Warn, Error) +- Avoid logging sensitive information + +## Memory Management +- Be mindful of memory usage in caching scenarios +- Use appropriate data structures for the use case +- Implement proper cleanup for long-running services +- Monitor memory usage in production \ No newline at end of file diff --git a/.cursor/rules/http-proxy-patterns.mdc b/.cursor/rules/http-proxy-patterns.mdc new file mode 100644 index 0000000..4aabcf1 --- /dev/null +++ b/.cursor/rules/http-proxy-patterns.mdc @@ -0,0 +1,59 @@ +--- +description: HTTP proxy and server patterns +--- + +# HTTP Proxy and Server Patterns + +## Request Handling +- Only support GET requests (Steam doesn't use other methods) +- Reject non-GET requests with 405 Method Not Allowed +- Handle health checks at "/" endpoint +- Support LanCache heartbeat at "/lancache-heartbeat" + +## Upstream Communication +- Use optimized HTTP transport with connection pooling +- Set appropriate timeouts (10s dial, 15s header, 60s total) +- Enable HTTP/2 and keep-alives for better performance +- Use large buffers (64KB) for better throughput + +## Response Streaming +- Stream responses directly to clients for better performance +- Support both full file and range request streaming +- Preserve original HTTP headers (excluding hop-by-hop headers) +- Add cache-specific headers (X-LanCache-Status, X-LanCache-Processed-By) + +## Error Handling +- Implement retry logic with exponential backoff +- Handle upstream server errors gracefully +- Return appropriate HTTP status codes +- Log errors with sufficient context for debugging + +## Concurrency Control +- Use semaphores to limit concurrent requests globally +- Implement per-client rate limiting +- Clean up old client limiters to prevent memory leaks +- Use proper synchronization for shared data structures + +## Header Management +- Copy relevant headers from upstream responses +- Exclude hop-by-hop headers (Connection, Keep-Alive, etc.) +- Add cache status headers for monitoring +- Preserve Content-Type and Content-Length headers + +## Client IP Detection +- Check X-Forwarded-For header first (for proxy setups) +- Fall back to X-Real-IP header +- Use RemoteAddr as final fallback +- Handle comma-separated IP lists in X-Forwarded-For + +## Performance Optimizations +- Set keep-alive headers for better connection reuse +- Use appropriate server timeouts +- Implement request coalescing for duplicate requests +- Use buffered I/O for better performance + +## Security Considerations +- Validate request URLs and paths +- Implement rate limiting to prevent abuse +- Log suspicious activity +- Handle malformed requests gracefully \ No newline at end of file diff --git a/.cursor/rules/logging-monitoring-patterns.mdc b/.cursor/rules/logging-monitoring-patterns.mdc new file mode 100644 index 0000000..725fd7d --- /dev/null +++ b/.cursor/rules/logging-monitoring-patterns.mdc @@ -0,0 +1,87 @@ +--- +description: Logging and monitoring patterns for SteamCache2 +--- + +# Logging and Monitoring Patterns + +## Structured Logging with Zerolog +- Use zerolog for all logging operations +- Include structured fields for better querying and analysis +- Use appropriate log levels: Debug, Info, Warn, Error +- Include timestamps and context in all log messages +- Configure log format (JSON for production, console for development) + +## Log Context and Fields +- Always include relevant context in log messages +- Use consistent field names: `client_ip`, `cache_key`, `url`, `service` +- Include operation duration with `Dur()` for performance monitoring +- Log cache hit/miss status for analytics +- Include file sizes and operation counts for monitoring + +## Performance Monitoring +- Log request processing times with `zduration` field +- Monitor cache hit/miss ratios +- Track memory and disk usage +- Log garbage collection events and statistics +- Monitor concurrent request counts and limits + +## Error Logging +- Log errors with full context and stack traces +- Include relevant request information in error logs +- Use structured error logging with `Err()` field +- Log configuration errors with file paths +- Include upstream server errors with status codes + +## Cache Operation Logging +- Log cache hits with key and response time +- Log cache misses with reason and upstream response time +- Log cache corruption detection and cleanup +- Log garbage collection operations and evicted items +- Log cache promotion events (disk to memory) + +## Service Detection Logging +- Log service detection results (Steam, Epic, etc.) +- Log User-Agent patterns and matches +- Log service configuration changes +- Log cache key generation for different services + +## HTTP Request Logging +- Log incoming requests with method, URL, and client IP +- Log response status codes and sizes +- Log upstream server communication +- Log rate limiting events and client limits +- Log health check and heartbeat requests + +## Configuration Logging +- Log configuration loading and validation +- Log default configuration generation +- Log configuration changes and overrides +- Log startup parameters and settings + +## Security Event Logging +- Log suspicious request patterns +- Log rate limiting violations +- Log authentication failures (if applicable) +- Log configuration security issues +- Log potential security threats + +## System Health Logging +- Log memory usage and fragmentation +- Log disk usage and capacity +- Log connection pool statistics +- Log goroutine counts and lifecycle +- Log system resource utilization + +## Log Rotation and Management +- Implement log rotation for long-running services +- Use appropriate log retention policies +- Monitor log file sizes and disk usage +- Configure log levels for different environments +- Use structured logging for log analysis tools + +## Monitoring Integration +- Design logs for easy parsing by monitoring tools +- Include metrics that can be scraped by Prometheus +- Use consistent field naming for dashboard creation +- Log events that can trigger alerts +- Include correlation IDs for request tracing \ No newline at end of file diff --git a/.cursor/rules/performance-optimization.mdc b/.cursor/rules/performance-optimization.mdc new file mode 100644 index 0000000..6162b29 --- /dev/null +++ b/.cursor/rules/performance-optimization.mdc @@ -0,0 +1,71 @@ +--- +description: Performance optimization guidelines +--- + +# Performance Optimization Guidelines + +## Memory Management +- Use appropriate data structures for the use case +- Implement proper cleanup for long-running services +- Monitor memory usage and implement limits +- Use memory pools for frequently allocated objects + +## I/O Optimization +- Use buffered I/O for better performance +- Implement connection pooling for HTTP clients +- Use appropriate buffer sizes (64KB for HTTP) +- Minimize system calls and context switches + +## Concurrency Patterns +- Use worker pools for CPU-intensive tasks +- Implement proper backpressure with semaphores +- Use channels for coordination between goroutines +- Avoid excessive goroutine creation + +## Caching Strategies +- Use tiered caching (memory + disk) for optimal performance +- Implement intelligent cache eviction policies +- Use cache warming for predictable access patterns +- Monitor cache hit ratios and adjust strategies + +## Network Optimization +- Use HTTP/2 when available +- Enable connection keep-alives +- Use appropriate timeouts for different operations +- Implement request coalescing for duplicate requests + +## Data Structures +- Choose appropriate data structures for access patterns +- Use sync.RWMutex for read-heavy operations +- Consider lock-free data structures where appropriate +- Minimize memory allocations in hot paths + +## Algorithm Selection +- Choose GC algorithms based on access patterns +- Use LRU for general gaming workloads +- Use LFU for gaming cafes with popular content +- Use Hybrid algorithms for mixed workloads + +## Monitoring and Profiling +- Implement performance metrics collection +- Use structured logging for performance analysis +- Monitor key performance indicators +- Profile the application under realistic loads + +## Resource Management +- Implement proper resource cleanup +- Use context.Context for cancellation +- Set appropriate limits on resource usage +- Monitor resource consumption over time + +## Scalability Considerations +- Design for horizontal scaling where possible +- Use sharding for large datasets +- Implement proper load balancing +- Consider distributed caching for large deployments + +## Bottleneck Identification +- Profile the application to identify bottlenecks +- Focus optimization efforts on the most critical paths +- Use appropriate tools for performance analysis +- Test performance under realistic conditions \ No newline at end of file diff --git a/.cursor/rules/project-structure.mdc b/.cursor/rules/project-structure.mdc new file mode 100644 index 0000000..89286a0 --- /dev/null +++ b/.cursor/rules/project-structure.mdc @@ -0,0 +1,57 @@ +--- +alwaysApply: true +--- + +# SteamCache2 Project Structure Guide + +This is a high-performance Steam download cache written in Go. The main entry point is [main.go](mdc:main.go), which delegates to the command structure in [cmd/](mdc:cmd/). + +## Core Architecture + +- **Main Entry**: [main.go](mdc:main.go) - Simple entry point that calls `cmd.Execute()` +- **Command Layer**: [cmd/root.go](mdc:cmd/root.go) - CLI interface using Cobra, handles configuration loading and service startup +- **Core Service**: [steamcache/steamcache.go](mdc:steamcache/steamcache.go) - Main HTTP proxy and caching logic +- **Configuration**: [config/config.go](mdc:config/config.go) - YAML-based configuration management +- **Virtual File System**: [vfs/](mdc:vfs/) - Abstracted storage layer supporting memory and disk caches + +## Key Components + +### VFS (Virtual File System) +- [vfs/vfs.go](mdc:vfs/vfs.go) - Core VFS interface +- [vfs/memory/](mdc:vfs/memory/) - In-memory cache implementation +- [vfs/disk/](mdc:vfs/disk/) - Disk-based cache implementation +- [vfs/cache/](mdc:vfs/cache/) - Cache coordination layer +- [vfs/gc/](mdc:vfs/gc/) - Garbage collection algorithms (LRU, LFU, FIFO, etc.) + +### Service Management +- Service detection via User-Agent patterns +- Support for multiple gaming services (Steam, Epic, etc.) +- SHA256-based cache key generation with service prefixes + +### Advanced Features +- [vfs/adaptive/](mdc:vfs/adaptive/) - Adaptive caching strategies +- [vfs/predictive/](mdc:vfs/predictive/) - Predictive cache warming +- Request coalescing for concurrent downloads +- Range request support for partial content + +## Development Workflow + +Use the [Makefile](mdc:Makefile) for development: +- `make` - Run tests and build +- `make test` - Run all tests +- `make run` - Run the application +- `make run-debug` - Run with debug logging + +## Testing + +- Unit tests: [steamcache/steamcache_test.go](mdc:steamcache/steamcache_test.go) +- Integration tests: [steamcache/integration_test.go](mdc:steamcache/integration_test.go) +- Test cache data: [steamcache/test_cache/](mdc:steamcache/test_cache/) + +## Configuration + +Default configuration is generated in [config.yaml](mdc:config.yaml) on first run. The application supports: +- Memory and disk cache sizing +- Garbage collection algorithm selection +- Concurrency limits +- Upstream server configuration \ No newline at end of file diff --git a/.cursor/rules/security-validation-patterns.mdc b/.cursor/rules/security-validation-patterns.mdc new file mode 100644 index 0000000..a3ba40a --- /dev/null +++ b/.cursor/rules/security-validation-patterns.mdc @@ -0,0 +1,89 @@ +--- +description: Security and validation patterns for SteamCache2 +--- + +# Security and Validation Patterns + +## Input Validation +- Validate all HTTP request parameters and headers +- Sanitize file paths and cache keys to prevent directory traversal +- Validate URL paths before processing +- Check Content-Length headers for reasonable values +- Reject malformed or suspicious requests + +## Cache Key Security +- Use SHA256 hashing for all cache keys to prevent collisions +- Never include user input directly in cache keys +- Strip query parameters from URLs before hashing +- Use service prefixes to isolate different services +- Validate cache key format and length + +## Content Integrity +- Always verify Content-Length matches received data +- Use SHA256 hashing for content integrity verification +- Don't cache chunked transfer encoding (no Content-Length) +- Reject files with invalid or missing Content-Length +- Implement cache file format validation with magic numbers + +## Rate Limiting and DoS Protection +- Implement global concurrency limits with semaphores +- Use per-client rate limiting to prevent abuse +- Clean up old client limiters to prevent memory leaks +- Set appropriate timeouts for all operations +- Monitor and log suspicious activity + +## HTTP Security +- Only support GET requests (Steam doesn't use other methods) +- Validate HTTP method and reject unsupported methods +- Handle malformed HTTP requests gracefully +- Implement proper error responses with appropriate status codes +- Use hop-by-hop header filtering + +## Client IP Detection +- Check X-Forwarded-For header for proxy setups +- Fall back to X-Real-IP header +- Use RemoteAddr as final fallback +- Handle comma-separated IP lists in X-Forwarded-For +- Log client IPs for monitoring and debugging + +## Service Detection Security +- Use regex patterns for User-Agent matching +- Validate service configurations before use +- Support multiple services with proper isolation +- Default to Steam service configuration +- Log service detection for monitoring + +## Error Handling Security +- Don't expose internal system information in error messages +- Log detailed errors for debugging but return generic messages to clients +- Handle errors gracefully without crashing +- Implement proper cleanup on errors +- Use structured logging for security events + +## Configuration Security +- Validate configuration values on startup +- Use sensible defaults for security-sensitive settings +- Validate file paths and permissions +- Check upstream server connectivity +- Log configuration changes + +## Memory and Resource Security +- Implement memory limits to prevent OOM attacks +- Use proper resource cleanup and garbage collection +- Monitor memory usage and implement alerts +- Use bounded data structures where possible +- Implement proper connection limits + +## Logging Security +- Don't log sensitive information (passwords, tokens) +- Use structured logging for security events +- Include relevant context (IPs, URLs, timestamps) +- Implement log rotation and retention policies +- Monitor logs for security issues + +## Network Security +- Use HTTPS for upstream connections when possible +- Implement proper TLS configuration +- Use connection pooling with appropriate limits +- Set reasonable timeouts for network operations +- Monitor network traffic for anomalies \ No newline at end of file diff --git a/.cursor/rules/steamcache2-overview.mdc b/.cursor/rules/steamcache2-overview.mdc new file mode 100644 index 0000000..bd630cb --- /dev/null +++ b/.cursor/rules/steamcache2-overview.mdc @@ -0,0 +1,48 @@ +--- +alwaysApply: true +--- + +# SteamCache2 Overview + +SteamCache2 is a high-performance HTTP proxy cache specifically designed for Steam game downloads. It reduces bandwidth usage and speeds up downloads by caching game files locally. + +## Key Features +- **Tiered Caching**: Memory + disk cache with intelligent promotion +- **Service Detection**: Automatically detects Steam clients via User-Agent +- **Request Coalescing**: Multiple clients share downloads of the same file +- **Range Support**: Serves partial content from cached full files +- **Garbage Collection**: Multiple algorithms (LRU, LFU, FIFO, Hybrid, etc.) +- **Adaptive Caching**: Learns from access patterns for better performance + +## Architecture +- **HTTP Proxy**: Intercepts Steam requests and serves from cache when possible +- **VFS Layer**: Abstracted storage supporting memory and disk caches +- **Service Manager**: Handles multiple gaming services (Steam, Epic, etc.) +- **GC System**: Intelligent cache eviction with configurable algorithms + +## Development +- **Language**: Go 1.23+ +- **Build**: Use `make` commands (see [Makefile](mdc:Makefile)) +- **Testing**: Comprehensive unit and integration tests +- **Configuration**: YAML-based with automatic generation + +## Performance +- **Concurrency**: Configurable request limits and rate limiting +- **Memory**: Dynamic memory management with configurable thresholds +- **Network**: Optimized HTTP transport with connection pooling +- **Storage**: Efficient cache file format with integrity verification + +## Use Cases +- **Gaming Cafes**: Reduce bandwidth costs and improve download speeds +- **LAN Events**: Share game downloads across multiple clients +- **Home Networks**: Speed up game updates for multiple gamers +- **Development**: Test game downloads without hitting Steam servers + +## Configuration +Default configuration is generated on first run. Key settings: +- Cache sizes (memory/disk) +- Garbage collection algorithms +- Concurrency limits +- Upstream server configuration + +See [config.yaml](mdc:config.yaml) for configuration options and [README.md](mdc:README.md) for detailed setup instructions. \ No newline at end of file diff --git a/.cursor/rules/testing-guidelines.mdc b/.cursor/rules/testing-guidelines.mdc new file mode 100644 index 0000000..ada6eb5 --- /dev/null +++ b/.cursor/rules/testing-guidelines.mdc @@ -0,0 +1,78 @@ +--- +globs: *_test.go +--- + +# Testing Guidelines for SteamCache2 + +## Test Structure +- Use table-driven tests for multiple test cases +- Group related tests in the same test function when appropriate +- Use descriptive test names that explain what is being tested +- Include both positive and negative test cases + +## Test Data Management +- Use `t.TempDir()` for temporary files and directories +- Clean up resources in defer statements +- Use unique temporary directories for each test to avoid conflicts +- Don't rely on external services in unit tests + +## Integration Testing +- Mark integration tests with `testing.Short()` checks +- Use real Steam URLs for integration tests when appropriate +- Test both cache hits and cache misses +- Verify response integrity between direct and cached responses +- Test against actual Steam servers for real-world validation +- Use `httptest.NewServer` for local testing scenarios +- Compare direct vs cached responses byte-for-byte + +## Mocking and Stubbing +- Use `httptest.NewServer` for HTTP server mocking +- Create mock responses that match real Steam responses +- Test error conditions and edge cases +- Use `httptest.NewRecorder` for response testing + +## Performance Testing +- Test with realistic data sizes +- Measure cache hit/miss ratios +- Test concurrent request handling +- Verify memory usage doesn't grow unbounded + +## Cache Testing +- Test cache key generation and uniqueness +- Verify cache file format serialization/deserialization +- Test garbage collection algorithms +- Test cache eviction policies +- Test cache corruption scenarios and recovery +- Verify cache file format integrity (magic numbers, hashes) +- Test range request handling from cached files +- Test request coalescing behavior + +## Service Detection Testing +- Test User-Agent pattern matching +- Test service configuration management +- Test cache key generation for different services +- Test service expandability (adding new services) + +## Error Handling Testing +- Test network failures and timeouts +- Test malformed requests and responses +- Test cache corruption scenarios +- Test resource exhaustion conditions + +## Test Timeouts +- All tests should run with appropriate timeouts +- Use `context.WithTimeout` for long-running operations +- Set reasonable timeouts for network operations +- Fail fast on obvious errors + +## Test Coverage +- Aim for high test coverage on critical paths +- Test edge cases and error conditions +- Test concurrent access patterns +- Test resource cleanup and memory management + +## Test Documentation +- Document complex test scenarios +- Explain the purpose of integration tests +- Include comments for non-obvious test logic +- Document expected behavior and assumptions \ No newline at end of file diff --git a/.cursor/rules/vfs-patterns.mdc b/.cursor/rules/vfs-patterns.mdc new file mode 100644 index 0000000..23a4261 --- /dev/null +++ b/.cursor/rules/vfs-patterns.mdc @@ -0,0 +1,72 @@ +--- +description: VFS (Virtual File System) patterns and architecture +--- + +# VFS (Virtual File System) Patterns + +## Core VFS Interface +- Implement the `vfs.VFS` interface for all storage backends +- Use interface implementation verification: `var _ vfs.VFS = (*Implementation)(nil)` +- Support both memory and disk-based storage with the same interface +- Provide size and capacity information for monitoring + +## Tiered Cache Architecture +- Use `vfs/cache/cache.go` for two-tier caching (memory + disk) +- Implement lock-free tier switching with `atomic.Value` +- Prefer disk tier for persistence, memory tier for speed +- Support cache promotion from disk to memory + +## Sharded File Systems +- Use sharded directory structures for Steam cache keys +- Implement 2-level sharding: `steam/XX/YY/hash` for optimal performance +- Use `vfs/locks/sharding.go` for sharded locking +- Reduce inode pressure with directory sharding + +## Memory Management +- Use `bytes.Buffer` for in-memory file storage +- Implement batched time updates for performance +- Use LRU lists for eviction tracking +- Monitor memory fragmentation and usage + +## Disk Storage +- Use memory-mapped files (`mmap`) for large file operations +- Implement efficient file path sharding +- Use batched operations for better I/O performance +- Support concurrent access with proper locking + +## Garbage Collection Integration +- Wrap VFS implementations with `vfs/gc/gc.go` +- Support multiple GC algorithms (LRU, LFU, FIFO, etc.) +- Implement async GC with configurable thresholds +- Use eviction functions from `vfs/eviction/eviction.go` + +## Performance Optimizations +- Use sharded locks to reduce contention +- Implement batched time updates (100ms intervals) +- Use atomic operations for lock-free updates +- Monitor and log performance metrics + +## Error Handling +- Use custom VFS errors from `vfs/vfserror/vfserror.go` +- Handle capacity exceeded scenarios gracefully +- Implement proper cleanup on errors +- Log VFS operations with context + +## File Information Management +- Use `vfs/types/types.go` for file metadata +- Track access times, sizes, and other statistics +- Implement efficient file info storage and retrieval +- Support batched metadata updates + +## Adaptive and Predictive Features +- Integrate with `vfs/adaptive/adaptive.go` for learning patterns +- Use `vfs/predictive/predictive.go` for cache warming +- Implement intelligent cache promotion strategies +- Monitor access patterns for optimization + +## Testing VFS Implementations +- Test with realistic file sizes and access patterns +- Verify concurrent access scenarios +- Test garbage collection behavior +- Validate sharding and path generation +- Test error conditions and edge cases \ No newline at end of file