Justin Harms
3703e40442
- Introduced multiple new markdown files detailing caching patterns, configuration management, development workflows, Go language conventions, HTTP proxy patterns, logging and monitoring practices, performance optimization guidelines, project structure, security validation, and VFS architecture. - Each document outlines best practices, patterns, and guidelines to enhance the understanding and implementation of various components within the SteamCache2 project. - This documentation aims to improve maintainability, facilitate onboarding for new contributors, and ensure consistent application of coding and architectural standards across the codebase.
72 lines
2.6 KiB
Plaintext
72 lines
2.6 KiB
Plaintext
---
|
|
description: VFS (Virtual File System) patterns and architecture
|
|
---
|
|
|
|
# VFS (Virtual File System) Patterns
|
|
|
|
## Core VFS Interface
|
|
- Implement the `vfs.VFS` interface for all storage backends
|
|
- Use interface implementation verification: `var _ vfs.VFS = (*Implementation)(nil)`
|
|
- Support both memory and disk-based storage with the same interface
|
|
- Provide size and capacity information for monitoring
|
|
|
|
## Tiered Cache Architecture
|
|
- Use `vfs/cache/cache.go` for two-tier caching (memory + disk)
|
|
- Implement lock-free tier switching with `atomic.Value`
|
|
- Prefer disk tier for persistence, memory tier for speed
|
|
- Support cache promotion from disk to memory
|
|
|
|
## Sharded File Systems
|
|
- Use sharded directory structures for Steam cache keys
|
|
- Implement 2-level sharding: `steam/XX/YY/hash` for optimal performance
|
|
- Use `vfs/locks/sharding.go` for sharded locking
|
|
- Reduce inode pressure with directory sharding
|
|
|
|
## Memory Management
|
|
- Use `bytes.Buffer` for in-memory file storage
|
|
- Implement batched time updates for performance
|
|
- Use LRU lists for eviction tracking
|
|
- Monitor memory fragmentation and usage
|
|
|
|
## Disk Storage
|
|
- Use memory-mapped files (`mmap`) for large file operations
|
|
- Implement efficient file path sharding
|
|
- Use batched operations for better I/O performance
|
|
- Support concurrent access with proper locking
|
|
|
|
## Garbage Collection Integration
|
|
- Wrap VFS implementations with `vfs/gc/gc.go`
|
|
- Support multiple GC algorithms (LRU, LFU, FIFO, etc.)
|
|
- Implement async GC with configurable thresholds
|
|
- Use eviction functions from `vfs/eviction/eviction.go`
|
|
|
|
## Performance Optimizations
|
|
- Use sharded locks to reduce contention
|
|
- Implement batched time updates (100ms intervals)
|
|
- Use atomic operations for lock-free updates
|
|
- Monitor and log performance metrics
|
|
|
|
## Error Handling
|
|
- Use custom VFS errors from `vfs/vfserror/vfserror.go`
|
|
- Handle capacity exceeded scenarios gracefully
|
|
- Implement proper cleanup on errors
|
|
- Log VFS operations with context
|
|
|
|
## File Information Management
|
|
- Use `vfs/types/types.go` for file metadata
|
|
- Track access times, sizes, and other statistics
|
|
- Implement efficient file info storage and retrieval
|
|
- Support batched metadata updates
|
|
|
|
## Adaptive and Predictive Features
|
|
- Integrate with `vfs/adaptive/adaptive.go` for learning patterns
|
|
- Use `vfs/predictive/predictive.go` for cache warming
|
|
- Implement intelligent cache promotion strategies
|
|
- Monitor access patterns for optimization
|
|
|
|
## Testing VFS Implementations
|
|
- Test with realistic file sizes and access patterns
|
|
- Verify concurrent access scenarios
|
|
- Test garbage collection behavior
|
|
- Validate sharding and path generation
|
|
- Test error conditions and edge cases |