s1d3sw1ped/steamcache2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Enhance DiskFS initialization and error handling

Browse Source 
- Updated `disk.New` to support asynchronous initialization for large caches, improving responsiveness during startup.
- Introduced an eviction function parameter to `disk.New`, ensuring proper handling of over-capacity scenarios.
- Enhanced error handling in various components, including memory and disk tests, to ensure robustness and clarity.
- Refactored tests to validate new behaviors, including checks for delayed attachment and proper error propagation.
- Removed obsolete error handling code and tests related to the now-deleted errors package, streamlining the codebase.
This commit is contained in:
Justin Harms
2026-05-27 13:15:33 -05:00
parent 4861f93e6f
commit feda55e225
18 changed files with 584 additions and 1380 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+8
View File
@@ -139,6 +139,14 @@ See `config.Validate()` and `steamcache.New` error paths. This ensures the LAN a
- Recommended: migrate to `NewWithOptions(Options{...})` (non-breaking) or rely on YAML config + cmd/root.go.
- No behavior change for existing configs (defaults preserve prior semantics).
#### Large Cache Initialization (async DiskFS population)
- `disk.New(root, capacity, evictFn)` signature changed (now takes evict func from `gc.GetGCAlgorithm`, returns error for ctor hygiene). Callers updated internally; direct vfs/disk users must pass the evict (or nil for no startup guard).
- DiskFS initialization is now fully asynchronous for large caches (millions of files): `New` returns immediately without scanning. The first `Size()` (and many internal callers) blocks on an internal barrier until bg streaming population + any startup over-cap eviction (using the evictFn) completes. Subsequent `Size()` calls are instant.
- During the "proxy window" (while bg scan runs): disk-only configs (memory.size=0) have TieredCache Create returning `ErrNotFound` (no disk writes/caching occurs until attach); mem+disk configs serve from memory tier only. This keeps `New` fast and avoids heavy disk I/O/eviction during long scans on slow storage.
- The explicit startup guard (reduce size if pre-existing on-disk > cap) runs as the literal last step of bg init, before the barrier opens.
- Add a note for operators: very large disk caches (tens/hundreds GB with millions files) may show extended "memory-only or no-cache" behavior at startup (seconds to minutes depending on storage speed); this is by design for responsiveness.
- Godoc on `disk.New` and `DiskFS.Size` expanded with the barrier/attach behavior.
#### Garbage Collection Algorithms
SteamCache2 supports different garbage collection algorithms for memory and disk caches, allowing you to optimize performance for each storage tier: