# SteamCache2

SteamCache2 is a blazing fast download cache for Steam, designed to reduce bandwidth usage and speed up game downloads.

## Features

- High-speed caching for Steam downloads
- Tiered storage for getting the most out of your storage media
- Garbage Collected storage for limiting the size of RAM or Disk cache and will not go above what you choose or stop caching unlike others
- Reduces bandwidth usage
- Easy to set up and configure aside from dns stuff to trick Steam into using it
- Supports multiple clients
- YAML configuration with automatic config generation on first run
- Makefile for development and validation workflows
- Cross-platform builds (Linux, macOS, Windows)

## Quick Start

### First Time Setup

1. **Clone and build:**
   ```bash
   git clone <repository-url>
   cd steamcache2
   make  # This will run tests and build the application
   ```

2. **Run the application** (it will create a default config):
   ```bash
   ./steamcache2
   # or on Windows:
   steamcache2.exe
   ```

   The application will automatically create a `config.yaml` file with default settings and exit, allowing you to customize it.

3. **Edit the configuration** (`config.yaml`) for a real Steam front-door:

   Leave `upstream` empty. With an empty upstream, steamcache2 fetches from the request `Host` (the same pattern SteamPrefill / real Steam clients use when DNS points them at your cache). Do **not** paste a fake host like `https://steam.cdn.com` — that is not a Steam CDN and will not put you in front of Steam.

   ```yaml
   listen_address: :80
   cache:
       memory:
           size: 1GB
           gc_algorithm: lru
       disk:
           size: 10GB
           path: ./disk
           gc_algorithm: hybrid
   # Empty upstream = use the client Host as the origin (Steam CDN names only).
   upstream: ""
   ```

4. **Point Steam (or SteamPrefill) at this cache** before you expect hits:

   - **LAN DNS:** resolve `lancache.steamcontent.com` (and other Steam content names your clients use) to this server's LAN IP.
   - **Single Windows PC:** add a hosts override — see [Windows Hosts File Override](#windows-hosts-file-override) (`<cache-ip> lancache.steamcontent.com`).
   - Restart Steam (or your prefill tool) after DNS/hosts changes.

   Empty-upstream direct fetch only allows Steam CDN host suffixes (`steamcontent.com`, `steampowered.com`, `steamstatic.com`). Literal IPs and unrelated hosts are rejected.

5. **Run the application again:**
   ```bash
   make run  # or ./steamcache2
   ```

### Quick check: is it caching?

After steamcache2 is running (default `listen_address: :80`) and has seen a little Steam traffic — a game download, a short SteamPrefill pass, or any cacheable request — confirm hits vs misses from the existing endpoints. You do not need a full benchmark or log diving.

```bash
make validate-check
# or, manually:
curl -s http://localhost/metrics
curl -s -i http://localhost/lancache-heartbeat
```

`make validate-check` prints the full `/metrics` dump, highlights hit/miss plus `upstream_errors` / `cache_write_failures` / `rate_limited`, and curls `/lancache-heartbeat`. It also asserts the empty-upstream Host allowlist: a non-Steam `Host` sent with a Steam `User-Agent` must be rejected with HTTP 400. Read these fields:

| Field | Meaning |
| --- | --- |
| `cache_hits` / `cache_misses` / `hit_rate` | Whether later requests were served from cache |
| `cache_coalesced` | Waiters on an in-flight identical miss share one upstream fill (`X-LanCache-Status: HIT-COALESCED`) |
| `negative_cache_hits` | 404/410 served from a still-valid negative cache entry (also counted in `cache_hits`) |
| `range_cache` / `range_upstream` | Range GETs served as 206 from a cached object vs after a full upstream fetch |
| `memory_cache_hits` / `disk_cache_hits` | Which tier served the hits |
| `total_requests` / `errors` | Volume and failures |
| `upstream_errors` / `cache_write_failures` / `rate_limited` | Upstream pipe, cache write, and rate-limit pressure (Quick check highlights these next to hit/miss) |
| `disk_tier_ready` | `0` while disk slow-tier attach pending; `1` when attached, or when no disk configured (N/A — not waiting) |
| `memory_cache_size` / `disk_cache_size` | Current cache occupancy per tier (bytes) |
| `memory_cache_capacity` / `disk_cache_capacity` | Configured capacity per tier (bytes); `disk_cache_capacity` is `0` when no disk is configured |
| `disk_cache_full_ratio` | `disk_cache_size / disk_cache_capacity` in [0,1]; 0 when no disk is configured or capacity is 0. Tells you "95% full" vs "barely filled" without reading the filesystem |
| `capacity_pressure_events` | Soft eviction under the memory or disk cap, and/or disk Create/Write/Mkdir hitting ENOSPC (volume full). Distinct from cold-cache misses and from the existing `evictions` counter. Logs `tier` (memory or disk) and `reason` (eviction or enospc). |

A first pass through new content is mostly misses (`hit_rate` near 0). Repeat the same content and `cache_hits` / `hit_rate` should rise.

Cache entries are keyed by depot object path (not the CDN `Host` header), so when Steam rotates CDN hostnames for the same depot path, hits still climb across the aliases.

Concurrent identical misses for the same key share one upstream GET: the leader is a `MISS` and waiters are `HIT-COALESCED` (`cache_coalesced`).

Steam clients lean on Range requests. When an object is already cached, a Range GET is served locally as 206 from that full object (`range_cache`). On a Range miss the cache still fetches and stores the full upstream body, then returns the requested byte range as 206 (`range_upstream`).

Definitive upstream 404/410 (gone depot objects) are stored as a short-TTL negative entry in the **same** cache, under the same depot-path key as a positive object. Repeating the request within `cache.negative_ttl` (default `5m`) is served as 404/410 without re-hitting upstream (`negative_cache_hits`). 5xx is not cached as negative. When the TTL expires the entry is deleted and the next request fetches again.

To confirm the process is up (HTTP 204 and `X-LanCache-Processed-By: SteamCache2`):

```bash
curl -s -i http://localhost/lancache-heartbeat
```

Use GET (`curl -i`), not HEAD (`curl -I`): the server only accepts GET.

Heartbeat also returns `X-SteamCache-Disk-Tier: pending|ready|disabled` (`disabled` = memory-only / no disk; `pending`/`ready` = disk configured attach state).

These are the cache process's own `/metrics` and `/lancache-heartbeat` endpoints. There is no separate metrics daemon.

`/metrics` is Prometheus text exposition format 0.0.4 (`Content-Type: text/plain; version=0.0.4; charset=utf-8`) so Prometheus and compatible scrapers can pull it. Metric names in the table above are unchanged; each series is preceded by `# HELP` and `# TYPE`.

If you changed `listen_address`, point curl at that host:port instead. For a full SteamPrefill validation workflow (small caches, coalescing, GC), see [Validating Full Functionality](#validating-full-functionality-with-external-tools).

### Development Workflow

Use `make` for the majority of common development tasks. The Makefile handles running tests, linting, hygiene checks, building, running the application, and other routine boilerplate work.

Run `make help` to see the full list of available commands.

This is the preferred approach for day-to-day development. Avoid running raw `go test`, `go run`, or `golangci-lint` commands directly for routine tasks.

### Validating Full Functionality with external tools

steamcache2 provides a convenient small-cache configuration and helper targets so you can easily validate behavior using external tools such as [SteamPrefill (tpill90/steam-lancache-prefill)](https://github.com/tpill90/steam-lancache-prefill).

This gives you:
- Real Steam manifest + chunk traffic (no reinventing the wheel)
- Excellent `benchmark setup` / `benchmark run` workflow with warmup, randomization, and mixed chunk sizes
- The ability to validate a **just-built binary** end-to-end (caching, coalescing, Range support, memory+disk tiers, GC/eviction, metrics, special endpoints, startup validation, etc.)

#### Validation server (recommended)

For easy validation with external tools (SteamPrefill, etc.), use:

```bash
make run-validation
# or
make validate
```

This starts `steamcache2` on port 80 using a deliberately small memory + disk configuration (good for exercising the disk tier, GC, coalescing, promotions, etc.).

`make run-validation` (and `make validate`) will automatically ensure the `cap_net_bind_service` capability is set on the binary it just built (one sudo prompt the first time after each rebuild). This keeps the server running as your normal user so the disk cache directory stays owned by you.

If you want the capability on the binary for other workflows (e.g. `make run`, or running the binary directly on port 80), use the explicit target:

```bash
make setcap
```

When the server is running, point your external SteamPrefill (or other load generator) at it:

```bash
./SteamPrefill benchmark run ...
```

When finished, you can get a quick metrics + heartbeat report with:

```bash
make validate-check
# or, manually:
curl -s http://localhost/metrics
curl -s -i http://localhost/lancache-heartbeat
```

See [Quick check: is it caching?](#quick-check-is-it-caching) for which fields to read. This is the recommended simple workflow. No automatic downloading or running of external tools.

#### Inspecting the Result

After a benchmark run you can ask for a quick report:

```bash
make validate-check
# or, manually:
curl -s http://localhost/metrics
curl -s -i http://localhost/lancache-heartbeat
```

`make validate-check` prints the full `/metrics` dump, highlights hit/miss fields, and curls `/lancache-heartbeat`. It also asserts a non-Steam Host is rejected with HTTP 400 when upstream is empty. Look for:
- High cache hit rate after the warmup pass (`cache_hits`, `hit_rate`, plus `memory_cache_hits` / `disk_cache_hits`)
- Non-zero `coalesced` and `disk` activity
- Zero unexpected `errors`, and quiet `upstream_errors` / `cache_write_failures` / `rate_limited`

Heartbeat should be HTTP 204 with `X-LanCache-Processed-By: SteamCache2`. Use GET (`curl -i`), not HEAD (`curl -I`).

#### The Validation Config

The recommended validation config is at [docs/examples/validate-config.yaml](docs/examples/validate-config.yaml). It enables both memory and disk tiers at modest sizes (128 MB / 512 MB) with conservative concurrency. Edit or copy it if you need larger caches for bigger workloads.

#### What Gets Validated

Running a realistic SteamPrefill benchmark workload through a built steamcache2 exercises the complete public surface that matters for production use:
- Steam User-Agent detection and depot/manifest/chunk URL patterns
- Full MISS → cache write → HIT (and HIT-COALESCED) paths
- Range request handling from cached full responses (local 206 on HIT via `range_cache`; MISS fetches the full object then serves the requested slice as 206 via `range_upstream`)
- Request coalescing under concurrent load
- Memory tier + disk tier interaction (including async disk attach)
- Garbage collection and eviction under pressure
- Metrics and special endpoints (`/`, `/lancache-heartbeat`, `/metrics`)
- Per-client and global rate limiting (with trusted proxy handling)
- Startup configuration validation and upstream behavior
- Clean shutdown hygiene

This is the closest practical equivalent to "run the thing real clients will run and make sure nothing is broken."

#### Troubleshooting

- **Low hit rate on first run**: Normal. The first `benchmark run` is the warmup that populates the cache.
- **Want to test real disk I/O (not RAM cache)**: Make sure your workload size (shown by `benchmark setup`) is larger than the total RAM on the machine running steamcache2.
- **Server won't start or bind on port 80 as non-root**: `make run-validation` and `make validate` automatically run `setcap` on the binary they just built. If it still fails, run `make setcap` explicitly and retry. The server always runs as your normal user (no root) so the disk cache directory ownership stays correct.
- **SteamPrefill not found**: Install it yourself from its GitHub releases. Then use `make validate` to start the server with small caches and point SteamPrefill at it manually.
- **SteamPrefill won't use server as cache properly**: SteamPrefill has some bad autodetectiong functions sometimes it works when the server is resolvable from localhost or 127.0.0.1 other times you have to fully override the dns for the proper dns name lancache.steamcontent.com to point to 127.0.0.1 i don't recommend doing it unless your okay with having to undo and redo it depending on if your running the server or not its a pain.

See also the SteamPrefill documentation for `benchmark setup` and `benchmark run` options.

### Command Line Flags

While most configuration is done via the YAML file, some runtime options are still available as command-line flags:

```bash
# Use a custom config file
./steamcache2 --config /path/to/my-config.yaml

# Set logging level
./steamcache2 --log-level debug --log-format json

# Override concurrency from the CLI (0 = use config.yaml)
./steamcache2 --max-concurrent-requests 8
./steamcache2 --max-requests-per-client 4

# Table-tier uplink shaping (empty/0 = use config / disabled)
./steamcache2 --uplink-bandwidth 10MB
./steamcache2 --max-bytes-per-client-per-sec 2500000

# Show help
./steamcache2 --help
```

### Configuration

SteamCache2 uses a YAML configuration file (`config.yaml`) for all settings. Here's a complete configuration example:

```yaml
# Server configuration
listen_address: :80

# Hardening (see Security Hardening section)
max_object_size: "0"   # 0=unlimited; set e.g. "256MB" for response size DoS protection
trusted_proxies: []    # empty = safe (ignore XFF for rate limit); set CIDRs for trusted proxies

# Table-tier uplink bandwidth shaping (bytes/sec). Empty/0 = disabled (unlimited).
# Distinct from max_requests_per_client (concurrency). See "Table-tier uplink fair-share".
uplink_bandwidth: ""                 # e.g. "10MB" = 10e6 bytes/sec shared fairly across active clients
max_bytes_per_client_per_sec: 0      # optional absolute per-client cap; 0 = no absolute cap

# Cache configuration
cache:
  # Memory cache settings
  memory:
    # Size of memory cache (e.g., "512MB", "1GB", "0" to disable)
    size: 1GB
    # Garbage collection algorithm
    gc_algorithm: lru

  # Disk cache settings
  disk:
    # Size of disk cache (e.g., "10GB", "50GB", "0" to disable)
    size: 10GB
    # Path to disk cache directory
    path: ./disk
    # Garbage collection algorithm
    gc_algorithm: hybrid

  # Short TTL for cached 404/410 (gone depot objects). Default 5m.
  # Same VFS cache and depot-path key as positive objects. Does not cache 5xx.
  negative_ttl: 5m

# Upstream server configuration
# Leave empty to fetch from the request Host (Steam CDN names only).
# Set only when chaining caches (table RAM cache -> room disk cache).
upstream: ""
```

#### Startup Validation
`steamcache2` performs strict validation on startup (after loading config + CLI overrides, before creating the cache). Invalid configs cause immediate clean failure (no default written, no panic):

- Negative `max_concurrent_requests` / `max_requests_per_client`: "negative concurrency not allowed"
- Invalid `gc_algorithm` (memory): "invalid memory gc algorithm: badvalue"
- Disk enabled (`size` non-zero/"") but no `path`: "disk cache enabled but no path specified"
- Invalid memory/disk `size` strings (via direct New): "invalid memory size: ..." / "invalid disk size: ..." (clean error return, no panic)

Example error on stderr + logs:
```
Error: Invalid configuration: invalid memory gc algorithm: foo. Please fix the config file and try again.
```

See `config.Validate()` and `steamcache.New` error paths. This ensures the LAN appliance fails fast on misconfig.

#### Security Hardening
- `max_object_size` (default "0" = unlimited): set e.g. "256MB" or "512MB" to reject oversized upstream responses with HTTP 413 before buffering/ReadAll. Prevents OOM DoS from large or malicious responses. Large legitimate Steam files still served if under limit.
- `trusted_proxies`: CIDR list (default empty). When empty (safe default), X-Forwarded-For and client IP spoofing are ignored for rate limiting — always uses `r.RemoteAddr` only. When set (e.g. your reverse proxy CIDR), uses correct "rightmost untrusted" extraction. Prevents bypass of `max_requests_per_client`. Documented for LAN proxy setups only.
- These + the startup validation make steamcache2 safe-by-default for LAN exposure.

#### Migration / Breaking Changes
- `New()` public signature gained trailing params (`maxObjectSize`, `trustedProxies`, `negativeTTL`). Direct callers (rare; most use config or NewWithOptions) must update. Empty `negativeTTL` means 5m.
- Recommended: migrate to `NewWithOptions(Options{...})` (non-breaking) or rely on YAML config + cmd/root.go.
- No behavior change for existing configs (defaults preserve prior semantics; `cache.negative_ttl` defaults to 5m).

#### Large Cache Initialization (async DiskFS population)
- `disk.New(root, capacity, evictFn)` signature changed (now takes evict func from `gc.GetGCAlgorithm`, returns error for ctor hygiene). Callers updated internally; direct vfs/disk users must pass the evict (or nil for no startup guard).
- DiskFS initialization is now fully asynchronous for large caches (millions of files): `New` returns immediately without scanning. The first `Size()` (and many internal callers) blocks on an internal barrier until bg streaming population + any startup over-cap eviction (using the evictFn) completes. Subsequent `Size()` calls are instant.
- During the "proxy window" (while bg scan runs): disk-only configs (memory.size=0) have TieredCache Create returning `ErrNotFound` (no disk writes/caching occurs until attach); mem+disk configs serve from memory tier only. This keeps `New` fast and avoids heavy disk I/O/eviction during long scans on slow storage.
- The explicit startup guard (reduce size if pre-existing on-disk > cap) runs as the literal last step of bg init, before the barrier opens.
- Add a note for operators: very large disk caches (tens/hundreds GB with millions files) may show extended "memory-only or no-cache" behavior at startup (seconds to minutes depending on storage speed); this is by design for responsiveness.
- Godoc on `disk.New` and `DiskFS.Size` expanded with the barrier/attach behavior.
- Startup logs: Info "Disk slow tier attach pending..." then later "Disk slow tier attached (...)" for disk-only and mixed modes.
- `/metrics` exposes `disk_tier_ready` 0/1 and stays responsive during attach (GetMetrics does not block on Size while pending).
- `/metrics` tier occupancy: `memory_cache_size` / `disk_cache_size` (bytes in use) next to `memory_cache_capacity` / `disk_cache_capacity` (configured capacity; `disk_cache_capacity` is 0 when no disk is configured), plus `disk_cache_full_ratio` (size/capacity in [0,1]). Capacity is a config read, so it is reported even while the disk attach is pending (size stays 0 until attach).
- `/lancache-heartbeat` header `X-SteamCache-Disk-Tier` mirrors that state.
- `/metrics` `capacity_pressure_events` counts times the cache dropped data under capacity pressure (soft eviction at the memory or disk cap, or disk Create/Write/Mkdir returning ENOSPC). Logs include `tier=memory|disk` and `reason=eviction|enospc` so operators can grep and tell this apart from a cold cache. The existing `evictions` counter is unchanged.

#### Garbage Collection Algorithms

SteamCache2 supports different garbage collection algorithms for memory and disk caches, allowing you to optimize performance for each storage tier:

**Available GC Algorithms:**

- **`lru`** (default): Least Recently Used - evicts oldest accessed files
- **`lfu`**: Least Frequently Used - evicts by lowest AccessCount (tiebreak older ATime); uses existing FileInfo counters
- **`fifo`**: First In, First Out - evicts oldest created files (predictable and terrible all in one) don't ever use it
- **`largest`**: Size-based - evicts largest files first (maximizes small file count) if used on memory greatly improves access time
- **`smallest`**: Size-based - evicts smallest files first (maximizes large file count) probably best used for disk since there kinda slow with small files
- **`hybrid`**: Recency + frequency hybrid - evicts by lowest time-decayed score (GetTimeDecayedScore combining ATime + AccessCount)

**Recommended Algorithms by Cache Type:**

**For Memory Cache (Fast, Limited Size):**
- **`lru`** - Best overall performance, good balance of speed and hit rate
- **`lfu`** - Excellent for gaming cafes where popular games stay cached
- **`hybrid`** - Optimal for mixed workloads with varying file sizes
- **`largest`** - Crazy good for access times since disks are slow with lots of tiny files

**For Disk Cache (Slow, Large Size):**
- **`hybrid`** - Recommended for optimal performance, balances speed and storage efficiency
- **`smallest`** - Good for maximizing linear reads which is the only place spinning disks have performance don't expect too much though steam kinda uses small files
- **`lru`** - Reliable default with good performance

**Use Cases:**
- **Gaming Cafes**: Use `largest` for memory, `hybrid` for disk
- **LAN Events**: Use `largest` for memory, `hybrid` for disk
- **Home Use**: Use `largest` for memory, `hybrid` for disk
- **Testing**: Use `fifo` for nothing its pointless
- **Large File Storage**: Use `smallest` for disk get rid of the slow tiny files first

### DNS Configuration

Configure your DNS to direct Steam traffic to your SteamCache2 server:

- If you're on Windows and don't want a whole network implementation, see the [Windows Hosts File Override](#windows-hosts-file-override) section below.

### Windows Hosts File Override

1. Open Notepad as Administrator:
    - Click on the Start menu, type `Notepad`, right-click on Notepad, and select `Run as administrator`.

2. Open the Hosts File:
    - In Notepad, go to `File` > `Open`.
    - Navigate to `C:\Windows\System32\drivers\etc`.
    - Select `All Files` from the dropdown menu to see the hosts file.
    - Open the `hosts` file.

3. Add the Override Entry:
    - At the end of the file, add a new line with the IP address of your SteamCache2 server followed by `lancache.steamcontent.com`. For example:
      ```plaintext
      192.168.1.100 lancache.steamcontent.com
      ```
      Replace `192.168.1.100` with the actual IP address of your SteamCache2 server.

4. Save the Hosts File:
    - Save the changes by going to `File` > `Save`.

5. Flush DNS Cache (optional but recommended):
    - Open Command Prompt as Administrator.
    - Run the following command to flush the DNS cache:
      ```sh
      ipconfig /flushdns
      ```

6. Restart
    - Restart Steam or Restart Your PC

This will direct any requests to `lancache.steamcontent.com` to your SteamCache2 server.

## Building from Source

### Prerequisites

- Go 1.27.0 or later
- Make (optional, but recommended)

### Build Commands

```bash
# Clone the repository
git clone <repository-url>
cd steamcache2

# Download dependencies
make deps

# Run tests
make test

# Build for current platform
go build -o steamcache2 .

# Build for specific platforms
GOOS=linux GOARCH=amd64 go build -o steamcache2-linux-amd64 .
GOOS=windows GOARCH=amd64 go build -o steamcache2-windows-amd64.exe .
```

### Development

```bash
# Run in development mode with debug logging
make run-debug

# Run all tests and start the application
make
```

## Troubleshooting

### Common Issues

1. **"Config file not found" on first run**
   - This is expected! SteamCache2 will automatically create a default `config.yaml` file
   - Edit the generated config file with your desired settings
   - Run the application again

2. **Permission denied when creating config**
   - Make sure you have write permissions in the current directory
   - Try running with elevated privileges if necessary

3. **Port already in use**
   - Change the `listen_address` in `config.yaml` to a different port (e.g., `:8080`)
   - Or stop the service using the current port

4. **High memory usage**
   - Reduce the memory cache size in `config.yaml`
   - Consider using disk-only caching by setting `memory.size: "0"`

5. **Slow disk performance**
   - Use SSD storage for the disk cache
   - Consider using a different GC algorithm like `hybrid`
   - Adjust the disk cache size to match available storage

6. **Not sure if it is caching**
   - Do not start with the full SteamPrefill chapter. Use [Quick check: is it caching?](#quick-check-is-it-caching): `make validate-check` (full `/metrics`, hit/miss fields, and `/lancache-heartbeat`)
   - A first pass is mostly `cache_misses`; repeating the same content should raise `cache_hits` / `hit_rate`
   - Confirm the process is up with `curl -s -i http://localhost/lancache-heartbeat` (GET, not HEAD)

### Getting Help

- Check the logs for detailed error messages
- Run with `--log-level debug` for more verbose output
- Ensure your upstream server is accessible
- Verify DNS configuration is working correctly

## License

See the [LICENSE](LICENSE) file for details.
But just for clarity this covers all files in this project unless stated in the individual file.

## Acknowledgements

- Inspired by [Lancache.net](https://lancache.net/)