jiggablend/README.md

# JiggaBlend - Blender Render Farm

A distributed Blender render farm system built with Go. The system consists of a manager server that handles job submission, file storage, and runner coordination, and runner clients that execute Blender renders on Linux amd64 systems.

## Architecture

- **Manager**: Central server with REST API, embedded web UI, SQLite database, and local file storage
- **Runner**: Linux amd64 client that connects to manager, receives jobs, executes Blender renders, and reports back

Both manager and runner are part of a single binary (`jiggablend`) with subcommands.

## Features

- **Authentication**: OAuth (Google and Discord) and local authentication with user management
- **Web UI**: Server-rendered Go templates with HTMX fragments for job submission and monitoring
- **Distributed Rendering**: Scale across multiple runners with automatic job distribution
- **Real-time Updates**: Polling-based UI updates with lightweight HTMX refreshes
- **Video Encoding**: Automatic video encoding from EXR sequences only. EXR→video always uses HDR (HLG, 10-bit); no option to disable. Codecs:
  - H.264 (MP4) - HDR (HLG)
  - AV1 (MP4) - Alpha channel support, HDR
  - VP9 (WebM) - Alpha channel and HDR
- **Output Formats**: EXR frame sequence only, or EXR + video (H.264, AV1, VP9). Blender always renders EXR.
- **Blender Version Management**: Support for multiple Blender versions with automatic detection
- **Metadata Extraction**: Automatic extraction of scene metadata from Blender files
- **Admin Panel**: User and runner management interface
- **Runner Management**: API key-based authentication for runners with health monitoring
- **HDR**: EXR→video is always encoded as HDR (HLG, 10-bit). There is no option to turn it off; for SDR-only output, download the EXR frames and encode locally.
- **Alpha**: Alpha is always preserved in EXR frames. In video, alpha is preserved when present in the EXR for AV1 and VP9; H.264 MP4 does not support alpha.

## Prerequisites

### Manager
- Go 1.25.4 or later
- SQLite (via Go driver)
- Blender installed and in PATH (for metadata extraction)
- ImageMagick installed (for EXR preview conversion)

### Runner
- Linux amd64
- FFmpeg installed (required for video encoding)
- Able to run Blender (the runner gets the job’s required Blender version from the manager; it does not need Blender pre-installed)

## Installation

1. Clone the repository:
```bash
git clone <repository-url>
cd jiggablend
```

2. Install dependencies:
```bash
go mod download
```

## Configuration

### Manager

Configuration is managed through the CLI using `jiggablend manager config` commands. The configuration is stored in the SQLite database.

#### Initial Setup

For testing, use the Makefile helper:
```bash
make init-test
```

This will:
- Enable local authentication
- Set a fixed API key for testing: `jk_r0_test_key_123456789012345678901234567890`
- Create a test admin user (test@example.com / testpassword)

#### Manual Configuration

```bash
# Enable local authentication
jiggablend manager config enable localauth

# Add a user
jiggablend manager config add user <email> <password> --admin

# Generate an API key for runners
jiggablend manager config add apikey <name> --scope manager

# Set OAuth credentials
jiggablend manager config set google-oauth <client-id> <client-secret> --redirect-url <url>
jiggablend manager config set discord-oauth <client-id> <client-secret> --redirect-url <url>

# View current configuration
jiggablend manager config show

# List users and API keys
jiggablend manager config list users
jiggablend manager config list apikeys
```

#### Environment Variables

You can also use environment variables with the `JIGGABLEND_` prefix:
- `JIGGABLEND_PORT` - Server port (default: 8080)
- `JIGGABLEND_DB` - Database path (default: jiggablend.db)
- `JIGGABLEND_STORAGE` - Storage path (default: ./jiggablend-storage)
- `JIGGABLEND_LOG_FILE` - Log file path
- `JIGGABLEND_LOG_LEVEL` - Log level (debug, info, warn, error)
- `JIGGABLEND_VERBOSE` - Enable verbose logging

### Runner

The runner requires an API key to connect to the manager. The runner will auto-detect hostname and IP.

## Usage

### Building

```bash
# Build the unified binary (includes embedded web UI)
make build

# Or build directly
go build -o bin/jiggablend ./cmd/jiggablend

# Build web UI separately
make build-web
```

### Running the Manager

```bash
# Using make (includes test setup)
make run-manager

# Or directly
bin/jiggablend manager

# With custom options
bin/jiggablend manager --port 8080 --db jiggablend.db --storage ./jiggablend-storage --log-file manager.log

# Using environment variables
JIGGABLEND_PORT=8080 JIGGABLEND_DB=jiggablend.db bin/jiggablend manager
```

The manager will start on `http://localhost:8080` by default.

### Running a Runner

```bash
# Using make (uses test API key)
make run-runner

# Or directly (requires API key)
bin/jiggablend runner --api-key <your-api-key>

# With custom options
bin/jiggablend runner --manager http://localhost:8080 --name my-runner --api-key <key> --log-file runner.log

# Hardware compatibility flags (force CPU + disable HIPRT)
bin/jiggablend runner --api-key <key> --force-cpu-rendering --disable-hiprt

# Using environment variables
JIGGABLEND_MANAGER=http://localhost:8080 JIGGABLEND_API_KEY=<key> bin/jiggablend runner
```

### Render Chunk Size Note

For one heavy production scene/profile, chunked rendering (`frames 800-804` in one Blender process) was much slower than one-frame tasks:

- Chunked task (`800-804`): `27m49s` end-to-end (`Task assigned` -> last `Saved`)
- Single-frame tasks (`800`, `801`, `802`, `803`, `804`): `15m04s` wall clock total

In that test, any chunk size greater than `1` caused a major slowdown after the first frame. Fresh installs should already have it set to `1`, but if you see similar performance degradation, try forcing one frame per task (hard reset Blender each frame): `jiggablend manager config set frames-per-render-task 1`. If `1` is worse on your scene/hardware, benchmark and use a higher chunk size instead.

### Running Both (for Testing)

```bash
# Run manager and runner in parallel
make run
```

This will start both the manager and a test runner with a fixed API key.

## OAuth Setup

### Google OAuth

1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project or select existing
3. Enable Google+ API
4. Create OAuth 2.0 credentials
5. Add authorized redirect URI: `http://localhost:8080/api/auth/google/callback`
6. Configure using CLI:
   ```bash
   jiggablend manager config set google-oauth <client-id> <client-secret> --redirect-url http://localhost:8080/api/auth/google/callback
   ```

### Discord OAuth

1. Go to [Discord Developer Portal](https://discord.com/developers/applications)
2. Create a new application
3. Go to OAuth2 section
4. Add redirect URI: `http://localhost:8080/api/auth/discord/callback`
5. Configure using CLI:
   ```bash
   jiggablend manager config set discord-oauth <client-id> <client-secret> --redirect-url http://localhost:8080/api/auth/discord/callback
   ```

## Project Structure

```
jiggablend/
├── cmd/
│   └── jiggablend/       # Unified CLI application
│       ├── cmd/          # Cobra command definitions
│       └── main.go       # Entry point
├── internal/
│   ├── auth/             # Authentication (OAuth, local, sessions)
│   ├── config/           # Configuration management
│   ├── database/         # SQLite database models and migrations
│   ├── logger/           # Logging utilities
│   ├── manager/          # Manager server logic
│   ├── runner/           # Runner client logic
│   │   ├── api/          # Manager API client
│   │   ├── blender/      # Blender version detection
│   │   ├── encoding/     # Video encoding (H.264, AV1, VP9)
│   │   ├── tasks/        # Task execution (render, encode, process)
│   │   └── workspace/    # Workspace management
│   └── storage/          # File storage operations
├── pkg/
│   ├── executils/        # Execution utilities
│   ├── scripts/          # Python scripts for Blender
│   └── types/             # Shared types and models
├── web/                   # Embedded templates + static assets
│   ├── templates/        # Go HTML templates and partials
│   └── static/           # CSS/JS assets
├── go.mod
└── Makefile
```

## API Endpoints

### Authentication
- `GET /api/auth/google/login` - Initiate Google OAuth
- `GET /api/auth/google/callback` - Google OAuth callback
- `GET /api/auth/discord/login` - Initiate Discord OAuth
- `GET /api/auth/discord/callback` - Discord OAuth callback
- `POST /api/auth/login` - Local authentication login
- `POST /api/auth/register` - User registration (if enabled)
- `POST /api/auth/logout` - Logout
- `GET /api/auth/me` - Get current user
- `POST /api/auth/password/change` - Change password

### Jobs
- `POST /api/jobs` - Create a new job
- `GET /api/jobs` - List user's jobs
- `GET /api/jobs/{id}` - Get job details
- `DELETE /api/jobs/{id}` - Cancel a job
- `POST /api/jobs/{id}/upload` - Upload job file (Blender file)
- `GET /api/jobs/{id}/files` - List job files
- `GET /api/jobs/{id}/files/{fileId}/download` - Download job file
- `GET /api/jobs/{id}/metadata` - Extract metadata from uploaded file
- `GET /api/jobs/{id}/outputs` - List job output files

### Blender
- `GET /api/blender/versions` - List available Blender versions

### Runners (Internal API)
- `POST /api/runner/register` - Register a runner (uses API key)
- `POST /api/runner/heartbeat` - Update runner heartbeat
- `GET /api/runner/tasks` - Get pending tasks for runner
- `POST /api/runner/tasks/{id}/complete` - Mark task as complete
- `GET /api/runner/files/{jobId}/{fileName}` - Download file for runner
- `POST /api/runner/files/{jobId}/upload` - Upload file from runner

### Admin (Admin Only)
- `GET /api/admin/runners` - List all runners
- `GET /api/admin/jobs` - List all jobs
- `GET /api/admin/users` - List all users
- `GET /api/admin/stats` - System statistics

### WebSocket
- `WS /api/jobs/ws` - Optional API channel for advanced clients
- The default web UI uses polling + HTMX for status updates and task views.

## Output Formats

The system supports the following output formats. Blender always renders EXR (linear); the chosen format is the deliverable (frames only or frames + video).

### Deliverable Formats
- **EXR** - EXR frame sequence only (no video)
- **EXR_264_MP4** - EXR frames + H.264 MP4 (always HDR, HLG)
- **EXR_AV1_MP4** - EXR frames + AV1 MP4 (alpha support, always HDR)
- **EXR_VP9_WEBM** - EXR frames + VP9 WebM (alpha and HDR)

Video encoding (EXR→video) is always HDR (HLG, 10-bit); there is no option to output SDR video. For SDR-only, download the EXR frames and encode locally.

Video encoding features:
- 2-pass encoding for optimal quality
- EXR→video only (no PNG source); always HLG (HDR), 10-bit, full range
- Alpha channel preservation (AV1 and VP9 only)
- Software encoding (libx264, libaom-av1, libvpx-vp9)

## Storage Structure

The manager uses a local storage directory (default: `./jiggablend-storage`) with the following structure:

```
jiggablend-storage/
├── blender-versions/     # Bundled Blender versions
│   └── <version>/
├── jobs/                 # Job context files
│   └── <job-id>/
│       └── context.tar
├── outputs/              # Rendered outputs
│   └── <job-id>/
├── temp/                 # Temporary files
└── uploads/              # Uploaded files
```

## Development

### Running Tests

```bash
make test
# Or directly
go test ./... -timeout 30s
```

### Web UI Development

The web UI is server-rendered from embedded templates and static assets in `web/templates` and `web/static`.
No Node/Vite build step is required.

## License

MIT