jiggablend/README.md

# JiggaBlend - Blender Render Farm

A distributed Blender render farm system built with Go. The system consists of a manager server that handles job submission, file storage, and runner coordination, and runner clients that execute Blender renders on Linux amd64 systems.

## Architecture

- **Manager**: Central server with REST API, embedded web UI, SQLite database, and local file storage
- **Runner**: Linux amd64 client that connects to manager, receives jobs, executes Blender renders, and reports back

Both manager and runner are part of a single binary (`jiggablend`) with subcommands.

## Features

- **Authentication**: OAuth (Google and Discord) and local authentication with user management
- **Web UI**: Modern React-based interface for job submission and monitoring
- **Distributed Rendering**: Scale across multiple runners with automatic job distribution
- **Real-time Updates**: WebSocket-based progress tracking and job status updates
- **Video Encoding**: Automatic video encoding from EXR/PNG sequences with multiple codec support:
  - H.264 (MP4) - SDR and HDR support
  - AV1 (MP4) - With alpha channel support
  - VP9 (WebM) - With alpha channel and HDR support
- **Output Formats**: PNG, JPEG, EXR, and video formats (MP4, WebM)
- **Blender Version Management**: Support for multiple Blender versions with automatic detection
- **Metadata Extraction**: Automatic extraction of scene metadata from Blender files
- **Admin Panel**: User and runner management interface
- **Runner Management**: API key-based authentication for runners with health monitoring
- **HDR Support**: Preserve HDR range in video encoding with HLG transfer function
- **Alpha Channel**: Preserve alpha channel in video encoding (AV1 and VP9)

## Prerequisites

### Manager
- Go 1.25.4 or later
- SQLite (via Go driver)
- Blender installed and in PATH (for metadata extraction)
- ImageMagick installed (for EXR preview conversion)

### Runner
- Linux amd64
- Blender installed (can use bundled versions from storage)
- FFmpeg installed (required for video encoding)

## Installation

1. Clone the repository:
```bash
git clone <repository-url>
cd jiggablend
```

2. Install dependencies:
```bash
go mod download
```

## Configuration

### Manager

Configuration is managed through the CLI using `jiggablend manager config` commands. The configuration is stored in the SQLite database.

#### Initial Setup

For testing, use the Makefile helper:
```bash
make init-test
```

This will:
- Enable local authentication
- Set a fixed API key for testing
- Create a test admin user (test@example.com / testpassword)

#### Manual Configuration

```bash
# Enable local authentication
jiggablend manager config enable localauth

# Add a user
jiggablend manager config add user <email> <password> --admin

# Generate an API key for runners
jiggablend manager config add apikey <name> --scope manager

# Set OAuth credentials
jiggablend manager config set google-oauth <client-id> <client-secret> --redirect-url <url>
jiggablend manager config set discord-oauth <client-id> <client-secret> --redirect-url <url>

# View current configuration
jiggablend manager config show

# List users and API keys
jiggablend manager config list users
jiggablend manager config list apikeys
```

#### Environment Variables

You can also use environment variables with the `JIGGABLEND_` prefix:
- `JIGGABLEND_PORT` - Server port (default: 8080)
- `JIGGABLEND_DB` - Database path (default: jiggablend.db)
- `JIGGABLEND_STORAGE` - Storage path (default: ./jiggablend-storage)
- `JIGGABLEND_LOG_FILE` - Log file path
- `JIGGABLEND_LOG_LEVEL` - Log level (debug, info, warn, error)
- `JIGGABLEND_VERBOSE` - Enable verbose logging

### Runner

The runner requires an API key to connect to the manager. The runner will auto-detect hostname and IP.

## Usage

### Building

```bash
# Build the unified binary (includes embedded web UI)
make build

# Or build directly
go build -o bin/jiggablend ./cmd/jiggablend

# Build web UI separately
make build-web
```

### Running the Manager

```bash
# Using make (includes test setup)
make run-manager

# Or directly
bin/jiggablend manager

# With custom options
bin/jiggablend manager --port 8080 --db jiggablend.db --storage ./jiggablend-storage --log-file manager.log

# Using environment variables
JIGGABLEND_PORT=8080 JIGGABLEND_DB=jiggablend.db bin/jiggablend manager
```

The manager will start on `http://localhost:8080` by default.

### Running a Runner

```bash
# Using make (uses test API key)
make run-runner

# Or directly (requires API key)
bin/jiggablend runner --api-key <your-api-key>

# With custom options
bin/jiggablend runner --manager http://localhost:8080 --name my-runner --api-key <key> --log-file runner.log

# Using environment variables
JIGGABLEND_MANAGER=http://localhost:8080 JIGGABLEND_API_KEY=<key> bin/jiggablend runner
```

### Running Both (for Testing)

```bash
# Run manager and runner in parallel
make run
```

This will start both the manager and a test runner with a fixed API key.

## OAuth Setup

### Google OAuth

1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project or select existing
3. Enable Google+ API
4. Create OAuth 2.0 credentials
5. Add authorized redirect URI: `http://localhost:8080/api/auth/google/callback`
6. Configure using CLI:
   ```bash
   jiggablend manager config set google-oauth <client-id> <client-secret> --redirect-url http://localhost:8080/api/auth/google/callback
   ```

### Discord OAuth

1. Go to [Discord Developer Portal](https://discord.com/developers/applications)
2. Create a new application
3. Go to OAuth2 section
4. Add redirect URI: `http://localhost:8080/api/auth/discord/callback`
5. Configure using CLI:
   ```bash
   jiggablend manager config set discord-oauth <client-id> <client-secret> --redirect-url http://localhost:8080/api/auth/discord/callback
   ```

## Project Structure

```
jiggablend/
├── cmd/
│   └── jiggablend/       # Unified CLI application
│       ├── cmd/          # Cobra command definitions
│       └── main.go       # Entry point
├── internal/
│   ├── auth/             # Authentication (OAuth, local, sessions)
│   ├── config/           # Configuration management
│   ├── database/         # SQLite database models and migrations
│   ├── logger/           # Logging utilities
│   ├── manager/          # Manager server logic
│   ├── runner/           # Runner client logic
│   │   ├── api/          # Manager API client
│   │   ├── blender/      # Blender version detection
│   │   ├── encoding/     # Video encoding (H.264, AV1, VP9)
│   │   ├── tasks/        # Task execution (render, encode, process)
│   │   └── workspace/    # Workspace management
│   └── storage/          # File storage operations
├── pkg/
│   ├── executils/        # Execution utilities
│   ├── scripts/          # Python scripts for Blender
│   └── types/             # Shared types and models
├── web/                   # React web UI
│   ├── src/              # Source files
│   └── dist/             # Built files (embedded in binary)
├── go.mod
└── Makefile
```

## API Endpoints

### Authentication
- `GET /api/auth/google/login` - Initiate Google OAuth
- `GET /api/auth/google/callback` - Google OAuth callback
- `GET /api/auth/discord/login` - Initiate Discord OAuth
- `GET /api/auth/discord/callback` - Discord OAuth callback
- `POST /api/auth/login` - Local authentication login
- `POST /api/auth/register` - User registration (if enabled)
- `POST /api/auth/logout` - Logout
- `GET /api/auth/me` - Get current user
- `POST /api/auth/password/change` - Change password

### Jobs
- `POST /api/jobs` - Create a new job
- `GET /api/jobs` - List user's jobs
- `GET /api/jobs/{id}` - Get job details
- `DELETE /api/jobs/{id}` - Cancel a job
- `POST /api/jobs/{id}/upload` - Upload job file (Blender file)
- `GET /api/jobs/{id}/files` - List job files
- `GET /api/jobs/{id}/files/{fileId}/download` - Download job file
- `GET /api/jobs/{id}/metadata` - Extract metadata from uploaded file
- `GET /api/jobs/{id}/outputs` - List job output files

### Blender
- `GET /api/blender/versions` - List available Blender versions

### Runners (Internal API)
- `POST /api/runner/register` - Register a runner (uses API key)
- `POST /api/runner/heartbeat` - Update runner heartbeat
- `GET /api/runner/tasks` - Get pending tasks for runner
- `POST /api/runner/tasks/{id}/complete` - Mark task as complete
- `GET /api/runner/files/{jobId}/{fileName}` - Download file for runner
- `POST /api/runner/files/{jobId}/upload` - Upload file from runner

### Admin (Admin Only)
- `GET /api/admin/runners` - List all runners
- `GET /api/admin/jobs` - List all jobs
- `GET /api/admin/users` - List all users
- `GET /api/admin/stats` - System statistics

### WebSocket
- `WS /api/ws` - WebSocket connection for real-time updates
  - Subscribe to job channels: `job:{jobId}`
  - Receive job status updates, progress, and logs

## Output Formats

The system supports the following output formats:

### Image Formats
- **PNG** - Standard PNG output
- **JPEG** - JPEG output
- **EXR** - OpenEXR format (HDR)

### Video Formats
- **EXR_264_MP4** - H.264 encoded MP4 from EXR sequence (SDR or HDR)
- **EXR_AV1_MP4** - AV1 encoded MP4 from EXR sequence (with alpha channel support)
- **EXR_VP9_WEBM** - VP9 encoded WebM from EXR sequence (with alpha channel and HDR support)

Video encoding features:
- 2-pass encoding for optimal quality
- HDR preservation using HLG transfer function
- Alpha channel preservation (AV1 and VP9 only)
- Automatic detection of source format (EXR or PNG)
- Software encoding (libx264, libaom-av1, libvpx-vp9)

## Storage Structure

The manager uses a local storage directory (default: `./jiggablend-storage`) with the following structure:

```
jiggablend-storage/
├── blender-versions/     # Bundled Blender versions
│   └── <version>/
├── jobs/                 # Job context files
│   └── <job-id>/
│       └── context.tar
├── outputs/              # Rendered outputs
│   └── <job-id>/
├── temp/                 # Temporary files
└── uploads/              # Uploaded files
```

## Development

### Running Tests

```bash
make test
# Or directly
go test ./... -timeout 30s
```

### Web UI Development

The web UI is built with React and Vite. To develop the UI:

```bash
cd web
npm install
npm run dev  # Development server
npm run build  # Build for production
```

The built files are embedded in the Go binary using `embed.FS`.

## License

MIT