s1d3sw1ped/jiggablend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Update README.md to reflect significant changes in architecture and features. Replace DuckDB with SQLite for the database, enhance authentication options, and introduce a modern React-based web UI. Expand job management capabilities, including video encoding support and metadata extraction. Revise installation and configuration instructions, and clarify output formats and storage structure. Improve development guidelines for building and testing the application.

Browse Source
This commit is contained in:
Justin Harms
2026-01-02 14:03:03 -06:00
parent 94490237fe
commit c7c8762164
1 changed files with 210 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

266
README.md
View File

@@ -4,28 +4,41 @@ A distributed Blender render farm system built with Go. The system consists of a
## Architecture ## Architecture
- **Manager**: Central server with REST API, web UI, DuckDB database, and local file storage - **Manager**: Central server with REST API, embedded web UI, SQLite database, and local file storage
- **Runner**: Linux amd64 client that connects to manager, receives jobs, executes Blender renders, and reports back - **Runner**: Linux amd64 client that connects to manager, receives jobs, executes Blender renders, and reports back
Both manager and runner are part of a single binary (`jiggablend`) with subcommands.
## Features ## Features
- OAuth authentication (Google and Discord) - **Authentication**: OAuth (Google and Discord) and local authentication with user management
- Web-based job submission and monitoring - **Web UI**: Modern React-based interface for job submission and monitoring
- Distributed rendering across multiple runners - **Distributed Rendering**: Scale across multiple runners with automatic job distribution
- Real-time job progress tracking - **Real-time Updates**: WebSocket-based progress tracking and job status updates
- File upload/download for Blender files and rendered outputs - **Video Encoding**: Automatic video encoding from EXR/PNG sequences with multiple codec support:
- Runner health monitoring - H.264 (MP4) - SDR and HDR support
- AV1 (MP4) - With alpha channel support
- VP9 (WebM) - With alpha channel and HDR support
- **Output Formats**: PNG, JPEG, EXR, and video formats (MP4, WebM)
- **Blender Version Management**: Support for multiple Blender versions with automatic detection
- **Metadata Extraction**: Automatic extraction of scene metadata from Blender files
- **Admin Panel**: User and runner management interface
- **Runner Management**: API key-based authentication for runners with health monitoring
- **HDR Support**: Preserve HDR range in video encoding with HLG transfer function
- **Alpha Channel**: Preserve alpha channel in video encoding (AV1 and VP9)
## Prerequisites ## Prerequisites
### Manager ### Manager
- Go 1.21 or later - Go 1.25.4 or later
- DuckDB (via Go driver) - SQLite (via Go driver)
- Blender installed and in PATH (for metadata extraction)
- ImageMagick installed (for EXR preview conversion)
### Runner ### Runner
- Linux amd64 - Linux amd64
- Blender installed and in PATH - Blender installed (can use bundled versions from storage)
- FFmpeg installed (optional, for video processing) - FFmpeg installed (required for video encoding)
## Installation ## Installation
@@ -44,44 +57,87 @@ go mod download
### Manager ### Manager
Set the following environment variables for authentication (optional): Configuration is managed through the CLI using `jiggablend manager config` commands. The configuration is stored in the SQLite database.
#### Initial Setup
For testing, use the Makefile helper:
```bash
make init-test
```
This will:
- Enable local authentication
- Set a fixed API key for testing
- Create a test admin user (test@example.com / testpassword)
#### Manual Configuration
```bash ```bash
# OAuth Providers (optional) # Enable local authentication
export GOOGLE_CLIENT_ID="your-google-client-id" jiggablend manager config enable localauth
export GOOGLE_CLIENT_SECRET="your-google-client-secret"
export GOOGLE_REDIRECT_URL="http://localhost:8080/api/auth/google/callback"
export DISCORD_CLIENT_ID="your-discord-client-id" # Add a user
export DISCORD_CLIENT_SECRET="your-discord-client-secret" jiggablend manager config add user <email> <password> --admin
export DISCORD_REDIRECT_URL="http://localhost:8080/api/auth/discord/callback"
# Local Authentication (optional) # Generate an API key for runners
export ENABLE_LOCAL_AUTH="true" jiggablend manager config add apikey <name> --scope manager
# Test User (optional, for testing only) # Set OAuth credentials
# Creates a local user on startup if it doesn't exist jiggablend manager config set google-oauth <client-id> <client-secret> --redirect-url <url>
export LOCAL_TEST_EMAIL="test@example.com" jiggablend manager config set discord-oauth <client-id> <client-secret> --redirect-url <url>
export LOCAL_TEST_PASSWORD="testpassword"
# View current configuration
jiggablend manager config show
# List users and API keys
jiggablend manager config list users
jiggablend manager config list apikeys
``` ```
#### Environment Variables
You can also use environment variables with the `JIGGABLEND_` prefix:
- `JIGGABLEND_PORT` - Server port (default: 8080)
- `JIGGABLEND_DB` - Database path (default: jiggablend.db)
- `JIGGABLEND_STORAGE` - Storage path (default: ./jiggablend-storage)
- `JIGGABLEND_LOG_FILE` - Log file path
- `JIGGABLEND_LOG_LEVEL` - Log level (debug, info, warn, error)
- `JIGGABLEND_VERBOSE` - Enable verbose logging
### Runner ### Runner
No configuration required. Runner will auto-detect hostname and IP. The runner requires an API key to connect to the manager. The runner will auto-detect hostname and IP.
## Usage ## Usage
### Building
```bash
# Build the unified binary (includes embedded web UI)
make build
# Or build directly
go build -o bin/jiggablend ./cmd/jiggablend
# Build web UI separately
make build-web
```
### Running the Manager ### Running the Manager
```bash ```bash
# Using make # Using make (includes test setup)
make run-manager make run-manager
# Or directly # Or directly
go run ./cmd/manager bin/jiggablend manager
# With custom options # With custom options
go run ./cmd/manager -port 8080 -db jiggablend.db -storage ./storage bin/jiggablend manager --port 8080 --db jiggablend.db --storage ./jiggablend-storage --log-file manager.log
# Using environment variables
JIGGABLEND_PORT=8080 JIGGABLEND_DB=jiggablend.db bin/jiggablend manager
``` ```
The manager will start on `http://localhost:8080` by default. The manager will start on `http://localhost:8080` by default.
@@ -89,26 +145,28 @@ The manager will start on `http://localhost:8080` by default.
### Running a Runner ### Running a Runner
```bash ```bash
# Using make # Using make (uses test API key)
make run-runner make run-runner
# Or directly # Or directly (requires API key)
go run ./cmd/runner bin/jiggablend runner --api-key <your-api-key>
# With custom options # With custom options
go run ./cmd/runner -manager http://localhost:8080 -name my-runner bin/jiggablend runner --manager http://localhost:8080 --name my-runner --api-key <key> --log-file runner.log
# Using environment variables
JIGGABLEND_MANAGER=http://localhost:8080 JIGGABLEND_API_KEY=<key> bin/jiggablend runner
``` ```
### Building ### Running Both (for Testing)
```bash ```bash
# Build manager # Run manager and runner in parallel
make build-manager make run
# Build runner (Linux amd64)
make build-runner
``` ```
This will start both the manager and a test runner with a fixed API key.
## OAuth Setup ## OAuth Setup
### Google OAuth ### Google OAuth
@@ -118,7 +176,10 @@ make build-runner
3. Enable Google+ API 3. Enable Google+ API
4. Create OAuth 2.0 credentials 4. Create OAuth 2.0 credentials
5. Add authorized redirect URI: `http://localhost:8080/api/auth/google/callback` 5. Add authorized redirect URI: `http://localhost:8080/api/auth/google/callback`
6. Set environment variables with Client ID and Secret 6. Configure using CLI:
```bash
jiggablend manager config set google-oauth <client-id> <client-secret> --redirect-url http://localhost:8080/api/auth/google/callback
```
### Discord OAuth ### Discord OAuth
@@ -126,25 +187,39 @@ make build-runner
2. Create a new application 2. Create a new application
3. Go to OAuth2 section 3. Go to OAuth2 section
4. Add redirect URI: `http://localhost:8080/api/auth/discord/callback` 4. Add redirect URI: `http://localhost:8080/api/auth/discord/callback`
5. Set environment variables with Client ID and Secret 5. Configure using CLI:
```bash
jiggablend manager config set discord-oauth <client-id> <client-secret> --redirect-url http://localhost:8080/api/auth/discord/callback
```
## Project Structure ## Project Structure
``` ```
jiggablend/ jiggablend/
├── cmd/ ├── cmd/
│ ├── manager/ # Manager server application │ └── jiggablend/ # Unified CLI application
│ └── runner/ # Runner client application │ ├── cmd/ # Cobra command definitions
│ └── main.go # Entry point
├── internal/ ├── internal/
│ ├── api/ # REST API handlers │ ├── auth/ # Authentication (OAuth, local, sessions)
│ ├── auth/ # OAuth authentication │ ├── config/ # Configuration management
│ ├── database/ # DuckDB database models and migrations │ ├── database/ # SQLite database models and migrations
│ ├── queue/ # Job queue management │ ├── logger/ # Logging utilities
│ ├── storage/ # File storage operations │ ├── manager/ # Manager server logic
│ └── runner/ # Runner management logic │ ├── runner/ # Runner client logic
│ │ ├── api/ # Manager API client
│ │ ├── blender/ # Blender version detection
│ │ ├── encoding/ # Video encoding (H.264, AV1, VP9)
│ │ ├── tasks/ # Task execution (render, encode, process)
│ │ └── workspace/ # Workspace management
│ └── storage/ # File storage operations
├── pkg/ ├── pkg/
│ └── types/ # Shared types and models │ ├── executils/ # Execution utilities
├── web/ # Static web UI files │ ├── scripts/ # Python scripts for Blender
│ └── types/ # Shared types and models
├── web/ # React web UI
│ ├── src/ # Source files
│ └── dist/ # Built files (embedded in binary)
├── go.mod ├── go.mod
└── Makefile └── Makefile
``` ```
@@ -156,27 +231,106 @@ jiggablend/
- `GET /api/auth/google/callback` - Google OAuth callback - `GET /api/auth/google/callback` - Google OAuth callback
- `GET /api/auth/discord/login` - Initiate Discord OAuth - `GET /api/auth/discord/login` - Initiate Discord OAuth
- `GET /api/auth/discord/callback` - Discord OAuth callback - `GET /api/auth/discord/callback` - Discord OAuth callback
- `POST /api/auth/login` - Local authentication login
- `POST /api/auth/register` - User registration (if enabled)
- `POST /api/auth/logout` - Logout - `POST /api/auth/logout` - Logout
- `GET /api/auth/me` - Get current user - `GET /api/auth/me` - Get current user
- `POST /api/auth/password/change` - Change password
### Jobs ### Jobs
- `POST /api/jobs` - Create a new job - `POST /api/jobs` - Create a new job
- `GET /api/jobs` - List user's jobs - `GET /api/jobs` - List user's jobs
- `GET /api/jobs/{id}` - Get job details - `GET /api/jobs/{id}` - Get job details
- `DELETE /api/jobs/{id}` - Cancel a job - `DELETE /api/jobs/{id}` - Cancel a job
- `POST /api/jobs/{id}/upload` - Upload job file - `POST /api/jobs/{id}/upload` - Upload job file (Blender file)
- `GET /api/jobs/{id}/files` - List job files - `GET /api/jobs/{id}/files` - List job files
- `GET /api/jobs/{id}/files/{fileId}/download` - Download job file - `GET /api/jobs/{id}/files/{fileId}/download` - Download job file
- `GET /api/jobs/{id}/metadata` - Extract metadata from uploaded file
- `GET /api/jobs/{id}/outputs` - List job output files
### Runners ### Blender
- `GET /api/admin/runners` - List all runners (admin only) - `GET /api/blender/versions` - List available Blender versions
- `POST /api/runner/register` - Register a runner (uses registration token)
- `POST /api/runner/heartbeat` - Update runner heartbeat (runner authenticated) ### Runners (Internal API)
- `POST /api/runner/register` - Register a runner (uses API key)
- `POST /api/runner/heartbeat` - Update runner heartbeat
- `GET /api/runner/tasks` - Get pending tasks for runner - `GET /api/runner/tasks` - Get pending tasks for runner
- `POST /api/runner/tasks/{id}/complete` - Mark task as complete - `POST /api/runner/tasks/{id}/complete` - Mark task as complete
- `GET /api/runner/files/{jobId}/{fileName}` - Download file for runner - `GET /api/runner/files/{jobId}/{fileName}` - Download file for runner
- `POST /api/runner/files/{jobId}/upload` - Upload file from runner - `POST /api/runner/files/{jobId}/upload` - Upload file from runner
### Admin (Admin Only)
- `GET /api/admin/runners` - List all runners
- `GET /api/admin/jobs` - List all jobs
- `GET /api/admin/users` - List all users
- `GET /api/admin/stats` - System statistics
### WebSocket
- `WS /api/ws` - WebSocket connection for real-time updates
- Subscribe to job channels: `job:{jobId}`
- Receive job status updates, progress, and logs
## Output Formats
The system supports the following output formats:
### Image Formats
- **PNG** - Standard PNG output
- **JPEG** - JPEG output
- **EXR** - OpenEXR format (HDR)
### Video Formats
- **EXR_264_MP4** - H.264 encoded MP4 from EXR sequence (SDR or HDR)
- **EXR_AV1_MP4** - AV1 encoded MP4 from EXR sequence (with alpha channel support)
- **EXR_VP9_WEBM** - VP9 encoded WebM from EXR sequence (with alpha channel and HDR support)
Video encoding features:
- 2-pass encoding for optimal quality
- HDR preservation using HLG transfer function
- Alpha channel preservation (AV1 and VP9 only)
- Automatic detection of source format (EXR or PNG)
- Software encoding (libx264, libaom-av1, libvpx-vp9)
## Storage Structure
The manager uses a local storage directory (default: `./jiggablend-storage`) with the following structure:
```
jiggablend-storage/
├── blender-versions/ # Bundled Blender versions
│ └── <version>/
├── jobs/ # Job context files
│ └── <job-id>/
│ └── context.tar
├── outputs/ # Rendered outputs
│ └── <job-id>/
├── temp/ # Temporary files
└── uploads/ # Uploaded files
```
## Development
### Running Tests
```bash
make test
# Or directly
go test ./... -timeout 30s
```
### Web UI Development
The web UI is built with React and Vite. To develop the UI:
```bash
cd web
npm install
npm run dev # Development server
npm run build # Build for production
```
The built files are embedded in the Go binary using `embed.FS`.
## License ## License
MIT MIT