diff --git a/README.md b/README.md index 3a88216..8a06a4f 100644 --- a/README.md +++ b/README.md @@ -4,28 +4,41 @@ A distributed Blender render farm system built with Go. The system consists of a ## Architecture -- **Manager**: Central server with REST API, web UI, DuckDB database, and local file storage +- **Manager**: Central server with REST API, embedded web UI, SQLite database, and local file storage - **Runner**: Linux amd64 client that connects to manager, receives jobs, executes Blender renders, and reports back +Both manager and runner are part of a single binary (`jiggablend`) with subcommands. + ## Features -- OAuth authentication (Google and Discord) -- Web-based job submission and monitoring -- Distributed rendering across multiple runners -- Real-time job progress tracking -- File upload/download for Blender files and rendered outputs -- Runner health monitoring +- **Authentication**: OAuth (Google and Discord) and local authentication with user management +- **Web UI**: Modern React-based interface for job submission and monitoring +- **Distributed Rendering**: Scale across multiple runners with automatic job distribution +- **Real-time Updates**: WebSocket-based progress tracking and job status updates +- **Video Encoding**: Automatic video encoding from EXR/PNG sequences with multiple codec support: + - H.264 (MP4) - SDR and HDR support + - AV1 (MP4) - With alpha channel support + - VP9 (WebM) - With alpha channel and HDR support +- **Output Formats**: PNG, JPEG, EXR, and video formats (MP4, WebM) +- **Blender Version Management**: Support for multiple Blender versions with automatic detection +- **Metadata Extraction**: Automatic extraction of scene metadata from Blender files +- **Admin Panel**: User and runner management interface +- **Runner Management**: API key-based authentication for runners with health monitoring +- **HDR Support**: Preserve HDR range in video encoding with HLG transfer function +- **Alpha Channel**: Preserve alpha channel in video encoding (AV1 and VP9) ## Prerequisites ### Manager -- Go 1.21 or later -- DuckDB (via Go driver) +- Go 1.25.4 or later +- SQLite (via Go driver) +- Blender installed and in PATH (for metadata extraction) +- ImageMagick installed (for EXR preview conversion) ### Runner - Linux amd64 -- Blender installed and in PATH -- FFmpeg installed (optional, for video processing) +- Blender installed (can use bundled versions from storage) +- FFmpeg installed (required for video encoding) ## Installation @@ -44,44 +57,87 @@ go mod download ### Manager -Set the following environment variables for authentication (optional): +Configuration is managed through the CLI using `jiggablend manager config` commands. The configuration is stored in the SQLite database. + +#### Initial Setup + +For testing, use the Makefile helper: +```bash +make init-test +``` + +This will: +- Enable local authentication +- Set a fixed API key for testing +- Create a test admin user (test@example.com / testpassword) + +#### Manual Configuration ```bash -# OAuth Providers (optional) -export GOOGLE_CLIENT_ID="your-google-client-id" -export GOOGLE_CLIENT_SECRET="your-google-client-secret" -export GOOGLE_REDIRECT_URL="http://localhost:8080/api/auth/google/callback" +# Enable local authentication +jiggablend manager config enable localauth -export DISCORD_CLIENT_ID="your-discord-client-id" -export DISCORD_CLIENT_SECRET="your-discord-client-secret" -export DISCORD_REDIRECT_URL="http://localhost:8080/api/auth/discord/callback" +# Add a user +jiggablend manager config add user --admin -# Local Authentication (optional) -export ENABLE_LOCAL_AUTH="true" +# Generate an API key for runners +jiggablend manager config add apikey --scope manager -# Test User (optional, for testing only) -# Creates a local user on startup if it doesn't exist -export LOCAL_TEST_EMAIL="test@example.com" -export LOCAL_TEST_PASSWORD="testpassword" +# Set OAuth credentials +jiggablend manager config set google-oauth --redirect-url +jiggablend manager config set discord-oauth --redirect-url + +# View current configuration +jiggablend manager config show + +# List users and API keys +jiggablend manager config list users +jiggablend manager config list apikeys ``` +#### Environment Variables + +You can also use environment variables with the `JIGGABLEND_` prefix: +- `JIGGABLEND_PORT` - Server port (default: 8080) +- `JIGGABLEND_DB` - Database path (default: jiggablend.db) +- `JIGGABLEND_STORAGE` - Storage path (default: ./jiggablend-storage) +- `JIGGABLEND_LOG_FILE` - Log file path +- `JIGGABLEND_LOG_LEVEL` - Log level (debug, info, warn, error) +- `JIGGABLEND_VERBOSE` - Enable verbose logging + ### Runner -No configuration required. Runner will auto-detect hostname and IP. +The runner requires an API key to connect to the manager. The runner will auto-detect hostname and IP. ## Usage +### Building + +```bash +# Build the unified binary (includes embedded web UI) +make build + +# Or build directly +go build -o bin/jiggablend ./cmd/jiggablend + +# Build web UI separately +make build-web +``` + ### Running the Manager ```bash -# Using make +# Using make (includes test setup) make run-manager # Or directly -go run ./cmd/manager +bin/jiggablend manager # With custom options -go run ./cmd/manager -port 8080 -db jiggablend.db -storage ./storage +bin/jiggablend manager --port 8080 --db jiggablend.db --storage ./jiggablend-storage --log-file manager.log + +# Using environment variables +JIGGABLEND_PORT=8080 JIGGABLEND_DB=jiggablend.db bin/jiggablend manager ``` The manager will start on `http://localhost:8080` by default. @@ -89,26 +145,28 @@ The manager will start on `http://localhost:8080` by default. ### Running a Runner ```bash -# Using make +# Using make (uses test API key) make run-runner -# Or directly -go run ./cmd/runner +# Or directly (requires API key) +bin/jiggablend runner --api-key # With custom options -go run ./cmd/runner -manager http://localhost:8080 -name my-runner +bin/jiggablend runner --manager http://localhost:8080 --name my-runner --api-key --log-file runner.log + +# Using environment variables +JIGGABLEND_MANAGER=http://localhost:8080 JIGGABLEND_API_KEY= bin/jiggablend runner ``` -### Building +### Running Both (for Testing) ```bash -# Build manager -make build-manager - -# Build runner (Linux amd64) -make build-runner +# Run manager and runner in parallel +make run ``` +This will start both the manager and a test runner with a fixed API key. + ## OAuth Setup ### Google OAuth @@ -118,7 +176,10 @@ make build-runner 3. Enable Google+ API 4. Create OAuth 2.0 credentials 5. Add authorized redirect URI: `http://localhost:8080/api/auth/google/callback` -6. Set environment variables with Client ID and Secret +6. Configure using CLI: + ```bash + jiggablend manager config set google-oauth --redirect-url http://localhost:8080/api/auth/google/callback + ``` ### Discord OAuth @@ -126,25 +187,39 @@ make build-runner 2. Create a new application 3. Go to OAuth2 section 4. Add redirect URI: `http://localhost:8080/api/auth/discord/callback` -5. Set environment variables with Client ID and Secret +5. Configure using CLI: + ```bash + jiggablend manager config set discord-oauth --redirect-url http://localhost:8080/api/auth/discord/callback + ``` ## Project Structure ``` jiggablend/ ├── cmd/ -│ ├── manager/ # Manager server application -│ └── runner/ # Runner client application +│ └── jiggablend/ # Unified CLI application +│ ├── cmd/ # Cobra command definitions +│ └── main.go # Entry point ├── internal/ -│ ├── api/ # REST API handlers -│ ├── auth/ # OAuth authentication -│ ├── database/ # DuckDB database models and migrations -│ ├── queue/ # Job queue management -│ ├── storage/ # File storage operations -│ └── runner/ # Runner management logic +│ ├── auth/ # Authentication (OAuth, local, sessions) +│ ├── config/ # Configuration management +│ ├── database/ # SQLite database models and migrations +│ ├── logger/ # Logging utilities +│ ├── manager/ # Manager server logic +│ ├── runner/ # Runner client logic +│ │ ├── api/ # Manager API client +│ │ ├── blender/ # Blender version detection +│ │ ├── encoding/ # Video encoding (H.264, AV1, VP9) +│ │ ├── tasks/ # Task execution (render, encode, process) +│ │ └── workspace/ # Workspace management +│ └── storage/ # File storage operations ├── pkg/ -│ └── types/ # Shared types and models -├── web/ # Static web UI files +│ ├── executils/ # Execution utilities +│ ├── scripts/ # Python scripts for Blender +│ └── types/ # Shared types and models +├── web/ # React web UI +│ ├── src/ # Source files +│ └── dist/ # Built files (embedded in binary) ├── go.mod └── Makefile ``` @@ -156,27 +231,106 @@ jiggablend/ - `GET /api/auth/google/callback` - Google OAuth callback - `GET /api/auth/discord/login` - Initiate Discord OAuth - `GET /api/auth/discord/callback` - Discord OAuth callback +- `POST /api/auth/login` - Local authentication login +- `POST /api/auth/register` - User registration (if enabled) - `POST /api/auth/logout` - Logout - `GET /api/auth/me` - Get current user +- `POST /api/auth/password/change` - Change password ### Jobs - `POST /api/jobs` - Create a new job - `GET /api/jobs` - List user's jobs - `GET /api/jobs/{id}` - Get job details - `DELETE /api/jobs/{id}` - Cancel a job -- `POST /api/jobs/{id}/upload` - Upload job file +- `POST /api/jobs/{id}/upload` - Upload job file (Blender file) - `GET /api/jobs/{id}/files` - List job files - `GET /api/jobs/{id}/files/{fileId}/download` - Download job file +- `GET /api/jobs/{id}/metadata` - Extract metadata from uploaded file +- `GET /api/jobs/{id}/outputs` - List job output files -### Runners -- `GET /api/admin/runners` - List all runners (admin only) -- `POST /api/runner/register` - Register a runner (uses registration token) -- `POST /api/runner/heartbeat` - Update runner heartbeat (runner authenticated) +### Blender +- `GET /api/blender/versions` - List available Blender versions + +### Runners (Internal API) +- `POST /api/runner/register` - Register a runner (uses API key) +- `POST /api/runner/heartbeat` - Update runner heartbeat - `GET /api/runner/tasks` - Get pending tasks for runner - `POST /api/runner/tasks/{id}/complete` - Mark task as complete - `GET /api/runner/files/{jobId}/{fileName}` - Download file for runner - `POST /api/runner/files/{jobId}/upload` - Upload file from runner +### Admin (Admin Only) +- `GET /api/admin/runners` - List all runners +- `GET /api/admin/jobs` - List all jobs +- `GET /api/admin/users` - List all users +- `GET /api/admin/stats` - System statistics + +### WebSocket +- `WS /api/ws` - WebSocket connection for real-time updates + - Subscribe to job channels: `job:{jobId}` + - Receive job status updates, progress, and logs + +## Output Formats + +The system supports the following output formats: + +### Image Formats +- **PNG** - Standard PNG output +- **JPEG** - JPEG output +- **EXR** - OpenEXR format (HDR) + +### Video Formats +- **EXR_264_MP4** - H.264 encoded MP4 from EXR sequence (SDR or HDR) +- **EXR_AV1_MP4** - AV1 encoded MP4 from EXR sequence (with alpha channel support) +- **EXR_VP9_WEBM** - VP9 encoded WebM from EXR sequence (with alpha channel and HDR support) + +Video encoding features: +- 2-pass encoding for optimal quality +- HDR preservation using HLG transfer function +- Alpha channel preservation (AV1 and VP9 only) +- Automatic detection of source format (EXR or PNG) +- Software encoding (libx264, libaom-av1, libvpx-vp9) + +## Storage Structure + +The manager uses a local storage directory (default: `./jiggablend-storage`) with the following structure: + +``` +jiggablend-storage/ +├── blender-versions/ # Bundled Blender versions +│ └── / +├── jobs/ # Job context files +│ └── / +│ └── context.tar +├── outputs/ # Rendered outputs +│ └── / +├── temp/ # Temporary files +└── uploads/ # Uploaded files +``` + +## Development + +### Running Tests + +```bash +make test +# Or directly +go test ./... -timeout 30s +``` + +### Web UI Development + +The web UI is built with React and Vite. To develop the UI: + +```bash +cd web +npm install +npm run dev # Development server +npm run build # Build for production +``` + +The built files are embedded in the Go binary using `embed.FS`. + ## License MIT