Universe Cluster Architecture and Task Dispatch

Universe separates orchestration concerns across two node roles — Master and Wrapper — connected by a Hazelcast cluster. The Master owns all authoritative state and exposes a REST API; Wrappers own nothing persistently but execute every concrete action: copying templates, replacing variables, and managing OS processes. Both roles run from the same fat JAR; a single configuration flag decides which role a node assumes at startup.

Master / Wrapper Topology Overview

When isMasterNode: true, the application startup sequence in UniverseApplication registers a ResilienceMembershipListener, loads all instance configurations from ./configuration/ into the Hazelcast IMap<String, Configuration>, starts the Ktor HTTP server, and launches InstanceCountEnforcer to maintain minimumServiceCount guarantees. When isMasterNode: false, the node joins the Hazelcast cluster using the Master’s masterAddress:masterPort, registers built-in runtime providers (tmux, screen, process), loads extensions, and begins listening for task objects dispatched over IExecutorService.

Architecture Diagram

The following ASCII diagram from the project shows the full component layout and data flow:

┌─────────────────────────────────────────────────────────────┐
│                        Master Node                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │  Ktor REST  │  │   Hazelcast │  │   Console Commands  │ │
│  │    API      │  │   IMap/Exec │  │   (Cloud v2)        │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
│         │                │                    │             │
│         ▼                ▼                    ▼             │
│  POST /api/instances  DeployInstanceTask   instance create │
│  PUT /api/instances   StopInstanceTask     instance stop   │
│  POST /api/commands   TemplateSyncTask     template sync   │
└─────────────────────────────────────────────────────────────┘
                              │
                              │ Hazelcast Cluster
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                       Wrapper Node(s)                       │
│  ┌─────────────┐  ┌─────────────┐  ┌──────────────────────┐ │
│  │ TaskRouter  │  │  Template   │  │  RuntimeProvider     │ │
│  │ (IExecutor) │  │   Manager   │  │ (screen/tmux/docker) │ │
│  └─────────────┘  └─────────────┘  └──────────────────────┘ │
│         │                │                    │             │
│         ▼                ▼                    ▼             │
│   Receive Tasks    Copy Templates      Start Processes      │
│   Route Actions    Replace Variables   Pipe Stdin           │
└─────────────────────────────────────────────────────────────┘

Component Reference

Master: Ktor REST API

KtorServerService starts only when isMasterNode = true. It runs on Ktor 3.4.3 with a Netty engine and registers six route groups:

Route file	Responsibility
`InstanceRoutes`	`GET/POST /api/instances`, `DELETE/PATCH /api/instances/{id}`, stdin execute, log retrieval, live-log WebSocket
`CommandRoutes`	`POST /api/commands/execute` — captures console output and returns it as JSON
`ClusterRoutes`	`GET /api/cluster/nodes`, node details, remote command execution
`NodeRoutes`	`GET /api/node`, `/api/node/config`, `POST /api/node/reload`, `GET /api/ping`
`ConfigurationRoutes`	Full CRUD on `./configuration/*.json` via Hazelcast `IMap<String, Configuration>`
`TemplateRoutes`	Template listing, info, and sync dispatch

All routes use Gson content negotiation. Bearer auth, CORS, and a global exception handler are registered as Ktor plugins.

Master: Hazelcast IMap State

ClusterStateService manages two distributed maps visible to every cluster member:

"instances" → IMap<String, InstanceInfo> — the single source of truth for all running, creating, offline, and stopped instances. Wrappers write state updates directly to this IMap via ClusterStateService.putInstance() — both nodes are Hazelcast members and share the same distributed map.
"configurations" → IMap<String, Configuration> — loaded from ./configuration/ on startup and reloadable at runtime via config reload or POST /api/node/reload.

ResilienceMembershipListener fires on memberRemoved: any instance whose wrapperNodeId matches the departed member is transitioned to OFFLINE rather than deleted, preserving history and enabling future recovery.

Master: Console Commands

CommandBootstrap reads System.in on a dedicated thread using JLine. The same command registry backs both the interactive console and POST /api/commands/execute, so every command is available over HTTP without a terminal. All console output is captured into a List<String> and returned in the REST response.Key command groups: cluster, instance, config, template, extension, s3 (via extension), help, stop.

Wrapper: TaskRouter and IExecutorService

The Master dispatches work to Wrappers by submitting UniverseCallableTask objects — Gson-serialized JSON strings wrapped in Callable<String> — to Hazelcast IExecutorService, targeting a specific cluster member UUID.TaskDeserializer on the receiving Wrapper deserialises the JSON back to a concrete task type. TaskRouter then dispatches to the appropriate handler:

Task	Handler
`DeployInstanceTask`	`TemplateManager.resolve()` → `RuntimeProvider.start()`
`StopInstanceTask`	`RuntimeProvider.stop()`
`ExecuteCommandTask`	`RuntimeProvider.executeCommand()` (pipes to process stdin)
`TemplateSyncTask`	`TemplateSyncService` — unzips received bytes into `./templates/`
`ShutdownNodeTask`	Stops all local instances, releases ports, and exits the JVM

Wrapper: TemplateManager

TemplateManager resolves the full file tree for an instance before the runtime starts:

Reads TemplateInstallationConfig from the instance’s Configuration.
Collects templates from allOf, allInGroups, oneOf, and oneInGroups fields, sorting by priority (ascending).
Copies each template tree from ./templates/<group>/<name>/ to ./running/<instance-id>/, optionally overwriting present files when onTemplatePasteOverridePresentFiles is true.
Scans every file listed in Configuration.fileModifications and replaces %VARIABLE% placeholders with actual values from built-in providers and any extension-contributed TemplateVariableProvider instances.

Wrapper: RuntimeProvider

RuntimeRegistry (a Guice-managed ConcurrentHashMap) maps string keys to RuntimeProvider implementations. Built-in providers registered at startup:

Key	Class	Mechanism
`screen`	`ScreenRuntimeProvider`	`screen -dmS` session; `screen -S <id> -X stuff` for stdin
`tmux`	`TmuxRuntimeProvider`	`tmux new-session -d`; `tmux send-keys` for stdin
`process`	`ProcessRuntimeProvider`	Direct `ProcessBuilder`; no session manager

Extension-provided runtimes (docker, k8s) register themselves in Extension.onLoad() via the injected RuntimeRegistry, and must be installed before the first deploy task arrives.

Hazelcast Task Dispatch

Task objects are defined in the api module and serialised to JSON strings before submission to IExecutorService. The full dispatch path for a POST /api/instances request is:

Ktor route handler receives {"configurationName": "default"}.
Master selects a target Wrapper node from Configuration.nodes (matched against live Hazelcast members).
Master writes a new InstanceInfo with state = CREATING into the "instances" IMap.
Master wraps a DeployInstanceTask as a UniverseCallableTask JSON payload.
TaskDispatcher.dispatch(task, targetMemberUUID) submits the callable to IExecutorService.
Hazelcast serialises and routes the callable to the target member.
Wrapper deserialises the payload in TaskDeserializer, extracts DeployInstanceTask.
TaskRouter calls TemplateManager (copy + variable replace), then RuntimeProvider.start().
Wrapper writes InstanceInfo.state = ONLINE directly to the "instances" IMap via ClusterStateService.putInstance() (the Wrapper is a full Hazelcast cluster member and shares the distributed map).

Instance Lifecycle

CREATING

The Master writes the InstanceInfo record to the Hazelcast "instances" IMap with state = CREATING immediately after selecting a Wrapper and before dispatching the task. The instance is visible in GET /api/instances at this point.

ONLINE

Once the Wrapper has copied templates, replaced variables, and successfully started the process via RuntimeProvider.start(), it writes the updated InstanceInfo (with state = ONLINE, the allocated port, and the process PID) directly to the shared Hazelcast IMap via ClusterStateService.putInstance(). Wrappers continue sending periodic heartbeat updates to keep lastHeartbeat current.

OFFLINE

If a Wrapper node disconnects from the Hazelcast cluster (network partition, crash, Docker container stop), ResilienceMembershipListener on the Master marks all instances owned by that Wrapper as OFFLINE. The records are not deleted, preserving history and allowing InstanceRecoveryService to attempt re-attachment on reconnect.

STOPPED

A deliberate stop via DELETE /api/instances/{id}, instance stop <id>, or PATCH /api/instances/{id}/lifecycle?target=stop dispatches a StopInstanceTask to the owning Wrapper. After RuntimeProvider.stop() completes, the state transitions to STOPPED and the working directory in ./running/<id>/ is cleaned up (unless static: true).

Data Flow: Creating an Instance

The following sequence shows every system involved when you run POST /api/instances:

Client
  │
  ▼  POST /api/instances {"configurationName":"default"}
Ktor REST API (Master)
  │  Resolves Configuration from Hazelcast IMap
  │  Selects target Wrapper node (matches config.nodes vs live members)
  │  Allocates port via PortAllocator (checks in-memory + IMap + OS bind)
  │  Writes InstanceInfo {state: CREATING} → IMap "instances"
  │
  ▼  IExecutorService.submit(UniverseCallableTask{DeployInstanceTask})
Hazelcast Cluster
  │
  ▼  Callable delivered to target Wrapper member
TaskDeserializer (Wrapper)
  │  Deserialises JSON → DeployInstanceTask
  │
  ▼
TaskRouter (Wrapper)
  │
  ├──▶ TemplateManager
  │      Resolves templates (allOf, oneOf, …)
  │      Copies trees → ./running/<id>/
  │      Scans fileModifications, replaces %VARIABLE%
  │
  └──▶ RuntimeProvider.start(instanceId, workDir, command, env)
         Spawns screen/tmux/process/docker/k8s

  ▼  ClusterStateService.putInstance({state: ONLINE, allocatedPort, processPid, …})
Hazelcast IMap "instances" (shared by all cluster members)
  │  Wrapper writes directly — no HTTP round-trip to Master
  ▼
Client  ←  InstanceInfo {state: ONLINE, allocatedPort: 25565, …}

Port Allocation Strategy

PortAllocator checks three sources before committing to a port, preventing conflicts across concurrent deployments and external services:

In-memory allocations — ports already claimed by this JVM instance during the current session.
Cluster-wide IMap scan — queries all ONLINE and CREATING instances in the "instances" IMap and skips their allocatedPort values.
OS-level probe — attempts a ServerSocket bind and a TCP connect probe on each candidate port to catch services already listening on the machine.

Module Table

Module	Description
`api`	Shared DTOs and interfaces: `InstanceInfo`, `Configuration`, `TemplateInstallationConfig`, `PortRange`, `InstanceState`, `RuntimeProvider`, `RuntimeRegistry`, task objects (`DeployInstanceTask`, `StopInstanceTask`, `ExecuteCommandTask`, `TemplateSyncTask`, `ShutdownNodeTask`)
`app`	Core orchestrator: `UniverseApplication`, `HazelcastService`, `ClusterStateService`, `KtorServerService`, `TemplateManager`, `TaskDispatcher`, `TaskRouter`, `PortAllocator`, `CommandBootstrap`, built-in runtimes, `InstanceCountEnforcer`, `InstanceHealthMonitor`, `InstanceRecoveryService`
`loader`	Bootstrap classloader: extracts `app.jarinjar` to a temp file, downloads runtime dependencies from `dependencies.txt` via `DependencyLoader`, appends to `URLClassLoader`, and invokes `AppKt.run()`
`extensions/extension-api`	Extension-facing contracts: `Extension` (lifecycle hooks `onLoad`, `onReload`, `onUnload`), `TemplateStorageProvider`, `TemplateVariableProvider`, `TemplateStorageRegistry`, `TemplateVariableRegistry`

Explore Further

Node Configuration

Full reference for every field in config.json, database.json, and the debug logging system.

Runtimes

How screen, tmux, and process runtimes work, and how to add Docker or Kubernetes support via extensions.

Extensions

Extension lifecycle, available registries, and how to write your own runtime, storage, or metrics extension.

REST API Reference

Complete endpoint reference including request/response schemas, WebSocket streams, and authentication.

Getting Started

Configuration

Runtimes

Extensions

Minecraft Integration

Operations

Universe Cluster Architecture and Task Dispatch

Master / Wrapper Topology Overview

Architecture Diagram

Component Reference

Hazelcast Task Dispatch

Instance Lifecycle

Data Flow: Creating an Instance

Port Allocation Strategy

Module Table

Explore Further

Node Configuration

Runtimes

Extensions

REST API Reference

Build docs developers (and LLMs) love

Getting Started

Configuration

Runtimes

Extensions

Minecraft Integration

Operations

Documentation Index

​Master / Wrapper Topology Overview

​Architecture Diagram

​Component Reference

​Hazelcast Task Dispatch

​Instance Lifecycle

​Data Flow: Creating an Instance

​Port Allocation Strategy

​Module Table

​Explore Further

Node Configuration

Runtimes

Extensions

REST API Reference

Build docs developers (and LLMs) love

Master / Wrapper Topology Overview

Architecture Diagram

Component Reference

Hazelcast Task Dispatch

Instance Lifecycle

Data Flow: Creating an Instance

Port Allocation Strategy

Module Table

Explore Further