TornadoExecutionPlan: Control GPU Execution Policy

TornadoExecutionPlan is the runtime orchestration object in TornadoVM. It wraps one or more ImmutableTaskGraph snapshots and exposes a rich fluent API for controlling every aspect of execution: device selection, profiling, warm-up passes, batch sizing, CUDA Graph capture, concurrent device dispatch, and memory limits. Because the class implements AutoCloseable, it integrates naturally with try-with-resources blocks. All configuration methods return a new TornadoExecutionPlan node chained to the previous one, preserving a complete trace of the applied settings that can be inspected with getTraceExecutionPlan().

TornadoExecutionPlan is a sealed class. Only the internal plan-type subclasses (WithDevice, WithProfiler, WithBatch, etc.) may extend it. Application code should interact exclusively through the public API described here.

ImmutableTaskGraph

ImmutableTaskGraph is the read-only snapshot produced by TaskGraph.snapshot(). It encapsulates a frozen copy of the task graph’s tasks, transfer directives, and device hints. You cannot add or remove tasks from an ImmutableTaskGraph — to change the graph, mutate the source TaskGraph and call snapshot() again.

TaskGraph tg = new TaskGraph("myGraph")
    .transferToDevice(DataTransferMode.EVERY_EXECUTION, input)
    .task("t0", Kernels::compute, input, output)
    .transferToHost(DataTransferMode.EVERY_EXECUTION, output);

ImmutableTaskGraph itg = tg.snapshot();  // immutable; safe to share

Pass one or more ImmutableTaskGraph instances to the TornadoExecutionPlan constructor:

TornadoExecutionPlan plan = new TornadoExecutionPlan(itg);

Constructor

TornadoExecutionPlan(ImmutableTaskGraph... immutableTaskGraphs)

constructor

Creates an execution plan over the given set of immutable task graphs. When more than one graph is supplied, the runtime automatically adjusts data-access types across graphs to avoid redundant transfers. All configuration methods applied to the plan affect every graph in the executor.

Parameter	Type	Description
`immutableTaskGraphs`	`ImmutableTaskGraph...`	One or more immutable task graphs to execute.

Core Execution

execute()

TornadoExecutionResult

Runs all task graphs managed by the plan, in order, on their assigned devices. Returns a TornadoExecutionResult that carries profiler data and supports on-demand host readback.

TornadoExecutionResult result = plan.execute();

withPreCompilation()

TornadoExecutionPlan

Invokes the TornadoVM JIT compiler for all tasks without running them. Use this to separate compilation latency from the first timed execution.

Device Selection

withDevice(TornadoDevice device)

TornadoExecutionPlan

Sets the target device for all task graphs in the plan. Overrides any device previously set on individual graphs.

Parameter	Type	Description
`device`	`TornadoDevice`	Target device obtained from `TornadoExecutionPlan.getDevice(driverIdx, deviceIdx)`.

TornadoDevice gpu = TornadoExecutionPlan.getDevice(0, 0);
plan.withDevice(gpu).execute();

withDevice(String taskName, TornadoDevice device)

TornadoExecutionPlan

Sets the target device for a single named task. The task name must be fully qualified as "graphName.taskId".

Parameter	Type	Description
`taskName`	`String`	Fully qualified task name, e.g. `"myGraph.t0"`.
`device`	`TornadoDevice`	Target device for this task only.

getDevice(int driverIndex, int deviceIndex)

TornadoDevice

Static helper to look up a device by backend (driver) index and device index within that backend.

withConcurrentDevices()

TornadoExecutionPlan

Enables simultaneous dispatch of all tasks across multiple devices. TornadoVM does not check for inter-task data dependencies in this mode — ensuring data independence across concurrent tasks is the caller’s responsibility.

Grid Scheduler

withGridScheduler(GridScheduler scheduler)

TornadoExecutionPlan

Attaches a GridScheduler that maps task names to explicit WorkerGrid thread configurations. Any task not found in the scheduler uses TornadoVM’s default thread-count heuristic.

Parameter	Type	Description
`scheduler`	`GridScheduler`	A scheduler with at least one registered worker grid.

WorkerGrid2D grid = new WorkerGrid2D(1024, 1024);
grid.setLocalWork(16, 16, 1);
GridScheduler sched = new GridScheduler("myGraph.matmul", grid);
plan.withGridScheduler(sched).execute();

withDefaultScheduler()

TornadoExecutionPlan

Reverts thread scheduling to TornadoVM’s built-in auto-tuned heuristic, discarding any previously attached GridScheduler.

Profiling

withProfiler(ProfilerMode profilerMode)

TornadoExecutionPlan

Enables the built-in profiler. After each execute(), the TornadoExecutionResult carries detailed timing for JIT compilation, data transfers, kernel dispatch, and device-side execution.

Parameter	Type	Description
`profilerMode`	`ProfilerMode`	One of `ProfilerMode.CONSOLE` or `ProfilerMode.SILENT`.

withoutProfiler()

TornadoExecutionPlan

Disables the profiler if it was previously enabled. This is the default state for a new execution plan.

ProfilerMode Values

CONSOLE

ProfilerMode

Prints a formatted profiler report to stdout after each execute() call. Useful for quick iteration during development.

SILENT

ProfilerMode

Collects profiler data silently. Query results via TornadoExecutionResult.getProfilerResult() after execution. Suitable for production benchmarking pipelines.

Warm-Up

withWarmUpTime(long milliseconds)

TornadoExecutionPlan

Runs the full execution plan — including data transfers and kernel execution — repeatedly for at least the specified duration in milliseconds. This ensures the JIT compiler has compiled all tasks before the timed benchmark begins.

Parameter	Type	Description
`milliseconds`	`long`	Minimum warm-up duration. Must be non-negative.

plan.withWarmUpTime(2000).execute();  // warm up for 2 seconds

withWarmUpIterations(int iterations)

TornadoExecutionPlan

Runs the full execution plan a fixed number of times as a warm-up pass.

Parameter	Type	Description
`iterations`	`int`	Number of warm-up iterations. Must be non-negative.

Batch Execution

withBatch(String batchSize)

TornadoExecutionPlan

Splits the iteration space into chunks of the given size. Use this when the full dataset exceeds the device’s global memory. TornadoVM transparently tiles the execution and stitches the results.

Parameter	Type	Description
`batchSize`	`String`	Size string in the format `"<number>MB"`, e.g. `"512MB"`.

plan.withBatch("256MB").execute();

CUDA-Specific Features

withCUDAGraph()

TornadoExecutionPlan

Enables CUDA Graph capture on the CUDA backend. On first execution TornadoVM records the entire kernel launch sequence into a CUDA Graph and replays it on subsequent calls, eliminating CPU-side launch overhead for repetitive workloads.

withIntraPlanConcurrency()

TornadoExecutionPlan

Routes independent operations (H2D copies, kernel launches, D2H copies) to separate CUDA streams so that they may overlap. Cross-stream ordering is enforced via device events derived from the bytecode dependency DAG. Currently active on the CUDA backend only; a no-op for OpenCL and Metal.

withStagedTransfers()

TornadoExecutionPlan

Enables pipelined host-to-device transfers via pinned staging buffers for large read-only uploads (FIRST_EXECUTION mode, ≥ 16 MB by default). Chunk i+1 is staged while chunk i is in flight over DMA. CUDA backend only; a no-op elsewhere.

Memory Management

withMemoryLimit(String memoryLimit)

TornadoExecutionPlan

Caps the total device memory that this execution plan may allocate. The runtime enforces the limit before each allocation.

Parameter	Type	Description
`memoryLimit`	`String`	Size string, e.g. `"1GB"` or `"512MB"`.

withoutMemoryLimit()

TornadoExecutionPlan

Removes any previously set memory cap, restoring the default behaviour (use as much device memory as the backend allows).

getCurrentDeviceMemoryUsage()

long

Returns the current number of bytes consumed on the device by all task graphs in this plan.

resetDevice()

TornadoExecutionPlan

Resets the internal GPU/CPU execution context to its default state. This cleans the code cache, clears all events associated with the current execution plan, and resets runtime parameters. Use after close() when you need guaranteed context cleanup in memory-constrained scenarios.

TornadoExecutionResult

TornadoExecutionResult is returned by every execute() call. It provides access to profiler data and supports explicit host readback for UNDER_DEMAND buffers.

getProfilerResult()

TornadoProfilerResult

Returns the TornadoProfilerResult object containing kernel time, data transfer time, compilation time, and byte counts. Only populated when withProfiler() was set on the execution plan.

transferToHost(Object... objects)

TornadoExecutionResult

Forces an immediate device-to-host copy for the specified objects. Required for any buffer tagged with DataTransferMode.UNDER_DEMAND. Returns this for fluent chaining.

isReady()

boolean

Returns true when all task graphs in the associated executor have finished execution.

Intra-Plan Graph Selection

Show Selective graph execution with withGraph() and withAllGraphs()

When an execution plan contains multiple ImmutableTaskGraph instances, you can execute only a subset of them in a given invocation:

TornadoExecutionPlan plan = new TornadoExecutionPlan(graphA, graphB, graphC);

// Execute only graph at index 1 (graphB)
plan.withGraph(1).execute();

// Restore full execution across all graphs
plan.withAllGraphs().execute();

withGraph(int graphIndex) selects a single graph by its position in the varargs list passed to the constructor. Any active GridScheduler is propagated to the selected graph automatically. This is useful for conditional pipelines where only some stages need to run on a given iteration.

Use withGraph() together with withConcurrentDevices() to route independent graphs to different GPUs in a multi-device machine, then restore with withAllGraphs() when all graphs need to run together.

AutoCloseable and Resource Management

TornadoExecutionPlan implements AutoCloseable. The close() method calls freeDeviceMemory(), releasing all device buffers back to the pool.

try (TornadoExecutionPlan plan = new TornadoExecutionPlan(itg)) {
    plan.withDevice(TornadoExecutionPlan.getDevice(0, 0))
        .withProfiler(ProfilerMode.SILENT)
        .execute()
        .transferToHost(output);
} // device buffers automatically released here

Closing an execution plan does not synchronously free device allocations at the OS level — it marks them as reusable. If you need guaranteed deallocation (e.g., in a memory-constrained loop), call resetDevice() after closing the plan.

Complete Fluent Example

import uk.ac.manchester.tornado.api.*;
import uk.ac.manchester.tornado.api.enums.*;

public class ExecutionPlanDemo {

    public static void saxpy(float alpha, float[] x, float[] y) {
        for (@Parallel int i = 0; i < x.length; i++) {
            y[i] = alpha * x[i] + y[i];
        }
    }

    public static void main(String[] args) throws Exception {
        int n = 1 << 20;
        float[] x = new float[n], y = new float[n];
        // ... initialise x and y ...

        TaskGraph tg = new TaskGraph("saxpy")
            .transferToDevice(DataTransferMode.EVERY_EXECUTION, x, y)
            .task("t0", ExecutionPlanDemo::saxpy, 2.0f, x, y)
            .transferToHost(DataTransferMode.EVERY_EXECUTION, y);

        ImmutableTaskGraph itg = tg.snapshot();

        try (TornadoExecutionPlan plan = new TornadoExecutionPlan(itg)) {
            TornadoExecutionResult result = plan
                .withDevice(TornadoExecutionPlan.getDevice(0, 0))
                .withProfiler(ProfilerMode.SILENT)
                .withWarmUpIterations(3)
                .execute();

            System.out.println("Kernel time (ns): " +
                result.getProfilerResult().getDeviceKernelTime());
        }
    }
}

Task Graph

Build and configure the task graphs that feed this execution plan.

Worker Grid

Control thread block dimensions with WorkerGrid and GridScheduler.

Core API

Data Types

Library Tasks

TornadoExecutionPlan: Control GPU Execution Policy

ImmutableTaskGraph

Constructor

Core Execution

Device Selection

Grid Scheduler

Profiling

ProfilerMode Values

Warm-Up

Batch Execution

CUDA-Specific Features

Memory Management

TornadoExecutionResult

Intra-Plan Graph Selection

AutoCloseable and Resource Management

Complete Fluent Example

Task Graph

Worker Grid

Build docs developers (and LLMs) love

Core API

Data Types

Library Tasks

Documentation Index

​ImmutableTaskGraph

​Constructor

​Core Execution

​Device Selection

​Grid Scheduler

​Profiling

​ProfilerMode Values

​Warm-Up

​Batch Execution

​CUDA-Specific Features

​Memory Management

​TornadoExecutionResult

​Intra-Plan Graph Selection

​AutoCloseable and Resource Management

​Complete Fluent Example

Task Graph

Worker Grid

Build docs developers (and LLMs) love

ImmutableTaskGraph

Constructor

Core Execution

Device Selection

Grid Scheduler

Profiling

ProfilerMode Values

Warm-Up

Batch Execution

CUDA-Specific Features

Memory Management

TornadoExecutionResult

Intra-Plan Graph Selection

AutoCloseable and Resource Management

Complete Fluent Example