TornadoVM Runtime Flags: Complete Reference Guide

TornadoVM exposes two parallel systems for controlling runtime behaviour: CLI flags passed directly to the tornado Python wrapper (e.g. --printKernel), and JVM system properties passed with the -D prefix via --jvm (e.g. -Dtornado.printKernel=true). CLI flags are convenience shortcuts — each one maps internally to one or more -D properties. The properties themselves are all defined in TornadoOptions.java inside the tornado-runtime module. Understanding which flag to use and when is key to diagnosing compilation failures, profiling kernel execution time, and tuning device scheduling.

In flag examples below, s0 refers to a task graph name and t0 to a specific task within that graph. These match the names you use when constructing TaskGraph objects in your application code.

Basic Usage

# Using a CLI flag
tornado --printKernel -m tornado.examples/uk.ac.manchester.tornado.examples.compute.Montecarlo 1024

# Using a JVM property via --jvm
tornado --jvm "-Dtornado.printKernel=true" \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.Montecarlo 1024

# Combining multiple flags
tornado --debug --printKernel --threadInfo \
  --jvm "-Dtornado.fullDebug=true" \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.Montecarlo 1024

Debugging and Logging

CLI Flags
JVM Properties

Flag	Description
`--fullDebug`	Enables full debug mode. Maps to `-Dtornado.fullDebug=true`.
`--debug`	Enables basic debug output: compilation status, device info, execution events.
`--printKernel`	Prints the generated OpenCL/CUDA/Metal kernel source to stdout.
`--threadInfo`	Displays the number of threads (global and local work sizes) used for each kernel launch.
`--devices`	Lists all available hardware devices across all enabled backends.

Property	Description
`-Dtornado.fullDebug=true`	Full debug output including bytecode, runtime internals, and compilation events.
`-Dtornado.printKernel=true`	Prints generated OpenCL/CUDA kernel source to stdout.
`-Dtornado.print.kernel.dir=FILENAME`	Saves generated kernel source to the specified file instead of stdout.
`-Dtornado.threadInfo=true`	Displays the number of threads used per kernel launch.
`-Dtornado.print.bytecodes=true`	Prints TornadoVM internal bytecodes to stdout.
`-Dtornado.dump.bytecodes.dir=FILENAME`	Dumps TornadoVM internal bytecodes to the specified file.

Debugging Workflow: Printing Generated Kernels

When a kernel behaves unexpectedly, the first step is to inspect the generated source:

tornado --printKernel \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 128

To save the kernel for further analysis:

tornado --jvm "-Dtornado.print.kernel.dir=/tmp/matmul.cl" \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 128

Profiling

CLI Flags
JVM Properties

Flag	Description
`--enableProfiler console`	Enables profiling and prints metrics as JSON to stdout after execution.
`--enableProfiler silent`	Collects profiling metrics internally; access them via the TornadoVM Profiler API.
`--dumpProfiler FILENAME`	Saves profiling output (JSON) to the specified file.

Property	Description
`-Dtornado.profiler=true`	Enables profiling and prints metrics as JSON to stdout.
`-Dtornado.log.profiler=true`	Collects profiling metrics internally without printing.
`-Dtornado.profiler.dump.dir=FILENAME`	Saves profiling output to the specified file.

Profiling Workflow: Measuring Kernel Execution Time

# Print profiling JSON to console
tornado --enableProfiler console \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.NBody 1024 1

# Save profiling results to file
tornado --enableProfiler console --dumpProfiler /tmp/nbody-profile.json \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.NBody 1024 1

The JSON output includes fields for task-graph total time, kernel execution time, data-transfer time (host-to-device and device-to-host), and compilation time.

Device Selection

CLI Flags
JVM Properties

Flag	Description
`--devices`	Prints all available backends and devices with their index pairs (`B:D`).

Property	Description
`-Dtornado.backend=N`	Sets the default backend index when none is explicitly selected (default: `0`).
`-Dtornado.device=N`	Sets the default device index within the chosen backend (default: `0`).
`-Ds0.t0.device=B:D`	Assigns task `t0` in task graph `s0` to backend `B`, device `D`.
`-Dtornado.{opencl,cuda,metal}.priority=X`	Sets backend priority. Higher value wins when multiple backends can run a task. Defaults: OpenCL=10, CUDA=0, Metal=0.

# List all available devices first
tornado --devices

# Example output:
# Number of Tornado drivers: 2
# Driver: OpenCL
#   0: 0 -- GeForce GTX 1050
#   0: 1 -- Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
# Driver: CUDA
#   1: 0 -- GeForce GTX 1050

# Force task t0 in graph s0 onto CUDA device 0
tornado --jvm "-Ds0.t0.device=1:0" \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 512

Performance and Scheduling

Work-group size tuning

Property	Description
`-Ds0.t0.global.workgroup.size=X,Y,Z`	Sets a custom global work-group size for task `t0` in graph `s0`.
`-Ds0.t0.local.workgroup.size=X,Y,Z`	Sets a custom local (thread-block) work-group size.
`-Dtornado.scheduler.block=true`	Partitions the iteration space into blocks, one per visible CPU core (default: `false`). Useful for CPU OpenCL devices.
`-Dtornado.concurrent.devices=true`	Enables concurrent execution of tasks across multiple devices (default: `false`).

Memory and buffer management

Property	Description
`-Dtornado.reuse.device.buffers=false`	Disables reusing device buffers across executions of the same task graph (default: `true`).
`-Dtornado.deallocate.buffers=false`	Disables freeing device resources when the execution plan closes (default: `true`).
`-Dtornado.ns.time=true`	Uses nanoseconds for all timing measurements (default: `true`).

Optimizations

Math and FMA optimizations

Property	Default	Description
`-Dtornado.enable.fma=true`	`true`	Enables fused multiply-add instructions. May cause precision differences on some platforms.
`-Dtornado.enable.mathOptimizations=true`	`true`	Enables math simplifications, e.g. `1/sqrt(x)` → `rsqrt`.
`-Dtornado.enable.fastMathOptimizations=true`	`true`	Enables aggressive fast-math optimisations.
`-Dtornado.enable.nativeFunctions=true`	`true`	Enables native math intrinsics in the generated kernel.
`-Dtornado.experimental.partial.unroll=true`	`false`	Enables partial loop unrolling. Combine with `-Dtornado.unroll.factor=N` (default: `4`).

CUDA C Backend

These flags apply only when using the CUDA C backend (make BACKEND=cuda).

CUDA compiler flags

Property	Description
`-Dtornado.cuda.compile.profile=PROFILE`	Named NVRTC compilation profile. Options: `default` (no extra flags), `fast` (`--use_fast_math --extra-device-vectorization`), `debug` (`-lineinfo` for Nsight profiling), `repro` (`--fmad=false` for reproducibility).
`-Dtornado.cuda.compiler.flags=FLAGS`	Appends additional flags to the NVRTC compiler invocation. Applied after the profile flags.
`-Dtornado.cuda.host.pinning=false`	Disables host memory pinning for host↔device transfers (default: `true`). Pinned memory enables faster DMA transfers.

Code cache (shared with OpenCL)

The CUDA backend’s code cache properties still carry the opencl prefix — inherited from the OpenCL backend’s implementation and not yet renamed. These apply to both backends.

Property	Default	Description
`-Dtornado.opencl.codecache.enable=true`	`true`	Enables the on-disk kernel code cache.
`-Dtornado.opencl.codecache.dump=true`	`false`	Dumps compiled binaries to the cache directory.
`-Dtornado.opencl.source.dump=true`	`false`	Dumps generated source files to the source directory.
`-Dtornado.opencl.codecache.dir=PATH`	`/var/opencl-codecache`	Directory for compiled kernel cache.
`-Dtornado.opencl.source.dir=PATH`	`/var/opencl-compiler`	Directory for generated kernel sources.
`-Dtornado.opencl.log.dir=PATH`	`/var/opencl-logs`	Directory for OpenCL/CUDA compiler logs.

Metal Backend (macOS / Apple Silicon)

These flags apply only when using the Metal backend (make BACKEND=metal) on macOS.

Property	Default	Description
`-Dtornado.metal.fastmath=true`	`false`	Compiles Metal kernels with fast/relaxed math. Trades some FP precision for speed.
`-Dtornado.metal.threadgroupHint=true`	`false`	Emits a `max_total_threads_per_threadgroup` attribute when the local work-group size is statically known, helping the Metal compiler tune occupancy.
`-Dtornado.metal.profiling.enable=false`	`true`	Disables Metal GPU profiling.
`-Dtornado.metal.compiler.flags=FLAGS`	(none)	Passes additional flags to the Metal compiler.

Common Debugging Workflows

Inspect the generated kernel

tornado --printKernel \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 256

Look for correct loop bounds, data types (float vs double), and that @Parallel loops map to the expected get_global_id() calls.

Check thread and work-group configuration

tornado --threadInfo \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 256

Verify that the global work size matches your array dimensions and that the local work size is a sensible divisor.

Profile execution time breakdown

tornado --enableProfiler console \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 256

The JSON output separates kernel time from data-transfer time, helping you identify whether your bottleneck is compute or PCIe bandwidth.

Enable full debug for compiler diagnostics

tornado --fullDebug \
  --jvm "-Dtornado.print.bytecodes=true" \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 256

This prints internal TornadoVM bytecodes before kernel compilation — useful when reporting compiler bugs or understanding how Java code maps to GPU operations.

Force execution on a specific device

# List devices first
tornado --devices

# Pin task t0 of graph s0 to device 0:1
tornado --jvm "-Ds0.t0.device=0:1" \
  -m tornado.examples/uk.ac.manchester.tornado.examples.compute.MatrixMultiplication2D 256

Quick-Reference: CLI Flags Summary

All CLI flags at a glance

CLI Flag	JVM Equivalent	Category
`--fullDebug`	`-Dtornado.fullDebug=true`	Debug
`--debug`	`-Dtornado.debug=true`	Debug
`--printKernel`	`-Dtornado.printKernel=true`	Debug
`--threadInfo`	`-Dtornado.threadInfo=true`	Debug
`--devices`	(prints device list and exits)	Device info
`--enableProfiler console`	`-Dtornado.profiler=true`	Profiling
`--enableProfiler silent`	`-Dtornado.log.profiler=true`	Profiling
`--dumpProfiler FILE`	`-Dtornado.profiler.dump.dir=FILE`	Profiling
`--jvm "FLAGS"`	(passes FLAGS directly to the JVM)	JVM pass-through
`-cp CLASSPATH`	(classpath)	Launch
`-m MODULE/CLASS`	(module-qualified main class)	Launch

Get Started

Programming Guide

Backends

NVIDIA Integration

Guides

Contributing

TornadoVM Runtime Flags: Complete Reference Guide

Basic Usage

Debugging and Logging

Debugging Workflow: Printing Generated Kernels

Profiling

Profiling Workflow: Measuring Kernel Execution Time

Device Selection

Performance and Scheduling

Optimizations

CUDA C Backend

Metal Backend (macOS / Apple Silicon)

Common Debugging Workflows

Quick-Reference: CLI Flags Summary

Build docs developers (and LLMs) love

Get Started

Programming Guide

Backends

NVIDIA Integration

Guides

Contributing

Documentation Index

​Basic Usage

​Debugging and Logging

​Debugging Workflow: Printing Generated Kernels

​Profiling

​Profiling Workflow: Measuring Kernel Execution Time

​Device Selection

​Performance and Scheduling

​Optimizations

​CUDA C Backend

​Metal Backend (macOS / Apple Silicon)

​Common Debugging Workflows

​Quick-Reference: CLI Flags Summary

Build docs developers (and LLMs) love

Basic Usage

Debugging and Logging

Debugging Workflow: Printing Generated Kernels

Profiling

Profiling Workflow: Measuring Kernel Execution Time

Device Selection

Performance and Scheduling

Optimizations

CUDA C Backend

Metal Backend (macOS / Apple Silicon)

Common Debugging Workflows

Quick-Reference: CLI Flags Summary