Benchmarking TornadoVM: Tools, Benchmarks, and Tips

Measuring GPU acceleration accurately is harder than it looks: JIT compilation, driver warm-up, data-transfer overhead, and JVM garbage collection can all skew results if you measure naively. TornadoVM ships a dedicated benchmarking module — tornado.benchmarks — that handles warm-up, iteration counting, and baseline comparison automatically. The suite covers a diverse set of compute kernels so you can evaluate TornadoVM across different memory access patterns, arithmetic intensities, and data sizes. This guide explains how to run the standard benchmarks, interpret the output, extend the suite with your own kernels, and use JMH for micro-benchmarking.

The Benchmark Suite

The tornado-benchmarks module lives at <tornadovm-root>/tornado-benchmarks/ and is compiled as part of the standard TornadoVM build. All benchmarks follow the same structure: a pure-Java sequential baseline alongside a TornadoVM-accelerated version, iterated a configurable number of times with the results reported as average, median, best, and speedup over the sequential baseline.

Available Benchmarks

saxpy

Single-precision a*x + y vector operation. Classic memory-bandwidth benchmark.

addImage

Element-wise addition of two 2D image buffers.

stencil

1D stencil operation — tests cache-friendly sequential access patterns.

convolvearray / convolveimage

2D convolution on flat arrays and image types. Tests spatial locality.

blackscholes

Black-Scholes option pricing — embarrassingly parallel, heavy on transcendentals.

montecarlo

Monte Carlo π estimation — independent random trials, highly parallel.

blurFilter

Gaussian blur on a 2D image — tests 2D stencil with a larger footprint.

nbody

N-body gravitational simulation — O(n²) all-pairs, high arithmetic intensity.

sgemm / dgemm

Single and double-precision matrix–matrix multiplication.

mandelbrot

Mandelbrot set rendering — divergent control flow, per-pixel independence.

dft

Discrete Fourier Transform — O(n²) reference implementation.

euler / renderTrack

Euler integration and ray-casting render tracking kernels.

Running Benchmarks

The `tornado-benchmarks.py` Runner

After building TornadoVM and sourcing the environment, the benchmark runner is available on your PATH:

source setvars.sh
tornado-benchmarks.py --help

usage: tornado-benchmarks.py [-h] [--validate] [--default] [--medium]
                             [--iterations ITERATIONS] [--full]
                             [--skipSequential] [--skipParallel]
                             [--skipDevices SKIP_DEVICES] [--verbose]
                             [--printBenchmarks]

Tool to execute benchmarks in TornadoVM. With no options, it runs all
benchmarks with the default size

optional arguments:
  -h, --help            show this help message and exit
  --validate            Enable result validation
  --default             Run default benchmark configuration
  --medium              Run benchmarks with medium sizes
  --iterations ITERATIONS
                        Set the number of iterations
  --full                Run for all sizes in all devices. Including big data
                        sizes
  --skipSequential      Skip java version
  --skipParallel        Skip parallel version
  --skipDevices SKIP_DEVICES
                        Skip devices. Provide a list of devices (e.g., 0,1)
  --verbose, -V         Enable verbose
  --printBenchmarks     Print the list of available benchmarks
  --jmh                 Run with JMH

Common Invocations

# Runs all benchmarks on all detected devices with default data sizes
# Takes 30–60 minutes
tornado-benchmarks.py

Interpreting Benchmark Output

Each benchmark prints one result line per device. Here is an annotated example:

bm=convolve-array-100-2048-2048-5, id=java-reference, average=2.612301e+08, median=2.609304e+08, firstIteration=4.006838e+08, best=2.544892e+08
bm=convolve-array-100-2048-2048-5, device=0:0, average=8.143104e+06, median=8.214443e+06, firstIteration=1.811648e+07, best=7.609697e+06, speedupAvg=32.0799, speedupMedian=31.7648, speedupFirstIteration=22.1171, CV=4.6348%, deviceName=NVIDIA CUDA -- GeForce GTX 1050

Field definitions

Field	Meaning
`bm=convolve-array-100-2048-2048-5`	Benchmark name, iteration count (100), data dimensions (2048×2048), filter size (5)
`id=java-reference`	This line is the pure-Java sequential baseline
`device=0:0`	TornadoVM backend index : device index (see `--devices`)
`average`	Mean execution time in nanoseconds across all iterations
`median`	Median execution time — more robust to outliers than mean
`firstIteration`	First-iteration time — includes JIT compilation and driver warm-up
`best`	Fastest single iteration — approximates peak throughput
`speedupAvg`	`java-reference average / device average` — overall speedup
`speedupMedian`	Speedup based on median values
`speedupFirstIteration`	Speedup including warm-up cost — typically lower
`CV`	Coefficient of variation (std dev / mean × 100%) — measures stability
`deviceName`	Human-readable backend and device name

Pay attention to CV (coefficient of variation). Values above 15–20% indicate unstable measurements — consider increasing --iterations or excluding first-iteration timings from your analysis.

JMH Integration

For micro-benchmarks following the Java Microbenchmark Harness (JMH) methodology, the suite provides a --jmh flag and per-benchmark JMH entry points.

Running All Benchmarks with JMH

# Full JMH run — takes approximately 3.5 hours
tornado-benchmarks.py --jmh

Running a Single Benchmark via JMH

Each benchmark has a dedicated JMH entry point following the naming convention JMH<BENCHMARK>:

# Run the DFT benchmark with JMH (~10 minutes)
tornado -m tornado.benchmarks/uk.ac.manchester.tornado.benchmarks.dft.JMHDFT

Example JMH output:

# JMH version: 1.23
# VM invoker: .../bin/java
# Benchmark: uk.ac.manchester.tornado.benchmarks.dft.JMHDFT.dftTornado

Benchmark          Mode  Cnt   Score   Error  Units
JMHDFT.dftJava     avgt    5  19.736 ± 1.589   s/op
JMHDFT.dftTornado  avgt    5   0.155 ± 0.008   s/op

This shows a 127× speedup for DFT on the test GPU — JMH’s averaged throughput mode (avgt) minimises JIT and GC noise better than a manual timing loop.

Writing Custom Benchmarks

Custom benchmarks extend the BenchmarkDriver base class, which provides the iteration loop, timing infrastructure, and result reporting automatically.

Extend BenchmarkDriver

Create your benchmark class by extending uk.ac.manchester.tornado.benchmarks.BenchmarkDriver:

package com.example.benchmarks;

import uk.ac.manchester.tornado.api.*;
import uk.ac.manchester.tornado.api.types.arrays.FloatArray;
import uk.ac.manchester.tornado.benchmarks.BenchmarkDriver;

public class VectorScaleBenchmark extends BenchmarkDriver {

    private final int size;
    private FloatArray input;
    private FloatArray output;
    private ImmutableTaskGraph immutableTaskGraph;
    private TornadoExecutionPlan executionPlan;

    public VectorScaleBenchmark(int iterations, int size) {
        super(iterations);
        this.size = size;
    }

    @Override
    public void setUp() {
        input  = new FloatArray(size);
        output = new FloatArray(size);

        // Initialise input data
        for (int i = 0; i < size; i++) {
            input.set(i, (float) i);
        }

        // Build task graph
        TaskGraph taskGraph = new TaskGraph("benchmark")
            .transferToDevice(DataTransferMode.FIRST_EXECUTION, input)
            .task("scale", VectorScaleBenchmark::scale, input, output, 2.0f)
            .transferToHost(DataTransferMode.EVERY_EXECUTION, output);

        immutableTaskGraph = taskGraph.snapshot();
        executionPlan = new TornadoExecutionPlan(immutableTaskGraph);
    }

    // The kernel — annotated for TornadoVM
    public static void scale(FloatArray in, FloatArray out, float factor) {
        for (@Parallel int i = 0; i < in.getSize(); i++) {
            out.set(i, in.get(i) * factor);
        }
    }

    @Override
    public void runBenchmark(TornadoDevice device) {
        executionPlan.withDevice(device).execute();
    }

    @Override
    public void tearDown() {
        executionPlan.freeDeviceMemory();
    }

    @Override
    public boolean validate(TornadoDevice device) {
        for (int i = 0; i < size; i++) {
            if (Math.abs(output.get(i) - (i * 2.0f)) > 1e-4f) return false;
        }
        return true;
    }
}

Run your custom benchmark

tornado -cp myapp.jar \
  com.example.benchmarks.VectorScaleBenchmark 100 1048576

The first argument is the number of iterations, the second is the data size.

Performance Tips

Choose the right data size

GPU kernels have a fixed overhead for kernel launch, JIT compilation, and PCIe data transfer. For most benchmarks, meaningful speedups only appear above 256K–1M elements. Always verify that your working set fits in GPU memory.

Allow enough warm-up iterations

The first iteration includes LLVM/NVRTC kernel compilation time and can be 10–100× slower than steady-state. Exclude firstIteration timings from speedup analysis, or use JMH which handles warm-up automatically. The default benchmark runner uses 131 iterations (approximately 130 measured iterations).

Understand JIT compilation effects

TornadoVM compiles kernels once per unique task graph configuration and caches them. Benchmarks that repeatedly change data size or device will re-trigger compilation. Enable the code cache with -Dtornado.opencl.codecache.enable=true to persist compiled kernels across JVM restarts.

Monitor CV for stability

A coefficient of variation (CV) above 15% in benchmark results usually means interference from the OS scheduler, GPU power management, or JVM GC. Try running with -Xms24G -Xmx24G -server to reduce GC pressure, and disable GPU Boost if your hardware supports it.

Compare against the right baseline

The java-reference baseline in the benchmark suite is a straightforward sequential Java loop — not a hand-tuned BLAS or multi-threaded baseline. For a fair comparison against optimised libraries, add your own baseline to the benchmark.

Get Started

Programming Guide

Backends

NVIDIA Integration

Guides

Contributing

Benchmarking TornadoVM: Tools, Benchmarks, and Tips

The Benchmark Suite

Available Benchmarks

saxpy

addImage

stencil

convolvearray / convolveimage

blackscholes

montecarlo

blurFilter

nbody

sgemm / dgemm

mandelbrot

dft

euler / renderTrack

Running Benchmarks

The `tornado-benchmarks.py` Runner

Common Invocations

Interpreting Benchmark Output

JMH Integration

Running All Benchmarks with JMH

Running a Single Benchmark via JMH

Writing Custom Benchmarks

Performance Tips

Build docs developers (and LLMs) love

Get Started

Programming Guide

Backends

NVIDIA Integration

Guides

Contributing

Documentation Index

​The Benchmark Suite

​Available Benchmarks

saxpy

addImage

stencil

convolvearray / convolveimage

blackscholes

montecarlo

blurFilter

nbody

sgemm / dgemm

mandelbrot

dft

euler / renderTrack

​Running Benchmarks

​The tornado-benchmarks.py Runner

​Common Invocations

​Interpreting Benchmark Output

​JMH Integration

​Running All Benchmarks with JMH

​Running a Single Benchmark via JMH

​Writing Custom Benchmarks

​Performance Tips

Build docs developers (and LLMs) love

The Benchmark Suite

Available Benchmarks

Running Benchmarks

The `tornado-benchmarks.py` Runner

Common Invocations

Interpreting Benchmark Output

JMH Integration

Running All Benchmarks with JMH

Running a Single Benchmark via JMH

Writing Custom Benchmarks

Performance Tips