Off-Heap Data Types for GPU-Ready Memory in TornadoVM

Standard Java arrays live on the JVM heap, which means the garbage collector can move or scan them at any moment. When TornadoVM needs to transfer data to a GPU it must either pin the array (blocking GC) or copy it to a separate native buffer — either way introducing latency and unpredictability. TornadoVM solves this by providing a set of off-heap data types built on Java’s Foreign Function & Memory API (Project Panama). These types allocate a MemorySegment — a contiguous block of native memory outside the JVM heap — that the TornadoVM runtime can map directly to device buffers, eliminate unnecessary copies, and free at deterministic points without GC involvement. Starting from TornadoVM v1.0, using these types is the recommended approach for all GPU-accelerated code.

Primitive Array Types

Each Java primitive array type has a direct off-heap counterpart:

Java heap array	TornadoVM off-heap type
`int[]`	`IntArray`
`float[]`	`FloatArray`
`double[]`	`DoubleArray`
`long[]`	`LongArray`
`short[]`	`ShortArray`
`byte[]`	`ByteArray`
`char[]`	`CharArray`
`HalfFloat[]`	`HalfFloatArray`

All types reside in uk.ac.manchester.tornado.api.types.arrays.

Unlike Java primitive arrays, off-heap types are not zero-initialised on allocation. Always call .clear() or .init(value) before use unless you immediately overwrite every element.

Creating Off-Heap Arrays

Constructor
fromArray (copy from heap)
fromElements (varargs)
fromSegment (Memory API)

Allocate a segment for a fixed number of elements. Contents are uninitialised.

import uk.ac.manchester.tornado.api.types.arrays.FloatArray;
import uk.ac.manchester.tornado.api.types.arrays.IntArray;

FloatArray floats = new FloatArray(1024);  // 1024 floats, uninitialised
IntArray   ints   = new IntArray(512);     // 512  ints,   uninitialised

Copy the contents of an existing Java primitive array into a new off-heap segment.

float[] heapData = {1.0f, 2.0f, 3.0f, 4.0f};
FloatArray offHeap = FloatArray.fromArray(heapData);

Construct directly from inline values — useful for small constants and test fixtures.

FloatArray weights = FloatArray.fromElements(0.1f, 0.5f, 0.4f);
IntArray   ids     = IntArray.fromElements(0, 1, 2, 3, 4);

Wrap an existing MemorySegment from Project Panama.

import java.lang.foreign.MemorySegment;

MemorySegment seg = ...;
FloatArray wrapped = FloatArray.fromSegment(seg);

Core Instance Methods

The following methods are available on all off-heap array types. The examples below use FloatArray, but the API is identical for IntArray, DoubleArray, LongArray, ShortArray, ByteArray, and CharArray.

get(int index) — read a value

FloatArray arr = FloatArray.fromElements(10.0f, 20.0f, 30.0f);
float val = arr.get(1);  // returns 20.0f

set(int index, T value) — write a value

FloatArray arr = new FloatArray(4);
arr.set(0, 3.14f);
arr.set(1, 2.71f);

init(T value) — fill the entire array with one value

FloatArray arr = new FloatArray(1024);
arr.init(1.0f);   // all 1024 elements are now 1.0f

clear() — zero all elements

FloatArray arr = new FloatArray(1024);
arr.clear();      // all 1024 elements are now 0.0f

getSize() — element count

FloatArray arr = new FloatArray(256);
int n = arr.getSize();  // 256

toHeapArray() — convert back to Java array

FloatArray offHeap = FloatArray.fromElements(1.0f, 2.0f, 3.0f);
float[] heap = offHeap.toHeapArray();  // creates a new float[] copy

getNumBytesOfSegment() / getNumBytesOfSegmentWithHeader()

FloatArray arr = new FloatArray(16);
long dataBytes   = arr.getNumBytesOfSegment();           // 64 bytes (16 × 4)
long totalBytes  = arr.getNumBytesOfSegmentWithHeader(); // includes header

Migration from Java Primitive Arrays

Migrating existing TornadoVM code from on-heap primitive arrays to the off-heap API requires three targeted changes:

Replace array declarations

// Before
float[] input = new float[numElements];
Arrays.fill(input, 10.0f);

// After
FloatArray input = new FloatArray(numElements);
input.init(10.0f);

Update element access in kernel methods

// Before
public static void process(float[] input, int value) {
    for (@Parallel int i = 0; i < input.length; i++) {
        input[i] = input[i] + value;
    }
}

// After
public static void process(FloatArray input, int value) {
    for (@Parallel int i = 0; i < input.getSize(); i++) {
        input.set(i, input.get(i) + value);
    }
}

Update TaskGraph declarations (no changes needed)

The TaskGraph API accepts both old-style arrays and new off-heap types using the same transferToDevice / transferToHost / task calls — no changes required in that layer.

TaskGraph tg = new TaskGraph("s0")
    .transferToDevice(DataTransferMode.FIRST_EXECUTION, input) // FloatArray
    .task("t0", Example::process, input, 1)
    .transferToHost(DataTransferMode.EVERY_EXECUTION, input);

Vector Types

TornadoVM ships SIMD-aware vector types that map to GPU hardware vector registers. Vector types use stack-like construction and support element-wise math operations. All vector types reside in uk.ac.manchester.tornado.api.types.vectors.

Float vectors

Float2, Float3, Float4, Float8, Float16

Int vectors

Int2, Int3, Int4, Int8, Int16

Double vectors

Double2, Double3, Double4, Double8, Double16

Creating and Using Vector Types

import uk.ac.manchester.tornado.api.types.vectors.Float2;
import uk.ac.manchester.tornado.api.types.vectors.Float4;

// Construct with component values
Float2 v2 = new Float2(1.0f, 2.0f);
Float4 v4 = new Float4(1.0f, 2.0f, 3.0f, 4.0f);

// Read components
float x = v4.getX();
float y = v4.getY();
float z = v4.getZ();
float w = v4.getW();

// Set a component
v4.setX(10.0f);

// Arithmetic (static methods)
Float2 sum  = Float2.add(v2, new Float2(3.0f, 4.0f));
Float4 diff = Float4.sub(v4, new Float4(1.0f, 1.0f, 1.0f, 1.0f));

Vector types also support dot products and cross products (where dimensionally appropriate) via TornadoMath utilities.

Matrix Types

TornadoVM provides 2D and 3D matrix types built on top of FloatArray, DoubleArray, and IntArray.

Type	Package
`Matrix2DFloat`, `Matrix2DDouble`, `Matrix2DInt`	`uk.ac.manchester.tornado.api.types.matrix`
`Matrix3DFloat`, `Matrix3DDouble`, `Matrix3DInt`	`uk.ac.manchester.tornado.api.types.matrix`
`Matrix2DFloat4`	`uk.ac.manchester.tornado.api.types.matrix`

Matrix Usage Example

import uk.ac.manchester.tornado.api.types.matrix.Matrix2DFloat;

int rows = 512, cols = 512;
Matrix2DFloat matA = new Matrix2DFloat(rows, cols);
Matrix2DFloat matB = new Matrix2DFloat(rows, cols);
Matrix2DFloat matC = new Matrix2DFloat(rows, cols);

// Set and get individual elements
matA.set(0, 0, 1.0f);
float val = matA.get(0, 0);  // 1.0f

// Matrices passed to TaskGraph exactly like flat arrays
TaskGraph tg = new TaskGraph("mxm")
    .transferToDevice(DataTransferMode.FIRST_EXECUTION, matA, matB)
    .task("t0", MyKernels::matMul, matA, matB, matC, rows)
    .transferToHost(DataTransferMode.EVERY_EXECUTION, matC);

Matrix2DFloat internally stores data in a flat FloatArray in row-major order. You can access the backing store via .getArray() if you need to pass the raw segment to another API.

Zero-Copy Semantics

When a TornadoVM off-heap array is registered with a TaskGraph, the runtime maps the underlying MemorySegment directly to the corresponding device buffer. This means:

No intermediate copy is required when transferring to device — the native memory is passed to the OpenCL / CUDA driver as-is.
Pinning is deterministic — the segment is pinned only during the actual transfer, not throughout the JVM session.
No GC interference — because the segment is outside the Java heap, the garbage collector never moves or invalidates the backing memory.

// Both of these operations work directly with the native MemorySegment
floatArray.set(0, 42.0f);               // CPU write to native memory
tg.transferToDevice(EVERY_EXECUTION, floatArray); // driver reads from same native memory

If you need to process GPU results with existing Java libraries that expect float[] or double[], use .toHeapArray() to create a heap copy after the transferToHost completes. For performance-critical paths where you loop over results, prefer get(i) directly on the off-heap type.

Utility Factory Methods

Beyond the primary constructors, off-heap arrays expose several convenience factories:

// Concatenate multiple FloatArrays into one contiguous array
FloatArray merged = FloatArray.concat(part1, part2, part3);

// Extract a view (slice) over a sub-range without copying
FloatArray view = merged.slice(offset, length);

// Construct from a NIO FloatBuffer
java.nio.FloatBuffer buf = FloatBuffer.allocate(64);
FloatArray fromBuf = FloatArray.fromFloatBuffer(buf);

slice() returns a view of the original segment — modifying the slice modifies the source array. Use it for read-only access patterns or when you intentionally want aliased writes.

Get Started

Programming Guide

Backends

NVIDIA Integration

Guides

Contributing

Off-Heap Data Types for GPU-Ready Memory in TornadoVM

Primitive Array Types

Creating Off-Heap Arrays

Core Instance Methods

Migration from Java Primitive Arrays

Vector Types

Float vectors

Int vectors

Double vectors

Creating and Using Vector Types

Matrix Types

Matrix Usage Example

Zero-Copy Semantics

Utility Factory Methods

Build docs developers (and LLMs) love

Get Started

Programming Guide

Backends

NVIDIA Integration

Guides

Contributing

Documentation Index

​Primitive Array Types

​Creating Off-Heap Arrays

​Core Instance Methods

​Migration from Java Primitive Arrays

​Vector Types

Float vectors

Int vectors

Double vectors

​Creating and Using Vector Types

​Matrix Types

​Matrix Usage Example

​Zero-Copy Semantics

​Utility Factory Methods

Build docs developers (and LLMs) love

Primitive Array Types

Creating Off-Heap Arrays

Core Instance Methods

Migration from Java Primitive Arrays

Vector Types

Creating and Using Vector Types

Matrix Types

Matrix Usage Example

Zero-Copy Semantics

Utility Factory Methods