Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/vortex-data/vortex/llms.txt

Use this file to discover all available pages before exploring further.

Vortex is an extensible, high-performance framework for columnar compression and file I/O. It combines a stable on-disk file format (.vortex) with a rich in-memory array ecosystem, enabling you to compress data once and query it directly — no decompression required for many operations.

Python Quickstart

Install vortex-data and read, write, and query Vortex files in Python in minutes.

Rust Quickstart

Add the vortex crate and get started with in-memory arrays and file I/O in Rust.

Core Concepts

Understand DTypes, Arrays, Encodings, Layouts, and the Scan API.

Query Engine Integrations

Use Vortex with DataFusion, DuckDB, Spark, Polars, and Ray.

​
Why Vortex?

Vortex is designed for the next generation of data systems backed by object storage. It separates logical types from physical encodings, allowing you to store the same data in dozens of compressed formats while using a single unified API to query it.

Blazing Fast

100x faster random access reads and 10–20x faster scans vs. modern Apache Parquet.

Extensible Architecture

Pluggable encodings, layouts, type system, and compression strategy — extend anything.

Zero-Copy Arrow

Seamless, zero-copy conversion to and from Apache Arrow arrays.

Stable File Format

The .vortex file format is stable since v0.36.0 with guaranteed backwards compatibility.

Multi-Language

First-class Rust, Python, Java, C, and C++ support.

Open Governance

An LF AI & Data incubation project. Apache-2.0 licensed.

​
Get started

1

Install Vortex

Install the library for your language of choice.
uv add vortex-data
# or
pip install vortex-data
2

Write a Vortex file

Create a compressed .vortex file from your data.
import vortex

array = vortex.array([1, 2, 3, 4, 5])
vortex.io.write(array, "data.vortex")
3

Read and query

Open the file and scan it with filter pushdown.
dataset = vortex.io.open("data.vortex")
result = dataset.filter(vortex.expr.column("value") > 2).to_arrow()
4

Integrate with your query engine

Connect Vortex to DataFusion, DuckDB, Spark, Polars, or Ray.See the Query Engine Integrations section to get started.

​
Explore the documentation

Core Concepts

Learn how DTypes, Arrays, Encodings, Layouts, and the Scan API fit together.

Extending Vortex

Write custom encodings, layouts, dtypes, and compute functions.

File Format Spec

Read the full specification for the .vortex file format.

Java Quickstart

Use Vortex with Spark, Trino, and other JVM-based engines.
The Vortex file format has been stable since v0.36.0. All future versions guarantee backwards compatibility — files written today will be readable by all future releases.

Build docs developers (and LLMs) love

Get started for freeTalk to us