Skip to main content
This guide covers building all Delta Sharing components from source code. You can build individual components or the entire project depending on your needs.

​
Prerequisites

Before building Delta Sharing from source, ensure you have the following installed:

Python

Python 3.8+ (or 3.6+ for older versions)

Java

Java 8+ for building Scala components

Scala

Scala 2.12.x or 2.13.x

Rust

Rust toolchain (if building delta-kernel-rust from source)
If you’re on Linux, ensure glibc version >= 2.31 for automatic delta-kernel-rust-sharing-wrapper installation.

​
Building Python Connector

The Python connector can be built and installed in several ways depending on your use case.

​
Development Mode Installation

For active development, install the package in editable mode. This allows you to make changes to the code and test them immediately without reinstalling:
1

Navigate to Python directory

cd python/
2

Install in develop mode

pip install -e .
This creates a symlink to your source code, so any changes you make are immediately reflected.

​
Local Installation

For a standard local installation: 
cd python/
pip install .

​
Building a Wheel Package

To create a distributable wheel file: 
cd python/
python setup.py sdist bdist_wheel
This generates python/dist/delta_sharing-x.y.z-py3-none-any.whl that can be distributed and installed on other systems.

​
Handling Rust Dependencies

If PyPI doesn’t have a pre-built wheel for delta-kernel-rust-sharing-wrapper for your environment, pip will build from source, requiring Rust to be installed.
To manually build the delta-kernel-rust-sharing-wrapper: 
cd python/delta-kernel-rust-sharing-wrapper
python3 -m venv .venv
source .venv/bin/activate
pip3 install maturin
maturin develop
If you encounter issues:
  • Verify Python version >= 3.8
  • Upgrade pip to the latest version: pip3 install --upgrade pip
  • Check Linux glibc version >= 2.31
  • Install Rust if needed

​
Building Spark Connector

The Apache Spark connector is built using SBT (Scala Build Tool).
1

Compile the project

build/sbt compile
This compiles all Scala source files in the Spark connector.
2

Package the connector

build/sbt spark/package
This creates the JAR file at:
spark/target/scala-2.12/delta-sharing-spark_2.12-x.y.z.jar
3

Use the connector

You can now use this JAR file with Spark:
spark-shell --jars spark/target/scala-2.12/delta-sharing-spark_2.12-x.y.z.jar

​
Building for Multiple Scala Versions

The project supports both Scala 2.12 and 2.13. To build for a specific version: 
# For Scala 2.12
build/sbt ++2.12.18 spark/package

# For Scala 2.13
build/sbt ++2.13.13 spark/package

​
Building Delta Sharing Server

The reference server can be built as a packaged distribution or Docker image.

​
Building the Server Package

Create a complete server distribution package:
1

Generate the package

build/sbt server/universal:packageBin
2

Locate the package

The build generates:
server/target/universal/delta-sharing-server-x.y.z.zip
3

Extract and configure

unzip server/target/universal/delta-sharing-server-x.y.z.zip
cd delta-sharing-server-x.y.z
cp conf/delta-sharing-server.yaml.template conf/delta-sharing-server.yaml
# Edit conf/delta-sharing-server.yaml with your configuration
4

Run the server

bin/delta-sharing-server -- --config conf/delta-sharing-server.yaml

​
Building the Docker Image

Build a local Docker image for containerized deployment: 
build/sbt server/docker:publishLocal
This creates a Docker image tagged delta-sharing-server:x.y.z. Run it with: 
docker run -p <host-port>:<container-port> \
  --mount type=bind,source=<path-to-config.yaml>,target=/config/delta-sharing-server-config.yaml \
  delta-sharing-server:x.y.z -- --config /config/delta-sharing-server-config.yaml
The <container-port> must match the port specified in your configuration file.

​
Multi-platform Docker Images

The server supports building for multiple platforms using Docker Buildx: 
build/sbt server/docker:publish
By default, this builds for:
  • linux/arm64 (Apple Silicon, ARM servers)
  • linux/amd64 (Intel/AMD x86-64)

​
Building All Components

# Compile all Scala components
build/sbt compile

​
Common Build Issues

Solution: Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
Or use an older delta-sharing version that doesn’t require Rust:
pip3 install delta-sharing==1.0.5
Solution: Increase JVM memory for SBT
export SBT_OPTS="-Xmx2G -XX:+UseConcMarkSweepGC"
build/sbt compile
Solution: Verify Docker is installed and running
docker --version
docker info
Ensure you have sufficient disk space for the image build.
Solution: Ensure all test dependencies are availableFor Python:
pip install pytest pytest-cov
For Scala, dependencies are managed by SBT automatically.

​
Advanced SBT Commands

For more control over the build process, you can use these SBT commands: 
# Build for all supported Scala versions
build/sbt +publishLocal

​
Next Steps

Testing

Run comprehensive tests on your builds

Contributing

Learn how to contribute your changes

Configuration

Configure the Delta Sharing Server

Deployment

Deploy your built components

Build docs developers (and LLMs) love

Get started for freeTalk to us