Docker Deployment

Overview

The Delta Sharing Reference Server is available as a pre-built Docker image on Docker Hub. This provides a consistent, isolated environment for running the server with minimal setup.

Docker Hub Repository: deltaio/delta-sharing-server

Prerequisites

Install Docker

Ensure Docker is installed on your system:

docker --version

If not installed, download from:

Linux: Use your package manager or Docker Engine installation guide
macOS: Docker Desktop for Mac
Windows: Docker Desktop for Windows

Prepare Configuration File

Create a Delta Sharing server configuration file following the configuration guide:

mkdir -p ~/delta-sharing-config
cd ~/delta-sharing-config
nano delta-sharing-server.yaml

Configure Cloud Storage (Optional)

If using Azure Blob Storage, ADLS Gen2, or Cloudflare R2, create a core-site.xml file:

nano core-site.xml

See Cloud Storage Authentication for details.

Running the Docker Container

The basic command to run the Delta Sharing server in Docker:

docker run -p <host-port>:<container-port> \
  --mount type=bind,source=<path-to-config-file>,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

host-port

integer

required

The port on your host machine to expose (e.g., 8080)

container-port

integer

required

The port inside the container (must match the port in your config file)

source

string

required

Absolute path to your configuration file on the host system

The <container-port> must match the port value defined in your delta-sharing-server.yaml configuration file.

Basic Example

Run the server on port 8080 with a configuration file:

docker run -p 8080:8080 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Advanced Docker Run Options

Running in Detached Mode

Run the container in the background:

docker run -d \
  --name delta-sharing-server \
  -p 8080:8080 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

View logs:

docker logs -f delta-sharing-server

Stop the container:

docker stop delta-sharing-server
docker rm delta-sharing-server

Mounting Multiple Configuration Files

For Azure or Cloudflare R2, mount both the server config and core-site.xml:

docker run -p 8080:8080 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  --mount type=bind,source=$(pwd)/core-site.xml,target=/opt/docker/conf/core-site.xml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

The core-site.xml file must be mounted to /opt/docker/conf/core-site.xml inside the container.

Using Environment Variables

Pass cloud credentials via environment variables:

AWS S3
Google Cloud Storage
Bearer Token

docker run -p 8080:8080 \
  -e AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
  -e AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  -e AWS_REGION=us-west-2 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

docker run -p 8080:8080 \
  -e GOOGLE_APPLICATION_CREDENTIALS=/config/gcp-key.json \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  --mount type=bind,source=$(pwd)/gcp-service-account.json,target=/config/gcp-key.json \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

docker run -p 8080:8080 \
  -e BEARER_TOKEN=dapi1234567890abcdefghijklmnopqrstuvwxyz \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Setting Memory Limits

Control container resource usage:

docker run -d \
  --name delta-sharing-server \
  -p 8080:8080 \
  --memory="2g" \
  --cpus="2.0" \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Docker Compose

For easier management, use Docker Compose:

docker-compose.yml

version: '3.8'

services:
  delta-sharing-server:
    image: deltaio/delta-sharing-server:0.7.8
    container_name: delta-sharing-server
    ports:
      - "8080:8080"
    volumes:
      - ./delta-sharing-server.yaml:/config/delta-sharing-server-config.yaml:ro
      - ./core-site.xml:/opt/docker/conf/core-site.xml:ro  # Optional, for Azure/R2
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_REGION=us-west-2
    command: -- --config /config/delta-sharing-server-config.yaml
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/delta-sharing/shares"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Start the service:

docker-compose up -d

View logs:

docker-compose logs -f

Stop the service:

docker-compose down

Networking Modes

Bridge Network (Default)

The default bridge network isolates the container:

docker run -p 8080:8080 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Host Network

Use the host’s network stack directly (Linux only):

docker run --network host \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Host networking is not supported on Docker Desktop for Mac or Windows.

Custom Network

Create a custom network for better isolation:

# Create network
docker network create delta-sharing-net

# Run container on custom network
docker run -d \
  --name delta-sharing-server \
  --network delta-sharing-net \
  -p 8080:8080 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Reverse Proxy with Docker

Run NGINX as a reverse proxy with TLS termination:

docker-compose.yml

version: '3.8'

services:
  delta-sharing-server:
    image: deltaio/delta-sharing-server:0.7.8
    container_name: delta-sharing-server
    volumes:
      - ./delta-sharing-server.yaml:/config/delta-sharing-server-config.yaml:ro
    command: -- --config /config/delta-sharing-server-config.yaml
    networks:
      - internal
  
  nginx:
    image: nginx:alpine
    container_name: nginx-proxy
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./ssl:/etc/nginx/ssl:ro
    depends_on:
      - delta-sharing-server
    networks:
      - internal

networks:
  internal:
    driver: bridge

NGINX configuration:

nginx.conf

events {
  worker_connections 1024;
}

http {
  upstream delta_sharing {
    server delta-sharing-server:8080;
  }
  
  server {
    listen 443 ssl http2;
    server_name delta-sharing.example.com;
    
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;
    
    location / {
      proxy_pass http://delta_sharing;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
    }
  }
  
  server {
    listen 80;
    server_name delta-sharing.example.com;
    return 301 https://$server_name$request_uri;
  }
}

Building Your Own Docker Image

To build a custom image from source:

Clone the Repository

git clone https://github.com/delta-io/delta-sharing.git
cd delta-sharing

Build the Docker Image

build/sbt server/docker:publishLocal

This creates an image tagged delta-sharing-server:x.y.z

Run Your Custom Image

docker run -p 8080:8080 \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  delta-sharing-server:x.y.z -- --config /config/delta-sharing-server-config.yaml

Kubernetes Deployment

Deploy to Kubernetes with this manifest:

deployment.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: delta-sharing-config
data:
  delta-sharing-server.yaml: |
    version: 1
    shares:
      - name: "my_share"
        schemas:
          - name: "my_schema"
            tables:
              - name: "my_table"
                location: "s3a://bucket/table"
                id: "00000000-0000-0000-0000-000000000000"
    host: "0.0.0.0"
    port: 8080
    endpoint: "/delta-sharing"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: delta-sharing-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: delta-sharing-server
  template:
    metadata:
      labels:
        app: delta-sharing-server
    spec:
      containers:
      - name: delta-sharing-server
        image: deltaio/delta-sharing-server:0.7.8
        args:
          - "--"
          - "--config"
          - "/config/delta-sharing-server-config.yaml"
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config
          mountPath: /config
          readOnly: true
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: access-key-id
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: secret-access-key
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /delta-sharing/shares
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /delta-sharing/shares
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: delta-sharing-config
---
apiVersion: v1
kind: Service
metadata:
  name: delta-sharing-service
spec:
  selector:
    app: delta-sharing-server
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  type: LoadBalancer

Apply the manifest:

kubectl apply -f deployment.yaml

Troubleshooting

Container exits immediately

Symptom: Container starts and stops immediatelySolutions:

Check logs: docker logs <container-id>
Verify config file syntax and path
Ensure port in config matches container port
Check volume mount paths are correct

Permission denied errors

Symptom: Cannot read configuration filesSolutions:

Check file permissions on host: chmod 644 delta-sharing-server.yaml
Use absolute paths for volume mounts
On Linux, check SELinux contexts: chcon -Rt svirt_sandbox_file_t /path/to/config

Port already in use

Symptom: Error starting userland proxy: listen tcp 0.0.0.0:8080: bind: address already in useSolutions:

Change host port: -p 8081:8080
Stop conflicting service: lsof -ti:8080 | xargs kill
Use a different port in your configuration

Cannot connect to cloud storage

Symptom: Authentication or connection errors to S3/Azure/GCSSolutions:

Verify environment variables are set: docker exec <container> env
Check core-site.xml is mounted correctly
Ensure IAM roles/service accounts have proper permissions
Test connectivity: docker exec <container> ping s3.amazonaws.com

Performance Tuning

JVM Memory Configuration

Customize JVM heap size using environment variables:

docker run -p 8080:8080 \
  -e JAVA_OPTS="-Xms1g -Xmx2g" \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Container Resource Limits

docker run -d \
  --name delta-sharing-server \
  -p 8080:8080 \
  --memory="4g" \
  --memory-swap="4g" \
  --cpus="2.0" \
  --mount type=bind,source=$(pwd)/delta-sharing-server.yaml,target=/config/delta-sharing-server-config.yaml \
  deltaio/delta-sharing-server:0.7.8 -- --config /config/delta-sharing-server-config.yaml

Next Steps

Create Profile Files

Generate profile files for recipients

Test with Python Client

Access shared data with the Python connector

Monitoring & Logging

Set up monitoring and observability

Security Best Practices

Review authorization and security setup

Get Started

Core Concepts

Python Connector

Apache Spark Connector

Reference Server

Community Connectors

Overview

Prerequisites

Running the Docker Container

Basic Example

Advanced Docker Run Options

Running in Detached Mode

Mounting Multiple Configuration Files

Using Environment Variables

Setting Memory Limits

Docker Compose

Networking Modes

Bridge Network (Default)

Host Network

Custom Network

Reverse Proxy with Docker

Building Your Own Docker Image

Kubernetes Deployment

Troubleshooting

Performance Tuning

JVM Memory Configuration

Container Resource Limits

Next Steps

Create Profile Files

Test with Python Client

Monitoring & Logging

Security Best Practices

Build docs developers (and LLMs) love

Get Started

Core Concepts

Python Connector

Apache Spark Connector

Reference Server

Community Connectors

​Overview

​Prerequisites

​Running the Docker Container

​Basic Example

​Advanced Docker Run Options

​Running in Detached Mode

​Mounting Multiple Configuration Files

​Using Environment Variables

​Setting Memory Limits

​Docker Compose

​Networking Modes

​Bridge Network (Default)

​Host Network

​Custom Network

​Reverse Proxy with Docker

​Building Your Own Docker Image

​Kubernetes Deployment

​Troubleshooting

​Performance Tuning

​JVM Memory Configuration

​Container Resource Limits

​Next Steps

Create Profile Files

Test with Python Client

Monitoring & Logging

Security Best Practices

Build docs developers (and LLMs) love

Overview

Prerequisites

Running the Docker Container

Basic Example

Advanced Docker Run Options

Running in Detached Mode

Mounting Multiple Configuration Files

Using Environment Variables

Setting Memory Limits

Docker Compose

Networking Modes

Bridge Network (Default)

Host Network

Custom Network

Reverse Proxy with Docker

Building Your Own Docker Image

Kubernetes Deployment

Troubleshooting

Performance Tuning

JVM Memory Configuration

Container Resource Limits

Next Steps