Docker Deployment

NativeLink provides Docker Compose configurations for quick deployment in development and testing environments. This guide covers both single-worker and multi-worker setups.

Prerequisites

Docker Engine 20.10+
Docker Compose v2.0+
4GB+ RAM available
20GB+ disk space

Docker deployments are suitable for development and testing. For production environments, see Production Deployment.

Single Worker Deployment

The simplest NativeLink deployment runs all components in a single setup.

Clone the repository

git clone https://github.com/TraceMachina/nativelink.git
cd nativelink/deployment-examples/docker-compose

Start services

docker compose up -d

This starts three services:

nativelink_local_cas: Content Addressable Storage server (port 50051)
nativelink_scheduler: Job scheduler (port 50052)
nativelink_executor: Worker executing builds

Verify deployment

docker compose ps
docker compose logs -f

Test with Bazel

bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052

Single Worker Configuration

services:
  nativelink_local_cas:
    image: trace_machina/nativelink:latest
    volumes:
      - ${NATIVELINK_DIR:-~/.cache/nativelink}:/root/.cache/nativelink
      - type: bind
        source: .
        target: /root
    environment:
      RUST_LOG: ${RUST_LOG:-warn}
    ports:
      - "50051:50051/tcp"
      - "127.0.0.1:50061:50061"
      - "50071:50071/tcp"
    command: nativelink /root/local-storage-cas.json5

  nativelink_scheduler:
    image: trace_machina/nativelink:latest
    volumes:
      - type: bind
        source: .
        target: /root
    environment:
      RUST_LOG: ${RUST_LOG:-warn}
      CAS_ENDPOINT: nativelink_local_cas
    ports:
      - "50052:50052/tcp"
    command: nativelink /root/scheduler.json5
    depends_on:
      - nativelink_local_cas

  nativelink_executor:
    image: trace_machina/nativelink:latest
    volumes:
      - ${NATIVELINK_DIR:-~/.cache/nativelink}:/root/.cache/nativelink
      - type: bind
        source: .
        target: /root
    environment:
      RUST_LOG: ${RUST_LOG:-warn}
      CAS_ENDPOINT: nativelink_local_cas
      SCHEDULER_ENDPOINT: nativelink_scheduler
    command: nativelink /root/worker.json5
    depends_on:
      - nativelink_local_cas
      - nativelink_scheduler

Multi-Worker Deployment

For distributed execution across multiple workers, use the multi-worker configuration.

Critical: All workers MUST share the same CAS storage path. Using different paths will cause “Object not found” errors.

Use multi-worker compose file

cd deployment-examples/docker-compose
docker compose -f docker-compose-multi-worker.yml build

Start multi-worker deployment

docker compose -f docker-compose-multi-worker.yml up -d

This creates:

1 CAS server
1 Scheduler
3 Workers (scalable)

Test with Bazel

bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --remote_default_exec_properties=cpu_count=2 \
  --jobs=10

Multi-Worker Configuration

version: '3.8'

services:
  cas-server:
    image: nativelink:latest
    volumes:
      - cas-data:/data/cas
      - ./cas-server-multi-worker.json5:/nativelink-config.json5
    ports:
      - "50051:50051"
    environment:
      - RUST_LOG=info
    command: nativelink /nativelink-config.json5
    networks:
      - nativelink-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:50051/status"]
      interval: 5s
      timeout: 5s
      retries: 5

  scheduler:
    image: nativelink:latest
    volumes:
      - ./scheduler-multi-worker.json5:/nativelink-config.json5
    ports:
      - "50052:50052"
      - "50061:50061"
    environment:
      - RUST_LOG=info
      - CAS_ENDPOINT=cas-server
    command: nativelink /nativelink-config.json5
    depends_on:
      cas-server:
        condition: service_started
    networks:
      - nativelink-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:50052/status"]
      interval: 5s
      timeout: 5s
      retries: 5

  worker-1:
    image: nativelink:latest
    volumes:
      - cas-data:/data/cas  # Shared CAS volume
      - worker1-data:/data/worker1
      - ./worker-shared-cas.json5:/nativelink-config.json5
    environment:
      - RUST_LOG=info
      - SCHEDULER_ENDPOINT=scheduler
      - CAS_ENDPOINT=cas-server
      - WORKER_NAME=worker-1
      - WORKER_ID=worker-1
    command: nativelink /nativelink-config.json5
    depends_on:
      scheduler:
        condition: service_started
    networks:
      - nativelink-network
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

  worker-2:
    image: nativelink:latest
    volumes:
      - cas-data:/data/cas  # Same shared volume
      - worker2-data:/data/worker2
      - ./worker-shared-cas.json5:/nativelink-config.json5
    environment:
      - RUST_LOG=info
      - SCHEDULER_ENDPOINT=scheduler
      - CAS_ENDPOINT=cas-server
      - WORKER_NAME=worker-2
      - WORKER_ID=worker-2
    command: nativelink /nativelink-config.json5
    depends_on:
      scheduler:
        condition: service_started
    networks:
      - nativelink-network
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

networks:
  nativelink-network:
    driver: bridge

volumes:
  cas-data:
    driver: local
  worker1-data:
    driver: local
  worker2-data:
    driver: local

Environment Variables

Variable	Default	Description
`NATIVELINK_DIR`	`~/.cache/nativelink`	Data storage directory
`RUST_LOG`	`warn`	Log level (error, warn, info, debug, trace)
`CAS_ENDPOINT`	`127.0.0.1`	CAS server hostname
`SCHEDULER_ENDPOINT`	`127.0.0.1`	Scheduler hostname
`WORKER_NAME`	-	Unique worker identifier
`WORKER_ID`	-	Worker ID for multi-worker setups

Scaling Workers

Dynamically scale the number of workers:

# Scale to 5 workers
docker compose -f docker-compose-multi-worker.yml up -d --scale worker=5

# Scale down to 2 workers
docker compose -f docker-compose-multi-worker.yml up -d --scale worker=2

Monitoring

View Logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f nativelink_scheduler

# Multi-worker specific worker
docker compose -f docker-compose-multi-worker.yml logs -f worker-1

Check Status

# Container status
docker compose ps

# Resource usage
docker stats

# CAS storage usage
docker exec -it nativelink_local_cas du -sh ~/.cache/nativelink

Health Checks

The multi-worker configuration includes health checks:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:50051/status"]
  interval: 5s
  timeout: 5s
  retries: 5

Troubleshooting

”Object not found” Errors

Symptom: Workers fail with errors about objects not found in store. Cause: Workers are using different CAS paths. Solution:

Verify all workers mount the same volume:

docker inspect <worker-container> | grep -A 5 Mounts

Ensure content_path is identical in all worker configs
Use the same shared volume for all workers

Ports Already in Use

Symptom: Cannot start services due to port conflicts. Solution: Change port mappings in docker-compose.yml:

ports:
  - "50052:50051"  # Map host port 50052 to container port 50051

Worker Connection Issues

Symptom: Workers cannot connect to scheduler. Solution:

Check service networking:

docker network inspect nativelink-network

Verify environment variables are set correctly
Check scheduler logs for connection errors

Out of Disk Space

Symptom: Services crash or slow down significantly. Solution:

Increase max_bytes in eviction policies

Clean old data:

docker compose down -v  # Removes volumes

Use external storage backends

Volume Management

Persistent Storage

Data is stored in Docker volumes by default. To use host directories:

volumes:
  - /path/on/host:/root/.cache/nativelink

Backup

# Backup CAS data
docker run --rm -v nativelink_cas-data:/data -v $(pwd):/backup \
  ubuntu tar czf /backup/cas-backup.tar.gz /data

# Restore CAS data
docker run --rm -v nativelink_cas-data:/data -v $(pwd):/backup \
  ubuntu tar xzf /backup/cas-backup.tar.gz -C /

Clean Up

# Stop and remove containers
docker compose down

# Remove volumes (deletes all data)
docker compose down -v

# Remove images
docker rmi trace_machina/nativelink:latest

For production deployments with monitoring, see Production Deployment and integrate with the metrics stack.

Getting Started

Core Concepts

Deployment

Integration

Operations

Security

Prerequisites

Single Worker Deployment

Single Worker Configuration

Multi-Worker Deployment

Multi-Worker Configuration

Environment Variables

Scaling Workers

Monitoring

View Logs

Check Status

Health Checks

Troubleshooting

”Object not found” Errors

Ports Already in Use

Worker Connection Issues

Out of Disk Space

Volume Management

Persistent Storage

Backup

Clean Up

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Deployment

Integration

Operations

Security

​Prerequisites

​Single Worker Deployment

​Single Worker Configuration

​Multi-Worker Deployment

​Multi-Worker Configuration

​Environment Variables

​Scaling Workers

​Monitoring

​View Logs

​Check Status

​Health Checks

​Troubleshooting

​”Object not found” Errors

​Ports Already in Use

​Worker Connection Issues

​Out of Disk Space

​Volume Management

​Persistent Storage

​Backup

​Clean Up

Build docs developers (and LLMs) love

Prerequisites

Single Worker Deployment

Single Worker Configuration

Multi-Worker Deployment

Multi-Worker Configuration

Environment Variables

Scaling Workers

Monitoring

View Logs

Check Status

Health Checks

Troubleshooting

”Object not found” Errors

Ports Already in Use

Worker Connection Issues

Out of Disk Space

Volume Management

Persistent Storage

Backup

Clean Up