Skip to main content
The Middle Manager service is a worker service that executes submitted tasks. Middle Managers forward tasks to Peons that run in separate JVMs.

​
Architecture Overview

Druid uses separate JVMs for tasks to isolate resources and logs. Each Peon is capable of running only one task at a time, whereas a Middle Manager may have multiple Peons.

Resource Isolation

Each task runs in its own JVM (Peon), preventing resource contention and failures from affecting other tasks

Log Separation

Task logs are isolated per Peon, making debugging and monitoring easier

Task Capacity

A single Middle Manager can manage multiple Peons, each running one task

Clean Termination

When a task completes, its Peon JVM is terminated, freeing all resources

​
Key Responsibilities

1

Receive task assignments

Accept task assignments from the Overlord service
2

Create Peon JVMs

Spawn separate JVM processes (Peons) for each task
3

Monitor task execution

Track the progress and status of running tasks
4

Report task status

Communicate task status back to the Overlord
5

Manage resources

Allocate and manage resources (CPU, memory, disk) for Peon processes

​
Configuration

For Apache Druid Middle Manager service configuration, see:

​
Running the Middle Manager

org.apache.druid.cli.Main server middleManager

​
HTTP Endpoints

For a list of API endpoints supported by the Middle Manager, see:

​
Middle Manager vs. Peon

Understanding the relationship between Middle Managers and Peons is crucial:
Parent Process
  • Runs as a persistent service
  • Manages multiple Peon JVMs
  • Communicates with the Overlord
  • Allocates resources to Peons
  • Monitors Peon health
The Middle Manager itself doesn’t execute tasks—it delegates to Peons.

​
Task Execution Workflow

​
Resource Management

The Middle Manager is responsible for allocating resources to Peon processes:

Task Slots

Configure druid.worker.capacity to set the maximum number of concurrent Peons (tasks) a Middle Manager can run

Memory Allocation

Each Peon gets its own JVM heap, configured via task-specific or global settings

CPU Allocation

Operating system handles CPU scheduling across Peon processes

Disk Space

Middle Manager allocates disk space for each Peon’s working directory

​
Example Capacity Planning

If you configure a Middle Manager with:
  • druid.worker.capacity = 4
  • 64 GB total RAM
  • 16 CPU cores
You might allocate:
  • ~12 GB heap per Peon (leaving headroom for OS and Middle Manager)
  • ~4 CPU cores per Peon (assuming concurrent execution)

​
Alternative: Indexer Service

Consider the Indexer service as an alternative to Middle Manager + Peon:
  • The Indexer runs tasks as threads within a single JVM instead of separate processes
  • Better resource sharing across tasks
  • Easier to configure and deploy
  • Still experimental but recommended for certain workloads
See Indexer Service for more details.

​
Architecture Integration

With Overlord

  • Receives task assignments
  • Reports task capacity and availability
  • Sends task status updates and completion notifications
  • Subject to blacklisting if tasks fail repeatedly

With Peons

  • Spawns Peon JVM processes
  • Passes task configuration to Peons
  • Monitors Peon health and resource usage
  • Collects task logs and metrics

With Deep Storage

  • Peons write completed segments to deep storage
  • Middle Manager manages temporary disk space for Peons

With Metadata Store

  • Peons publish segment metadata upon task completion
  • Task status and logs may be persisted

​
Common Configuration Parameters

​
druid.worker.capacity
integer
required
Maximum number of tasks (Peons) that can run concurrently on this Middle Manager
​
druid.worker.ip
string
IP address of the Middle Manager. Used for task assignment and communication
​
druid.worker.version
string
Version identifier for this worker. Can be used for rolling deployments
​
druid.indexer.runner.javaOpts
string
JVM options to pass to Peon processes. Configure heap size, GC settings, etc.
​
druid.indexer.task.baseTaskDir
string
Base directory for Peon working directories

​
Example Configuration

druid.worker.capacity=4
druid.indexer.runner.javaOpts=-Xms3g -Xmx3g -XX:+UseG1GC
druid.indexer.task.baseTaskDir=/mnt/druid/task
druid.worker.ip=10.0.0.100
druid.worker.version=1

​
Monitoring and Debugging

Best practices for monitoring Middle Managers:
  1. Track task capacity utilization: Monitor how many task slots are in use
  2. Watch memory usage: Ensure Peons have enough heap for their workloads
  3. Monitor task duration: Identify slow or stuck tasks
  4. Check task logs: Each Peon has separate logs for debugging
  5. Alert on repeated failures: High failure rates may indicate configuration issues

​
Performance Considerations

To optimize Middle Manager performance:
  1. Right-size task capacity: Don’t over-commit resources; leave headroom for OS operations
  2. Use fast local disk: Task working directories benefit from SSD storage
  3. Tune Peon JVM settings: Optimize heap size and GC for your workload
  4. Monitor network bandwidth: Peons download data and upload segments
  5. Consider Indexer service: For memory-intensive workloads, the Indexer may be more efficient
Resource OverheadEach Peon JVM has overhead:
  • JVM startup time
  • Heap and metaspace memory
  • Thread stacks
  • OS file descriptors
Factor this overhead into your capacity planning, especially if running many concurrent tasks.

Build docs developers (and LLMs) love

Get started for freeTalk to us