Hardware Provisioning - Apache Spark

Choosing the right hardware configuration is crucial for running Spark efficiently. While the optimal hardware depends on your specific workload, this guide provides general recommendations that work well for most Spark deployments.

Overview

Spark’s performance depends on four key hardware components:

Storage Systems

Proximity to data sources like HDFS

Local Disks

For intermediate data and spills

Memory

For in-memory computation and caching

Network

For shuffle and data transfer

CPU Cores

For parallel task execution

Storage Systems

Since most Spark jobs read input data from external storage systems, placing Spark as close to this system as possible is critical for performance.

Co-location with HDFS

Best Option
Alternative
Special Cases

Run Spark on the same nodes as HDFSThe simplest approach is to set up a Spark standalone mode cluster on the same nodes and configure resource usage to avoid interference:Hadoop configuration:

mapred.child.java.opts - Control per-task memory
mapreduce.tasktracker.map.tasks.maximum - Limit map tasks
mapreduce.tasktracker.reduce.tasks.maximum - Limit reduce tasks

Alternatively, run Hadoop and Spark on a common cluster manager like YARN.

Data locality is crucial for Spark performance. Always aim to minimize the distance between computation and storage.

Local Disks

While Spark performs much of its computation in memory, it still uses local disks for data that doesn’t fit in RAM and for preserving intermediate output between stages.

Disk Configuration

Number of Disks

We recommend 4-8 disks per node for optimal I/O throughput.

RAID Configuration

Configure disks without RAID (just as separate mount points). This provides better I/O parallelism.

Mount Options

On Linux, mount disks with the noatime option to reduce unnecessary writes:

mount -o noatime /dev/sdb1 /mnt/disk1

Spark Configuration

Configure spark.local.dir as a comma-separated list of the local disks:

spark.local.dir=/mnt/disk1,/mnt/disk2,/mnt/disk3,/mnt/disk4

If you’re running HDFS, it’s fine to use the same disks as HDFS. Spark will write to different directories.

Disk Recommendations

Configuration	Recommendation	Notes
Number of disks	4-8 per node	More disks = more I/O parallelism
RAID	No RAID	Better performance without RAID
Disk type	SSD or fast HDD	SSDs recommended for shuffle-heavy workloads
Mount option	`noatime`	Reduces write overhead

Memory

Memory is one of the most important resources for Spark applications. Proper memory allocation ensures efficient in-memory computation while leaving enough for the operating system.

Memory Sizing Guidelines

General Range
Determining Requirements
Large Memory Nodes

Spark can run well with anywhere from 8 GiB to hundreds of gigabytes of memory per machine.

We recommend allocating at most 75% of the memory for Spark. Leave the rest for the operating system and buffer cache.

To determine how much memory your application needs:

Load part of your dataset in a Spark RDD
Use the Storage tab of Spark’s web UI at http://<driver-node>:4040
Check the RDD’s size in memory

Memory usage is greatly affected by storage level and serialization format. See the tuning guide for tips on reducing memory usage.

The Java VM does not always behave well with more than 200 GiB of RAM.If you have machines with more RAM:

Launch multiple executors per node
In standalone mode, a worker launches multiple executors based on available memory and cores
Each executor runs in a separate Java VM

Example configuration:

spark.executor.memory=64g
spark.executor.cores=8
# On a 512GB machine, this would create ~7 executors

Memory Allocation Formula

Total Machine Memory: 256 GB
├─ OS and Buffer Cache: 64 GB (25%)
└─ Spark Available: 192 GB (75%)
   ├─ Executor 1: 64 GB
   ├─ Executor 2: 64 GB
   └─ Executor 3: 64 GB

Memory Configuration Examples

Small Cluster (8 GB per node)

spark.executor.memory=6g
spark.driver.memory=4g

Leaves 2 GB for OS on executor nodes.

Medium Cluster (64 GB per node)

spark.executor.memory=48g
spark.executor.cores=8
spark.driver.memory=16g

Single executor per node with 16 GB reserved for OS.

Large Cluster (256 GB per node)

spark.executor.memory=56g
spark.executor.cores=8
spark.executor.instances=3
spark.driver.memory=32g

Three executors per node, each with 56 GB. Total: 168 GB for Spark, 88 GB for OS.

Network

In our experience, when data is in memory, many Spark applications become network-bound. A fast network is essential for shuffle-heavy workloads.

Network Recommendations

Network Speed

Use a 10 Gigabit or higher network

Network Topology

Ensure low latency between nodes in the same rack

This is especially true for “distributed reduce” applications such as group-bys, reduce-bys, and SQL joins.

Monitoring Network Usage

You can see how much data Spark shuffles across the network in the application’s monitoring UI at http://<driver-node>:4040. Look for:

Shuffle read/write metrics
Network I/O time
Data locality levels

If you see high shuffle volumes or poor data locality, consider:

Increasing network bandwidth
Optimizing your application to reduce shuffles
Improving data co-location

Network Configuration

Metric	Recommendation	Use Case
Bandwidth	10 Gbps minimum	Standard workloads
Bandwidth	25-100 Gbps	Large-scale shuffle operations
Latency	< 1ms within rack	Low-latency requirements
Latency	< 5ms across racks	Acceptable for most workloads

CPU Cores

Spark scales well to tens of CPU cores per machine because it performs minimal sharing between threads. More cores allow more tasks to run in parallel.

CPU Recommendations

Minimum Cores

Provision at least 8-16 cores per machine for good parallelism.

Scaling Considerations

Once data is in memory, most applications are either CPU-bound or network-bound. Depending on your workload’s CPU cost, you may need more cores.

Hyperthreading

Hyperthreading can provide some benefit, but physical cores are more valuable than logical cores.

Core Configuration

# Allocate cores per executor
spark.executor.cores=4

# Total number of executors
spark.executor.instances=10

# Total cores used: 4 * 10 = 40 cores

Spark performs minimal thread contention, so it scales efficiently with more cores. Don’t be afraid to use all available cores.

CPU Configuration Examples

CPU-Bound Workloads

For compute-intensive operations like machine learning:

spark.executor.cores=16
spark.executor.memory=32g

More cores per executor for better CPU utilization.

Balanced Workloads

For typical ETL and analytics:

spark.executor.cores=4
spark.executor.memory=16g

Balanced core-to-memory ratio.

Memory-Bound Workloads

For caching-heavy applications:

spark.executor.cores=2
spark.executor.memory=32g

Fewer cores with more memory per executor.

Hardware Provisioning Checklist

Assess Your Workload

Determine if your workload is CPU-bound, memory-bound, or network-bound.

Plan Storage

Co-locate Spark with your data source when possible.

Configure Disks

Set up 4-8 local disks per node without RAID.

Allocate Memory

Use 75% of system memory for Spark, leaving 25% for the OS.

Provision Network

Ensure at least 10 Gbps network connectivity.

Configure CPUs

Provision 8-16 cores per machine minimum.

Test and Tune

Run representative workloads and adjust based on monitoring data.

Example Hardware Configurations

Small Cluster
Medium Cluster
Large Cluster

Suitable for development and small datasets

Component	Specification
Nodes	5-10 nodes
CPU	8 cores per node
Memory	32 GB per node
Local disks	2-4 x 1TB HDD
Network	1 Gbps
Storage	Co-located HDFS

Total cluster: 40-80 cores, 160-320 GB RAM

Suitable for production workloads

Component	Specification
Nodes	20-50 nodes
CPU	16 cores per node
Memory	128 GB per node
Local disks	4-6 x 2TB SSD
Network	10 Gbps
Storage	Co-located HDFS

Total cluster: 320-800 cores, 2.5-6.4 TB RAM

Suitable for large-scale data processing

Component	Specification
Nodes	100+ nodes
CPU	32+ cores per node
Memory	256-512 GB per node
Local disks	6-8 x 4TB NVMe SSD
Network	25-100 Gbps
Storage	Co-located HDFS or separate storage cluster

Total cluster: 3200+ cores, 25-51 TB RAM

Cloud Deployment Recommendations

When deploying Spark on cloud platforms:

AWS

Recommended instance types:

r5d.4xlarge (128 GB RAM, 16 vCPUs)
r5d.8xlarge (256 GB RAM, 32 vCPUs)
i3.8xlarge (244 GB RAM, 32 vCPUs, NVMe SSDs)

Azure

Recommended instance types:

Standard_D16s_v3 (64 GB RAM, 16 vCPUs)
Standard_E32s_v3 (256 GB RAM, 32 vCPUs)
Standard_L16s_v2 (128 GB RAM, 16 vCPUs, NVMe SSDs)

GCP

Recommended instance types:

n1-highmem-16 (104 GB RAM, 16 vCPUs)
n1-highmem-32 (208 GB RAM, 32 vCPUs)
n2-standard-32 (128 GB RAM, 32 vCPUs)

Cloud instance costs vary by region and commitment level. Use reserved instances or committed use discounts for production clusters.

Get Started

Core Concepts

Spark SQL

Structured Streaming

Machine Learning

Graph Processing

Deployment

Configuration & Tuning

Monitoring

​Overview

Storage Systems

Local Disks

Memory

Network

CPU Cores

​Storage Systems

​Co-location with HDFS

​Local Disks

​Disk Configuration

​Disk Recommendations

​Memory

​Memory Sizing Guidelines

​Memory Allocation Formula

​Memory Configuration Examples

​Network

​Network Recommendations

Network Speed

Network Topology

​Monitoring Network Usage

​Network Configuration

​CPU Cores

​CPU Recommendations

​Core Configuration

​CPU Configuration Examples

​Hardware Provisioning Checklist

​Example Hardware Configurations

​Cloud Deployment Recommendations

AWS

Azure

GCP

​Next Steps

Performance Tuning

Spark Configuration

Build docs developers (and LLMs) love

Overview

Storage Systems

Co-location with HDFS

Local Disks

Disk Configuration

Disk Recommendations

Memory

Memory Sizing Guidelines

Memory Allocation Formula

Memory Configuration Examples

Network

Network Recommendations

Monitoring Network Usage

Network Configuration

CPU Cores

CPU Recommendations

Core Configuration

CPU Configuration Examples

Hardware Provisioning Checklist

Example Hardware Configurations

Cloud Deployment Recommendations

Next Steps