Skip to main content
In addition to running on the YARN cluster manager, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use the provided launch scripts.
Security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s important to secure access to the cluster to prevent unauthorized applications from running on the cluster. See Spark Security before running Spark.

​
Installing Spark Standalone to a Cluster

To install Spark Standalone mode, you simply place a compiled version of Spark on each node on the cluster. You can obtain pre-built versions of Spark with each release or build it yourself.

​
Starting a Cluster Manually

You can start a standalone master server by executing: 
./sbin/start-master.sh
Once started, the master will print out a spark://HOST:PORT URL for itself, which you can use to connect workers to it, or pass as the “master” argument to SparkContext. You can also find this URL on the master’s web UI, which is http://localhost:8080 by default. Similarly, you can start one or more workers and connect them to the master via: 
./sbin/start-worker.sh <master-spark-URL>
Once you have started a worker, look at the master’s web UI (http://localhost:8080 by default). You should see the new node listed there, along with its number of CPUs and memory (minus one gigabyte left for the OS).

​
Master and Worker Configuration Options

The following configuration options can be passed to the master and worker:
ArgumentMeaning
-h HOST, --host HOSTHostname to listen on
-p PORT, --port PORTPort for service to listen on (default: 7077 for master, random for worker)
--webui-port PORTPort for web UI (default: 8080 for master, 8081 for worker)
-c CORES, --cores CORESTotal CPU cores to allow Spark applications to use on the machine (default: all available); only on worker
-m MEM, --memory MEMTotal amount of memory to allow Spark applications to use on the machine, in a format like 1000M or 2G (default: your machine’s total RAM minus 1 GiB); only on worker
-d DIR, --work-dir DIRDirectory to use for scratch space and job output logs (default: SPARK_HOME/work); only on worker
--properties-file FILEPath to a custom Spark properties file to load (default: conf/spark-defaults.conf)

​
Cluster Launch Scripts

To launch a Spark standalone cluster with the launch scripts, you should create a file called conf/workers in your Spark directory, which must contain the hostnames of all the machines where you intend to start Spark workers, one per line.
If conf/workers does not exist, the launch scripts defaults to a single machine (localhost), which is useful for testing.
The master machine accesses each of the worker machines via ssh. By default, ssh is run in parallel and requires password-less (using a private key) access to be setup.

​
Available Scripts

Once you’ve set up the conf/workers file, you can launch or stop your cluster with the following shell scripts:
  • sbin/start-master.sh - Starts a master instance on the machine the script is executed on
  • sbin/start-workers.sh - Starts a worker instance on each machine specified in the conf/workers file
  • sbin/start-worker.sh - Starts a worker instance on the machine the script is executed on
  • sbin/start-all.sh - Starts both a master and a number of workers as described above
  • sbin/stop-master.sh - Stops the master that was started via the sbin/start-master.sh script
  • sbin/stop-worker.sh - Stops all worker instances on the machine the script is executed on
  • sbin/stop-workers.sh - Stops all worker instances on the machines specified in the conf/workers file
  • sbin/stop-all.sh - Stops both the master and the workers as described above
These scripts must be executed on the machine you want to run the Spark master on, not your local machine.

​
Environment Variables

You can optionally configure the cluster further by setting environment variables in conf/spark-env.sh. Create this file by starting with the conf/spark-env.sh.template, and copy it to all your worker machines for the settings to take effect.

​
Master Environment Variables

Environment VariableMeaning
SPARK_MASTER_HOSTBind the master to a specific hostname or IP address, for example a public one
SPARK_MASTER_PORTStart the master on a different port (default: 7077)
SPARK_MASTER_WEBUI_PORTPort for the master web UI (default: 8080)
SPARK_MASTER_OPTSConfiguration properties that apply only to the master in the form “-Dx=y” (default: none)

​
Worker Environment Variables

Environment VariableMeaning
SPARK_WORKER_CORESTotal number of cores to allow Spark applications to use on the machine (default: all available cores)
SPARK_WORKER_MEMORYTotal amount of memory to allow Spark applications to use on the machine, e.g. 1000m, 2g (default: total memory minus 1 GiB)
SPARK_WORKER_PORTStart the Spark worker on a specific port (default: random)
SPARK_WORKER_WEBUI_PORTPort for the worker web UI (default: 8081)
SPARK_WORKER_DIRDirectory to run applications in, which will include both logs and scratch space (default: SPARK_HOME/work)
SPARK_WORKER_OPTSConfiguration properties that apply only to the worker in the form “-Dx=y” (default: none)

​
General Environment Variables

Environment VariableMeaning
SPARK_LOCAL_DIRSDirectory to use for “scratch” space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks
SPARK_LOG_DIRWhere log files are stored (default: SPARK_HOME/logs)
SPARK_PID_DIRWhere pid files are stored (default: /tmp)
SPARK_DAEMON_MEMORYMemory to allocate to the Spark master and worker daemons themselves (default: 1g)
SPARK_PUBLIC_DNSThe public DNS name of the Spark master and workers (default: none)

​
Connecting an Application to the Cluster

To run an application on the Spark cluster, simply pass the spark://IP:PORT URL of the master to the SparkContext constructor. To run an interactive Spark shell against the cluster, run the following command: 
./bin/spark-shell --master spark://IP:PORT
You can also pass an option --total-executor-cores <numCores> to control the number of cores that spark-shell uses on the cluster.

​
Launching Applications

​
Using spark-submit

The spark-submit script provides the most straightforward way to submit a compiled Spark application to the cluster. For standalone clusters, Spark currently supports two deploy modes:
In client mode, the driver is launched in the same process as the client that submits the application. The driver is launched directly within the spark-submit process which acts as a client to the cluster.

​
Supervise Mode

Standalone cluster mode supports restarting your application automatically if it exited with non-zero exit code. To use this feature, you may pass in the --supervise flag to spark-submit when launching your application: 
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --deploy-mode cluster \
  --supervise \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000
To kill an application that is failing repeatedly: 
./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID>
You can find the driver ID through the standalone Master web UI at http://<master url>:8080.

​
Resource Scheduling

The standalone cluster mode currently only supports a simple FIFO scheduler across applications. However, to allow multiple concurrent users, you can control the maximum number of resources each application will use. By default, it will acquire all cores in the cluster, which only makes sense if you just run one application at a time. You can cap the number of cores by setting spark.cores.max in your SparkConf: 
val conf = new SparkConf()
  .setMaster(...)
  .setAppName(...)
  .set("spark.cores.max", "10")
val sc = new SparkContext(conf)
In addition, you can configure spark.deploy.defaultCores on the cluster master process to change the default for applications that don’t set spark.cores.max: 
export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=<value>"
This is useful on shared clusters where users might not have configured a maximum number of cores individually.

​
High Availability

By default, standalone scheduling clusters are resilient to Worker failures. However, the scheduler uses a Master to make scheduling decisions, and this (by default) creates a single point of failure. In order to circumvent this, we have two high availability schemes.

​
Standby Masters with ZooKeeper

Utilizing ZooKeeper to provide leader election and some state storage, you can launch multiple Masters in your cluster connected to the same ZooKeeper instance. One will be elected “leader” and the others will remain in standby mode. If the current leader dies, another Master will be elected, recover the old Master’s state, and then resume scheduling. The entire recovery process (from the time the first leader goes down) should take between 1 and 2 minutes.
This delay only affects scheduling new applications — applications that were already running during Master failover are unaffected.

​
Configuration

To enable this recovery mode, set SPARK_DAEMON_JAVA_OPTS in spark-env by configuring spark.deploy.recoveryMode and related spark.deploy.zookeeper.* configurations. After you have a ZooKeeper cluster set up, start multiple Master processes on different nodes with the same ZooKeeper configuration. When starting applications or adding Workers to the cluster, pass a list of Masters: 
spark://host1:port1,host2:port2

​
Single-Node Recovery with Local File System

If you just want to be able to restart the Master if it goes down, FILESYSTEM mode can take care of it. When applications and Workers register, they have enough state written to the provided directory so that they can be recovered upon a restart of the Master process.

​
Configuration

Set SPARK_DAEMON_JAVA_OPTS in spark-env:
System PropertyDefaultMeaning
spark.deploy.recoveryModeNONESet to FILESYSTEM to enable file-system-based single-node recovery mode, ZOOKEEPER to use Zookeeper-based recovery mode
spark.deploy.recoveryDirectory""The directory in which Spark will store recovery state
While filesystem recovery seems straightforwardly better than not doing any recovery at all, this mode may be suboptimal for certain development or experimental purposes. Killing a master via stop-master.sh does not clean up its recovery state, so whenever you start a new Master, it will enter recovery mode.

​
Monitoring and Logging

Spark’s standalone mode offers a web-based user interface to monitor the cluster. The master and each worker has its own web UI that shows cluster and job statistics. By default, you can access the web UI for the master at port 8080. The port can be changed either in the configuration file or via command-line options. In addition, detailed log output for each job is also written to the work directory of each worker node (SPARK_HOME/work by default). You will see two files for each job, stdout and stderr, with all output it wrote to its console.

Build docs developers (and LLMs) love

Get started for freeTalk to us