Skip to main content
The spark-submit script in Spark’s bin directory is used to launch applications on a cluster. It can use all of Spark’s supported cluster managers through a uniform interface so you don’t have to configure your application especially for each one.

​
Bundling Your Application’s Dependencies

If your code depends on other projects, you need to package them alongside your application to distribute the code to a Spark cluster. To do this, create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins.
When creating assembly jars, list Spark and Hadoop as provided dependencies; these need not be bundled since they are provided by the cluster manager at runtime.
Once you have an assembled jar, you can call the bin/spark-submit script as shown here while passing your jar.

​
Python Dependencies

For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg.

​
Launching Applications with spark-submit

Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports: 
./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

​
Common Options

​
--class
string
The entry point for your application (e.g. org.apache.spark.examples.SparkPi)
​
--master
string
The master URL for the cluster (e.g. spark://23.195.26.187:7077)
​
--deploy-mode
string
default:"client"
Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client)
​
--conf
string
Arbitrary Spark configuration property in key=value format. For values that contain spaces wrap “key=value” in quotes. Multiple configurations should be passed as separate arguments.
​
application-jar
string
Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.
​
application-arguments
string
Arguments passed to the main method of your main class, if any

​
Deploy Modes

In client mode, the driver is launched directly within the spark-submit process which acts as a client to the cluster. The input and output of the application is attached to the console. This mode is especially suitable for applications that involve the REPL (e.g. Spark shell).A common deployment strategy is to submit your application from a gateway machine that is physically co-located with your worker machines (e.g. Master node in a standalone EC2 cluster). In this setup, client mode is appropriate.

​
Example Usage

Here are a few examples of common options: 
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master "local[8]" \
  /path/to/examples.jar \
  100

​
Master URLs

The master URL passed to Spark can be in one of the following formats:
Master URLMeaning
localRun Spark locally with one worker thread (i.e. no parallelism at all)
local[K]Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine)
local[K,F]Run Spark locally with K worker threads and F maxFailures
local[*]Run Spark locally with as many worker threads as logical cores on your machine
local[*,F]Run Spark locally with as many worker threads as logical cores on your machine and F maxFailures
local-cluster[N,C,M]Local-cluster mode is only for unit tests. It emulates a distributed cluster in a single JVM with N workers, C cores per worker and M MiB of memory per worker
spark://HOST:PORTConnect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default
spark://HOST1:PORT1,HOST2:PORT2Connect to the given Spark standalone cluster with standby masters with Zookeeper. The list must have all the master hosts in the high availability cluster
yarnConnect to a YARN cluster in client or cluster mode depending on the value of --deploy-mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable
k8s://HOST:PORTConnect to a Kubernetes cluster in client or cluster mode. The HOST and PORT refer to the Kubernetes API Server. It connects using TLS by default

​
Loading Configuration from a File

The spark-submit script can load default Spark configuration values from a properties file and pass them on to your application. By default, Spark will read options from conf/spark-defaults.conf in the SPARK_HOME directory. You can specify a custom properties file using the --properties-file parameter: 
./bin/spark-submit --properties-file my-spark-config.conf ...
Loading default Spark configurations this way can obviate the need for certain flags to spark-submit. For instance, if the spark.master property is set, you can safely omit the --master flag.

​
Configuration Precedence

Configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file. If you are ever unclear where configuration options are coming from, you can print out fine-grained debugging information by running spark-submit with the --verbose option.

​
Advanced Dependency Management

When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas.

​
URL Schemes

Spark uses the following URL scheme to allow different strategies for disseminating jars:
Absolute paths and file:/ URIs are served by the driver’s HTTP file server, and every executor pulls the file from the driver HTTP server.
These pull down files and JARs from the URI as expected.
A URI starting with local:/ is expected to exist as a local file on each worker node. This means that no network IO will be incurred, and works well for large files/JARs that are pushed to each worker, or shared via NFS, GlusterFS, etc.
JARs and files are copied to the working directory for each SparkContext on the executor nodes. This can use up a significant amount of space over time and will need to be cleaned up. With YARN, cleanup is handled automatically, and with Spark standalone, automatic cleanup can be configured with the spark.worker.cleanup.appDataTtl property.

​
Maven Coordinates

You can also include dependencies by supplying a comma-delimited list of Maven coordinates with --packages. All transitive dependencies will be handled when using this command: 
./bin/spark-submit \
  --packages org.apache.hadoop:hadoop-aws:3.4.1 \
  ...
Additional repositories can be added with the --repositories flag.

Build docs developers (and LLMs) love

Get started for freeTalk to us