Running Spark on Kubernetes

Spark can run on clusters managed by Kubernetes. This feature makes use of native Kubernetes scheduler that has been added to Spark.

Security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s important to secure access to the cluster to prevent unauthorized applications from running on the cluster. See Spark Security before running Spark.

Prerequisites

Kubernetes Cluster

A running Kubernetes cluster at version >= 1.33 with access configured using kubectl

Permissions

Appropriate permissions to list, create, edit and delete pods in your cluster

DNS

Kubernetes DNS configured in your cluster

Docker Images

Container images for Spark driver and executor pods

If you don’t have a Kubernetes cluster, you can set up a test cluster on your local machine using minikube. We recommend 3 CPUs and 4g of memory to be able to start a simple Spark application with a single executor.

How It Works

spark-submit can be directly used to submit a Spark application to a Kubernetes cluster. The submission mechanism works as follows:

Spark creates a Spark driver running within a Kubernetes pod
The driver creates executors which are also running within Kubernetes pods and connects to them, and executes application code
When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists logs and remains in “completed” state until it’s eventually garbage collected or manually cleaned up

In the completed state, the driver pod does not use any computational or memory resources.

Docker Images

Kubernetes requires users to supply images that can be deployed into containers within pods. Spark ships with a Dockerfile that can be used for this purpose, or customized to match an individual application’s needs. It can be found in the kubernetes/dockerfiles/ directory. Spark also ships with a bin/docker-image-tool.sh script that can be used to build and publish the Docker images:

# Build the images
$ ./bin/docker-image-tool.sh -r <repo> -t my-tag build

# Push the images to a registry
$ ./bin/docker-image-tool.sh -r <repo> -t my-tag push

Language-Specific Images

By default bin/docker-image-tool.sh builds docker image for running JVM jobs. You need to opt-in to build additional language binding docker images:

$ ./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build

You can also use the Apache Spark Docker images (such as apache/spark:<version>) directly.

Submitting Applications to Kubernetes

Cluster Mode

To launch Spark Pi in cluster mode:

$ ./bin/spark-submit \
    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=<spark-image> \
    local:///path/to/examples.jar

--master

string

required

The Kubernetes API server URL with format k8s://<api_server_host>:<k8s-apiserver-port>. The port must always be specified, even if it’s the HTTPS port 443. If no HTTP protocol is specified, it defaults to https.

--deploy-mode

string

default:"cluster"

Must be cluster for Kubernetes mode

spark.kubernetes.container.image

string

required

Container image to use for the Spark application (e.g., apache/spark:v3.5.0)

Finding the API Server URL

One way to discover the apiserver URL is by executing kubectl cluster-info:

$ kubectl cluster-info
Kubernetes master is running at http://127.0.0.1:6443

You can also use the authenticating proxy, kubectl proxy:

$ kubectl proxy

If the local proxy is running at localhost:8001, use --master k8s://http://127.0.0.1:8001 as the argument to spark-submit.

Client Mode

Starting with Spark 2.4.0, it is possible to run Spark applications on Kubernetes in client mode. When your application runs in client mode, the driver can run inside a pod or on a physical host.

Client Mode Networking

Spark executors must be able to connect to the Spark driver over a hostname and a port that is routable from the Spark executors. When running your driver inside a Kubernetes pod, you can use a headless service to allow your driver pod to be routable from the executors.Specify the driver’s hostname via spark.driver.host and your spark driver’s port to spark.driver.port.

Executor Pod Garbage Collection

If you run your Spark driver in a pod, it is highly recommended to set spark.kubernetes.driver.pod.name to the name of that pod. When this property is set, the Spark scheduler will deploy the executor pods with an OwnerReference, which in turn will ensure that once the driver pod is deleted from the cluster, all of the application’s executor pods will also be deleted.

Dependency Management

If your application’s dependencies are all hosted in remote locations like HDFS or HTTP servers, they may be referred to by their appropriate remote URIs. Application dependencies can also be pre-mounted into custom-built Docker images. Dependencies can be added to the classpath by referencing them with local:// URIs and/or setting the SPARK_EXTRA_CLASSPATH environment variable in your Dockerfiles.

Using S3 for Dependencies

A typical example using S3:

./bin/spark-submit \
  --master k8s://https://<k8s-apiserver-host>:<port> \
  --deploy-mode cluster \
  --packages org.apache.hadoop:hadoop-aws:3.4.1 \
  --conf spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path \
  --conf spark.hadoop.fs.s3a.access.key=... \
  --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
  --conf spark.hadoop.fs.s3a.fast.upload=true \
  --conf spark.hadoop.fs.s3a.secret.key=... \
  --conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \
  file:///full/path/to/app.jar

The app jar file will be uploaded to S3 and then when the driver is launched it will be downloaded to the driver pod and added to its classpath.

All client-side dependencies will be uploaded with a flat directory structure so file names must be unique otherwise files will be overwritten.

Secret Management

Kubernetes Secrets can be used to provide credentials for a Spark application to access secured services.

Mounting Secrets as Files

To mount a secret into the driver and executor containers:

--conf spark.kubernetes.driver.secrets.spark-secret=/etc/secrets
--conf spark.kubernetes.executor.secrets.spark-secret=/etc/secrets

Using Secrets as Environment Variables

To use a secret through an environment variable:

--conf spark.kubernetes.driver.secretKeyRef.ENV_NAME=name:key
--conf spark.kubernetes.executor.secretKeyRef.ENV_NAME=name:key

Pod Template

Kubernetes allows defining pods from template files. Spark users can similarly use template files to define the driver or executor pod configurations that Spark configurations do not support.

--conf spark.kubernetes.driver.podTemplateFile=s3a://bucket/driver.yml
--conf spark.kubernetes.executor.podTemplateFile=s3a://bucket/executor.yml

The executor pod template file will be automatically mounted onto a volume in the driver pod when it’s created.

Using Kubernetes Volumes

Users can mount the following types of Kubernetes volumes into the driver and executor pods:

hostPath: mounts a file or directory from the host node’s filesystem into a pod
emptyDir: an initially empty volume created when a pod is assigned to a node
nfs: mounts an existing NFS (Network File System) into a pod
persistentVolumeClaim: mounts a PersistentVolume into a pod

Mounting Volumes

To mount a volume into the driver pod:

--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path=<mount path>
--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly=<true|false>

Example: NFS Volume

--conf spark.kubernetes.driver.volumes.nfs.images.options.server=example.com
--conf spark.kubernetes.driver.volumes.nfs.images.options.path=/data
--conf spark.kubernetes.driver.volumes.nfs.images.mount.path=/mnt/nfs

Example: Persistent Volume Claim

--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=check-point-pvc-claim
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint

On-Demand PVC

You can mount a dynamically-created persistent volume claim per executor by using OnDemand as a claim name:

--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName=OnDemand
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass=gp
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit=500Gi
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path=/data

Local Storage

Spark supports using volumes to spill data during shuffles and other operations. To use a volume as local storage, the volume’s name should start with spark-local-dir-:

--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=gp
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=500Gi
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data

Introspection and Debugging

Accessing Logs

Logs can be accessed using the Kubernetes API and the kubectl CLI:

$ kubectl -n=<namespace> logs -f <driver-pod-name>

Accessing Driver UI

The UI associated with any application can be accessed locally using kubectl port-forward:

$ kubectl port-forward <driver-pod-name> 4040:4040

Then, the Spark driver UI can be accessed on http://localhost:4040.

Debugging

To get basic information about the scheduling decisions made around the driver pod:

$ kubectl describe pod <spark-driver-pod>

If the pod has encountered a runtime error:

$ kubectl logs <spark-driver-pod>

Deleting the driver pod will clean up the entire Spark application, including all executors, associated service, etc. The driver pod can be thought of as the Kubernetes representation of the Spark application.

RBAC

In Kubernetes clusters with RBAC enabled, users can configure Kubernetes RBAC roles and service accounts used by the various Spark on Kubernetes components. The Spark driver pod uses a Kubernetes service account to access the Kubernetes API server to create and watch executor pods. The service account must have the appropriate permissions.

Creating a Service Account

$ kubectl create serviceaccount spark

Creating a Role Binding

$ kubectl create clusterrolebinding spark-role \
    --clusterrole=edit \
    --serviceaccount=default:spark \
    --namespace=default

Using the Service Account

--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark

Application Management

Kubernetes provides simple application management via the spark-submit CLI tool in cluster mode.

Killing an Application

$ spark-submit --kill spark:spark-pi-1547948636094-driver --master k8s://https://192.168.2.8:8443

Checking Application Status

$ spark-submit --status spark:spark-pi-1547948636094-driver --master k8s://https://192.168.2.8:8443

Using Glob Patterns

$ spark-submit --kill spark:spark-pi* --master k8s://https://192.168.2.8:8443

You can specify the grace period for pod termination via the spark.kubernetes.appKillPodDeletionGracePeriod property (default is 30 seconds).

Get Started

Core Concepts

Spark SQL

Structured Streaming

Machine Learning

Graph Processing

Deployment

Configuration & Tuning

Monitoring

​Prerequisites

Kubernetes Cluster

Permissions

DNS

Docker Images

​How It Works

​Docker Images

​Language-Specific Images

​Submitting Applications to Kubernetes

​Cluster Mode

​Finding the API Server URL

​Client Mode

​Dependency Management

​Using S3 for Dependencies

​Secret Management

​Mounting Secrets as Files

​Using Secrets as Environment Variables

​Pod Template

​Using Kubernetes Volumes

​Mounting Volumes

​Example: NFS Volume

​Example: Persistent Volume Claim

​On-Demand PVC

​Local Storage

​Introspection and Debugging

​Accessing Logs

​Accessing Driver UI

​Debugging

​RBAC

​Creating a Service Account

​Creating a Role Binding

​Using the Service Account

​Application Management

​Killing an Application

​Checking Application Status

​Using Glob Patterns

Build docs developers (and LLMs) love

Prerequisites

How It Works

Docker Images

Language-Specific Images

Submitting Applications to Kubernetes

Cluster Mode

Finding the API Server URL

Client Mode

Dependency Management

Using S3 for Dependencies

Secret Management

Mounting Secrets as Files

Using Secrets as Environment Variables

Pod Template

Using Kubernetes Volumes

Mounting Volumes

Example: NFS Volume

Example: Persistent Volume Claim

On-Demand PVC

Local Storage

Introspection and Debugging

Accessing Logs

Accessing Driver UI

Debugging

RBAC

Creating a Service Account

Creating a Role Binding

Using the Service Account

Application Management

Killing an Application

Checking Application Status

Using Glob Patterns