Security Configuration - Apache Spark

Spark provides comprehensive security features to protect your cluster and data. However, security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s critical to secure access properly.

Spark is not secure by default. You must explicitly enable and configure security features based on your deployment environment.

Security Overview

Spark supports multiple security mechanisms:

RPC Authentication

Authenticate communication between Spark processes

Network Encryption

Encrypt data in transit using TLS or AES

Web UI Security

Control access to Spark web interfaces

Local Storage Encryption

Encrypt temporary data written to disk

Kerberos Integration

Integrate with enterprise authentication systems

ACLs

Control who can view and modify applications

Not all deployment types support all security features. Check your deployment-specific documentation for supported security options.

RPC Authentication

Spark supports authentication for RPC channels using a shared secret. This ensures only authorized processes can communicate within your Spark cluster.

Enabling Authentication

Enable authentication by setting:

spark.authenticate=true

Deployment-Specific Authentication

YARN
Kubernetes
Standalone

For Spark on YARN, Spark automatically handles generating and distributing the shared secret. Each application uses a unique shared secret.

This feature relies on YARN RPC encryption being enabled for secure distribution of secrets.

YARN-specific configuration:

Property	Default	Description
`spark.yarn.shuffle.server.recovery.disabled`	false	Set to true for applications with higher security requirements. Secret won’t be saved in the database, but shuffle data won’t be recovered after External Shuffle Service restarts.

On Kubernetes, Spark automatically generates an authentication secret unique to each application. The secret is propagated to executor pods using environment variables.

Any user who can list pods in the namespace can see authentication secrets. Ensure proper access control rules are set up by the Kubernetes admin.

File-based authentication:

Property	Default	Description
`spark.authenticate.secret.file`	None	Path to secret key file. File is loaded on both driver and executors.
`spark.authenticate.secret.driver.file`	Value of `spark.authenticate.secret.file`	Override location for driver in client mode.
`spark.authenticate.secret.executor.file`	Value of `spark.authenticate.secret.file`	Override location for executors in client mode.

Spark will not mount these files into containers for you. You must ensure secret files are deployed securely into your containers.

For standalone mode, you must define the shared secret manually:

spark.authenticate=true
spark.authenticate.secret=your-secret-key-here

The same secret is shared by all Spark applications and daemons, which limits security especially on multi-tenant clusters.

REST Submission Server Authentication

The REST Submission Server supports HTTP Authorization header with JSON Web Token (JWT):

# On Spark Master
spark.master.rest.filters=org.apache.spark.ui.JWSFilter
spark.org.apache.spark.ui.JWSFilter.param.secretKey=BASE64URL-ENCODED-KEY

Clients must provide an HTTP Authorization header containing a JWT signed by the shared secret key.

Network Encryption

Spark supports two mutually exclusive forms of encryption for RPC connections:

TLS/SSL Encryption (Recommended)
AES-based Encryption (Legacy)

The preferred method uses TLS encryption via Netty’s support for SSL. This is standardized and considered more secure.Key benefits:

Industry-standard protocol
Better security library
Compliance with security policies

SSL is not automatically enabled even if spark.ssl.enabled is set. You must explicitly enable it for RPC.

Configuration:

spark.ssl.rpc.enabled=true
spark.ssl.rpc.protocol=TLSv1.3
spark.ssl.rpc.keyStore=/path/to/keystore.jks
spark.ssl.rpc.keyStorePassword=password
spark.ssl.rpc.trustStore=/path/to/truststore.jks
spark.ssl.rpc.trustStorePassword=password

The legacy method is an AES-based encryption mechanism relying on a shared secret. This requires RPC authentication to also be enabled.When to use:

Simpler configuration needs
Only requirement is data encryption in transit
Not bound by compliance requirements

The legacy protocol has two versions. Version 2 is recommended for better security properties.

Configuration:

spark.authenticate=true
spark.network.crypto.enabled=true
spark.network.crypto.cipher=AES/GCM/NoPadding
spark.network.crypto.authEngineVersion=2

AES Encryption Configuration

Property	Default	Description
`spark.network.crypto.enabled`	false	Enable AES-based RPC encryption, including the new authentication protocol added in 2.2.0.
`spark.network.crypto.cipher`	AES/CTR/NoPadding	Cipher mode to use. Recommended: `AES/GCM/NoPadding` (authenticated encryption).
`spark.network.crypto.authEngineVersion`	1	Version of AES-based RPC encryption (1 or 2). Version 2 is recommended.
`spark.network.crypto.saslFallback`	true	Whether to fall back to SASL authentication for old shuffle services.
`spark.authenticate.enableSaslEncryption`	false	Enable SASL-based encrypted communication (deprecated).

Local Storage Encryption

Spark supports encrypting temporary data written to local disks, including shuffle files, shuffle spills, and cached data blocks.

Local storage encryption does not cover output data generated by applications with APIs like saveAsHadoopFile or saveAsTable.

Enabling Storage Encryption

spark.io.encryption.enabled=true
spark.io.encryption.keySizeBits=256
spark.io.encryption.keygen.algorithm=HmacSHA1

It’s strongly recommended that RPC encryption be enabled when using local storage encryption.

Storage Encryption Configuration

Property	Default	Description
`spark.io.encryption.enabled`	false	Enable local disk I/O encryption. Supported by all deployment modes.
`spark.io.encryption.keySizeBits`	128	IO encryption key size in bits. Supported values: 128, 192, 256.
`spark.io.encryption.keygen.algorithm`	HmacSHA1	Algorithm to use when generating the IO encryption key.

Web UI Security

Authentication and Authorization

Spark supports access control to the Web UI when an authentication filter is present.

Spark does not provide built-in authentication filters. You must implement or use a third-party filter.

Access Control Lists (ACLs)

Spark differentiates between two types of permissions:

View permissions: Who can see the application’s UI
Modify permissions: Who can kill jobs in a running application

Configuration:

# Enable ACLs
spark.acls.enable=true

# Admin users (view and modify access)
spark.admin.acls=user1,user2
spark.admin.acls.groups=admin-group

# View-only access
spark.ui.view.acls=viewer1,viewer2
spark.ui.view.acls.groups=viewer-group

# Modify access
spark.modify.acls=operator1,operator2
spark.modify.acls.groups=operator-group

Use a wildcard (*) to give all users the respective privilege. By default, only the user submitting the application is added to the ACLs.

Web UI ACL Configuration

Property	Default	Description
`spark.acls.enable`	false	Whether UI ACLs should be enabled. Requires authentication filter to be installed.
`spark.admin.acls`	None	Comma-separated list of users with view and modify access.
`spark.admin.acls.groups`	None	Comma-separated list of groups with view and modify access.
`spark.ui.view.acls`	None	Comma-separated list of users with view access.
`spark.ui.view.acls.groups`	None	Comma-separated list of groups with view access.
`spark.modify.acls`	None	Comma-separated list of users with modify access.
`spark.modify.acls.groups`	None	Comma-separated list of groups with modify access.
`spark.user.groups.mapping`	ShellBasedGroupsMappingProvider	Group mapping service for determining user groups.

History Server ACLs

spark.history.ui.acls.enable=true
spark.history.ui.admin.acls=admin1,admin2
spark.history.ui.admin.acls.groups=admin-group

SSL Configuration

SSL configuration is organized hierarchically. You can configure default SSL settings that apply to all protocols unless overridden.

SSL Namespaces

Namespace	Component
`spark.ssl`	Default SSL configuration for all namespaces
`spark.ssl.ui`	Spark application Web UI
`spark.ssl.standalone`	Standalone Master / Worker Web UI
`spark.ssl.historyServer`	History Server Web UI
`spark.ssl.rpc`	Spark RPC communication

All settings are inherited from the default namespace, except for spark.ssl.rpc.enabled which must be explicitly set.

Common SSL Configuration

Property	Default	Description	Namespaces
`${ns}.enabled`	false	Enables SSL. Requires `${ns}.protocol` to be set.	ui,standalone,historyServer,rpc
`${ns}.protocol`	None	TLS protocol to use (e.g., TLSv1.2, TLSv1.3).	ui,standalone,historyServer,rpc
`${ns}.keyStore`	None	Path to the key store file.	ui,standalone,historyServer,rpc
`${ns}.keyStorePassword`	None	Password to the key store.	ui,standalone,historyServer,rpc
`${ns}.keyPassword`	None	Password to the private key in the key store.	ui,standalone,historyServer,rpc
`${ns}.trustStore`	None	Path to the trust store file.	ui,standalone,historyServer,rpc
`${ns}.trustStorePassword`	None	Password for the trust store.	ui,standalone,historyServer,rpc

SSL Configuration Example

# Default SSL settings
spark.ssl.enabled=true
spark.ssl.protocol=TLSv1.3
spark.ssl.keyStore=/path/to/keystore.jks
spark.ssl.keyStorePassword=changeit
spark.ssl.trustStore=/path/to/truststore.jks
spark.ssl.trustStorePassword=changeit

# RPC-specific (must be explicitly enabled)
spark.ssl.rpc.enabled=true

Generating Key Stores

Generate key stores using the keytool program:

Generate Key Pair

keytool -genkeypair -alias spark -keyalg RSA -keysize 2048 \
  -keystore keystore.jks -storepass changeit

Export Public Key

keytool -exportcert -alias spark -keystore keystore.jks \
  -file spark.cer -storepass changeit

Import to Trust Store

keytool -importcert -alias spark -file spark.cer \
  -keystore truststore.jks -storepass changeit

Distribute Files

Distribute key stores and trust stores to all nodes in your cluster.

HTTP Security Headers

Spark can include HTTP headers to prevent Cross-Site Scripting (XSS) and other attacks:

# XSS Protection
spark.ui.xXssProtection=1; mode=block

# Content Type Options
spark.ui.xContentTypeOptions.enabled=true

# Strict Transport Security (HTTPS only)
spark.ui.strictTransportSecurity=max-age=31536000; includeSubDomains

Property	Default	Description
`spark.ui.xXssProtection`	1; mode=block	Value for HTTP X-XSS-Protection response header.
`spark.ui.xContentTypeOptions.enabled`	true	When enabled, X-Content-Type-Options is set to “nosniff”.
`spark.ui.strictTransportSecurity`	None	Value for HTTP Strict Transport Security (HSTS). Only used when SSL/TLS is enabled.

Kerberos Integration

Spark supports Kerberos authentication for deployments in secure Hadoop environments.

Kerberos Overview

Use kinit to obtain credentials from the configured KDC.

Delegation Tokens

Spark automatically obtains delegation tokens for HDFS, Hive, and HBase (if in classpath and properly configured).

Token Renewal

For long-running applications, Spark can automatically renew tokens using a keytab or ticket cache.

Supported Services

Spark ships with delegation token support for:

HDFS and other Hadoop filesystems
Hive (when hive.metastore.uris is not empty)
HBase (when hbase.security.authentication=kerberos)

Kerberos Configuration

Property	Default	Description
`spark.security.credentials.${service}.enabled`	true	Controls whether to obtain credentials for services when security is enabled.
`spark.kerberos.access.hadoopFileSystems`	(none)	Comma-separated list of secure Hadoop filesystems your application will access.

Example:

spark.kerberos.access.hadoopFileSystems=hdfs://nn1.com:8032,hdfs://nn2.com:8032,webhdfs://nn3.com:50070

Long-Running Applications

For applications that run longer than the maximum delegation token lifetime:

Using a Keytab
Using Ticket Cache

Provide Spark with a principal and keytab:

spark-submit \
  --principal user@REALM \
  --keytab /path/to/user.keytab \
  --class com.example.MyApp \
  myapp.jar

In cluster mode, the keytab is copied to the driver machine. Ensure YARN and HDFS are secured with encryption.

Use the local Kerberos ticket cache:

spark.kerberos.renewal.credentials=ccache

Spark will keep the ticket renewed during its renewable life. After expiration, you must run kinit again.

Customize ticket cache location:

export KRB5CCNAME=/tmp/krb5cc_1000

Kubernetes with Kerberos

Three methods to submit Kerberos jobs on Kubernetes:

Local Ticket Cache
Local Keytab
Pre-populated Secrets

kinit -kt <keytab_file> <username>/<krb5 realm>
spark-submit \
  --deploy-mode cluster \
  --master k8s://<endpoint> \
  --conf spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf \
  local:///opt/spark/examples/jars/spark-examples.jar

spark-submit \
  --deploy-mode cluster \
  --master k8s://<endpoint> \
  --conf spark.kerberos.keytab=<keytab_file> \
  --conf spark.kerberos.principal=<principal> \
  --conf spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf \
  local:///opt/spark/examples/jars/spark-examples.jar

spark-submit \
  --deploy-mode cluster \
  --master k8s://<endpoint> \
  --conf spark.kubernetes.kerberos.tokenSecret.name=<secret_name> \
  --conf spark.kubernetes.kerberos.tokenSecret.itemKey=<secret_key> \
  --conf spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf \
  local:///opt/spark/examples/jars/spark-examples.jar

In all cases, define HADOOP_CONF_DIR or spark.kubernetes.hadoop.configMapName and ensure the KDC is visible from inside containers.

Configuring Network Ports

Generally, Spark services are private and should only be accessible within your organization’s network. Limit access to hosts and ports used by Spark services.

Standalone Mode Ports

From	To	Default Port	Purpose	Configuration
Browser	Standalone Master	8080	Web UI	`spark.master.ui.port`
Browser	Standalone Worker	8081	Web UI	`spark.worker.ui.port`
Driver/Worker	Standalone Master	7077	Submit job/Join cluster	`SPARK_MASTER_PORT`
External Service	Standalone Master	6066	REST API	`spark.master.rest.port`
Master	Worker	(random)	Schedule executors	`SPARK_WORKER_PORT`

All Cluster Managers

From	To	Default Port	Purpose	Configuration
Browser	Application	4040	Web UI	`spark.ui.port`
Browser	History Server	18080	Web UI	`spark.history.ui.port`
Executor/Master	Driver	(random)	Connect to application	`spark.driver.port`
Executor/Driver	Executor/Driver	(random)	Block Manager port	`spark.blockManager.port`

Securing Ports with JWT

Spark supports HTTP Authorization header with JWT for all UI ports:

spark.ui.filters=org.apache.spark.ui.JWSFilter
spark.org.apache.spark.ui.JWSFilter.param.secretKey=BASE64URL-ENCODED-KEY

Security Best Practices

Enable Authentication

Always enable RPC authentication (spark.authenticate=true) in production environments, especially on multi-tenant clusters.

Use TLS Encryption

Enable TLS/SSL encryption for RPC and Web UI communication. Use version 2 of AES encryption if TLS is not feasible.

Encrypt Local Storage

Enable local storage encryption (spark.io.encryption.enabled=true) for sensitive data.

Configure ACLs

Set up proper access control lists to restrict who can view and modify applications.

Secure Event Logs

Set event log directory permissions to drwxrwxrwxt to prevent unauthorized access.

Use Kerberos

Integrate with Kerberos in enterprise environments for centralized authentication.

Regular Security Audits

Regularly review security configurations and update to latest Spark versions with security patches.

Event Logging Security

If your applications use event logging, secure the log directory:

# Create directory with proper permissions
mkdir -p /var/log/spark-events
chmod 1777 /var/log/spark-events
chown spark:spark /var/log/spark-events

Permissions drwxrwxrwxt allow all users to write but prevent unprivileged users from reading, removing, or renaming files they don’t own.

Get Started

Core Concepts

Spark SQL

Structured Streaming

Machine Learning

Graph Processing

Deployment

Configuration & Tuning

Monitoring

​Security Overview

RPC Authentication

Network Encryption

Web UI Security

Local Storage Encryption

Kerberos Integration

ACLs

​RPC Authentication

​Enabling Authentication

​Deployment-Specific Authentication

​REST Submission Server Authentication

​Network Encryption

​AES Encryption Configuration

​Local Storage Encryption

​Enabling Storage Encryption

​Storage Encryption Configuration

​Web UI Security

​Authentication and Authorization

​Access Control Lists (ACLs)

​Web UI ACL Configuration

​History Server ACLs

​SSL Configuration

​SSL Namespaces

​Common SSL Configuration

​SSL Configuration Example

​Generating Key Stores

​HTTP Security Headers

​Kerberos Integration

​Kerberos Overview

​Supported Services

​Kerberos Configuration

​Long-Running Applications

​Kubernetes with Kerberos

​Configuring Network Ports

​Standalone Mode Ports

​All Cluster Managers

​Securing Ports with JWT

​Security Best Practices

​Event Logging Security

​Next Steps

Spark Configuration

Performance Tuning

Build docs developers (and LLMs) love

Security Overview

RPC Authentication

Enabling Authentication

Deployment-Specific Authentication

REST Submission Server Authentication

Network Encryption

AES Encryption Configuration

Local Storage Encryption

Enabling Storage Encryption

Storage Encryption Configuration

Web UI Security

Authentication and Authorization

Access Control Lists (ACLs)

Web UI ACL Configuration

History Server ACLs

SSL Configuration

SSL Namespaces

Common SSL Configuration

SSL Configuration Example

Generating Key Stores

HTTP Security Headers

Kerberos Integration

Kerberos Overview

Supported Services

Kerberos Configuration

Long-Running Applications

Kubernetes with Kerberos

Configuring Network Ports

Standalone Mode Ports

All Cluster Managers

Securing Ports with JWT

Security Best Practices

Event Logging Security

Next Steps