Components
Spark applications run as independent sets of processes on a cluster, coordinated by theSparkContext object in your main program (called the driver program).
Specifically, to run on a cluster, the SparkContext can connect to several types of cluster managers (either Spark’s own standalone cluster manager, YARN or Kubernetes), which allocate resources across applications. Once connected, Spark acquires executors on nodes in the cluster, which are processes that run computations and store data for your application.
Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends tasks to the executors to run.
Architecture Highlights
There are several useful things to note about this architecture:Process Isolation
Each application gets its own executor processes, which stay up for the duration of the whole application and run tasks in multiple threads. This isolates applications from each other on both the scheduling side and executor side.
Cluster Manager Agnostic
Spark is agnostic to the underlying cluster manager. As long as it can acquire executor processes, and these communicate with each other, it is relatively easy to run it even on a cluster manager that also supports other applications.
Network Addressability
The driver program must listen for and accept incoming connections from its executors throughout its lifetime. The driver program must be network addressable from the worker nodes.
Network Proximity
Because the driver schedules tasks on the cluster, it should be run close to the worker nodes, preferably on the same local area network. If you’d like to send requests remotely, open an RPC to the driver rather than running a driver far away from the worker nodes.
Cluster Manager Types
The system currently supports several cluster managers:Standalone
A simple cluster manager included with Spark that makes it easy to set up a cluster
Hadoop YARN
The resource manager in Hadoop 3
Kubernetes
An open-source system for automating deployment, scaling, and management of containerized applications
Submitting Applications
Applications can be submitted to a cluster of any type using thespark-submit script. See the application submission guide for details on how to do this.
Monitoring
Each driver program has a web UI, typically on port 4040, that displays information about running tasks, executors, and storage usage.The monitoring guide describes other monitoring options available for Spark applications.
Job Scheduling
Spark gives control over resource allocation both across applications (at the level of the cluster manager) and within applications (if multiple computations are happening on the same SparkContext). The job scheduling overview describes this in more detail.Glossary
The following table summarizes terms you’ll see used to refer to cluster concepts:| Term | Meaning |
|---|---|
| Application | User program built on Spark. Consists of a driver program and executors on the cluster |
| Application jar | A jar containing the user’s Spark application. In some cases users will want to create an “uber jar” containing their application along with its dependencies. The user’s jar should never include Hadoop or Spark libraries, however, these will be added at runtime |
| Driver program | The process running the main() function of the application and creating the SparkContext |
| Cluster manager | An external service for acquiring resources on the cluster (e.g. standalone manager, YARN, Kubernetes) |
| Deploy mode | Distinguishes where the driver process runs. In “cluster” mode, the framework launches the driver inside of the cluster. In “client” mode, the submitter launches the driver outside of the cluster |
| Worker node | Any node that can run application code in the cluster |
| Executor | A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors |
| Task | A unit of work that will be sent to one executor |
| Job | A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect); you’ll see this term used in the driver’s logs |
| Stage | Each job gets divided into smaller sets of tasks called stages that depend on each other (similar to the map and reduce stages in MapReduce); you’ll see this term used in the driver’s logs |