Back to Understanding Spark Architecture

Spark Architecture Walkthrough

Follow a Spark application from startup to its first tasks.

Start with the driver

The driver coordinates one Spark application. It asks the cluster manager for CPU and memory so the application has somewhere to run its work.

Driver requesting resources from a cluster manager, with executors hosted on three workers

Start executors on worker nodes

The cluster manager arranges for executor processes to start on workers. A worker supplies CPU and memory. An executor uses those resources to run tasks for the application.

Executor process inside a worker node, with allocated CPU cores and memory

Send tasks to the executors

After the executors register with the driver, it can assign tasks. In this example, two executors have two cores each. With four tasks ready and one core needed per task, all four can run at once.

Two executors with two cores each running four tasks concurrently

Report back to the driver

The executors report progress, failures, or results directly to the driver. The cluster manager handles resources. The driver stays responsible for the work and can assign another task as capacity frees up.

Driver sends tasks directly to executors, which report back to the driver