Back to Understanding Spark Architecture
Spark Architecture Walkthrough
Follow a Spark application from startup to its first tasks.
Start with the driver
The driver coordinates one Spark application. It asks the cluster manager for CPU and memory so the application has somewhere to run its work.
Start executors on worker nodes
The cluster manager arranges for executor processes to start on workers. A worker supplies CPU and memory. An executor uses those resources to run tasks for the application.
Send tasks to the executors
After the executors register with the driver, it can assign tasks. In this example, two executors have two cores each. With four tasks ready and one core needed per task, all four can run at once.
Report back to the driver
The executors report progress, failures, or results directly to the driver. The cluster manager handles resources. The driver stays responsible for the work and can assign another task as capacity frees up.