Explain the run-time architecture of Spark?

[DataFlair Team]

Define runtime architecture of Spark.
What are the components of runtime architecture of Spark? Describe them.

[DataFlair Team]

There are 3 important components of Runtime architecture of Apache Spark as described below.

<li style=”list-style-type: none”>

• Client process
• Driver
• Executor

Responsibilities of the client process component

The client process starts the driver program.
For example, the client process can be a spark-submit script for running applications,
a spark-shell script, or a custom application using Spark API.
The client process prepares the classpath and all configuration options for the Spark application.
It also passes application arguments, if any, to the application running on the driver.

Responsibilities of the driver component

The driver orchestrates and monitors the execution of a Spark application.
There’s always one driver per Spark application.
The driver is like a wrapper around the application.
The driver and its subcomponents (the Spark context and scheduler ) are responsible for:

<li style=”list-style-type: none”>

• requesting memory and CPU resources from cluster managers
• breaking application logic into stages and tasks
• sending tasks to executors
• collecting the results

Responsibilities of the executors

The executors, which is a JVM processes, accept tasks from the driver, execute those tasks,
and return the results to the driver.Each executor has several task slots (or CPU cores) for running tasks in parallel.

For detailed description of run-time architecture refer How Spark works.

[Author]

Posts