Free Online Certification Courses – Learn Today. Lead Tomorrow. › Forums › Apache Spark › Explain the run-time architecture of Spark?
- This topic has 1 reply, 1 voice, and was last updated 5 years, 6 months ago by DataFlair Team.
-
AuthorPosts
-
-
September 20, 2018 at 4:29 pm #5875DataFlair TeamSpectator
Define runtime architecture of Spark.
What are the components of runtime architecture of Spark? Describe them. -
September 20, 2018 at 4:29 pm #5877DataFlair TeamSpectator
There are 3 important components of Runtime architecture of Apache Spark as described below.
-
<li style=”list-style-type: none”>
- Client process
- Driver
- Executor
Responsibilities of the client process component
The client process starts the driver program.
For example, the client process can be a spark-submit script for running applications,
a spark-shell script, or a custom application using Spark API.
The client process prepares the classpath and all configuration options for the Spark application.
It also passes application arguments, if any, to the application running on the driver.Responsibilities of the driver component
The driver orchestrates and monitors the execution of a Spark application.
There’s always one driver per Spark application.
The driver is like a wrapper around the application.
The driver and its subcomponents (the Spark context and scheduler ) are responsible for:-
<li style=”list-style-type: none”>
- requesting memory and CPU resources from cluster managers
- breaking application logic into stages and tasks
- sending tasks to executors
- collecting the results
Responsibilities of the executors
The executors, which is a JVM processes, accept tasks from the driver, execute those tasks,
and return the results to the driver.Each executor has several task slots (or CPU cores) for running tasks in parallel.For detailed description of run-time architecture refer How Spark works.
-
-
AuthorPosts
- You must be logged in to reply to this topic.