Explain the run-time architecture of Spark?

Viewing 1 reply thread
  • Author
    Posts
    • #5875
      DataFlair Team
      Moderator

      Define runtime architecture of Spark.
      What are the components of runtime architecture of Spark? Describe them.

    • #5877
      DataFlair Team
      Moderator

      There are 3 important components of Runtime architecture of Apache Spark as described below.

        <li style=”list-style-type: none”>
      • Client process
      • Driver
      • Executor

      Responsibilities of the client process component

      The client process starts the driver program.
      For example, the client process can be a spark-submit script for running applications,
      a spark-shell script, or a custom application using Spark API.
      The client process prepares the classpath and all configuration options for the Spark application.
      It also passes application arguments, if any, to the application running on the driver.

      Responsibilities of the driver component

      The driver orchestrates and monitors the execution of a Spark application.
      There’s always one driver per Spark application.
      The driver is like a wrapper around the application.
      The driver and its subcomponents (the Spark context and scheduler ) are responsible for:

        <li style=”list-style-type: none”>
      • requesting memory and CPU resources from cluster managers
      • breaking application logic into stages and tasks
      • sending tasks to executors
      • collecting the results

      Responsibilities of the executors

      The executors, which is a JVM processes, accept tasks from the driver, execute those tasks,
      and return the results to the driver.Each executor has several task slots (or CPU cores) for running tasks in parallel.

      For detailed description of run-time architecture refer How Spark works.

Viewing 1 reply thread
  • You must be logged in to reply to this topic.