What is role of Driver program in Spark Application ?

Viewing 1 reply thread
  • Author
    Posts
    • #5327
      DataFlair Team
      Moderator

      What is role of Driver program in Spark Application ?

    • #5330
      DataFlair Team
      Moderator
      • Driver program is responsible for launching various parallel operations on the cluster.
      • Driver program contains application’s main() function.
      • It is the process which is running the user code which in turn create the SparkContext object, create RDDsand performs transformation and action operation on RDD.
      • Driver program access Apache Spark through a SparkContext object which represents a connection to computing cluster (From Spark 2.0 onwards we can access SparkContext object through SparkSession).
      • Driver program is responsible for converting user program into the unit of physical execution called task.
      • It also defines distributed datasets on the cluster and we can apply different operations on Dataset (transformation and action).
      • Spark program creates a logical plan called Directed Acyclic graph which is converted to physical execution plan by the driver when driver program runs.
Viewing 1 reply thread
  • You must be logged in to reply to this topic.