What is the role of Spark Driver in spark applications

Viewing 2 reply threads
  • Author
    Posts
    • #5835
      DataFlair Team
      Moderator

      What is the use of Spark driver, where it gets executed on the cluster ?

    • #5836
      DataFlair Team
      Moderator

      The spark driver is that the program that defines the transformations and actions on RDDs of knowledge and submits request to the master. Spark driver is a program that runs on the master node of the machine which declares transformations and actions on knowledge RDDs.

      In easy terms, the driver in Spark creates SparkContext, connected to a given Spark Master.It conjointly delivers the RDD graphs to Master, wherever the standalone cluster manager runs.

      Also see How Spark works

    • #5837
      DataFlair Team
      Moderator

      Spark driver (aka an application’s driver process) is a JVM process that hosts SparkContextfor a Spark application. It is the master node in a Spark application.
      It is the cockpit of jobs and tasks execution (using DAGScheduler and Task Scheduler).
      It hosts Web UI for the environment.
      It splits a Spark application into tasks and schedules them to run on executors.
      A driver is where the task scheduler lives and spawns tasks across workers.
      A driver coordinates workers and overall execution of tasks.

Viewing 2 reply threads
  • You must be logged in to reply to this topic.