Different Running Modes of Apache Spark

Viewing 2 reply threads
  • Author
    • #6461
      DataFlair TeamDataFlair Team

      Different Running Modes of Apache Spark

    • #6462
      DataFlair TeamDataFlair Team

      Apache Spark can be run in following three mode :

      (1) Local mode
      (2) Standalone mode
      (3) Cluster mode

      To study in detail about Spark running mode follow http://data-flair.training/blogs/apache-spark-cluster-managers-tutorial/

    • #6463
      DataFlair TeamDataFlair Team

      We can launch spark application in four modes:

      1) Local Mode (local[*],local,local[2]…etc)
      -> When you launch spark-shell without control/configuration argument, It will launch in local mode
      spark-shell –master local[1]
      -> spark-submit –class com.df.SparkWordCount SparkWC.jar local[1]

      2) Spark Standalone cluster manger:
      -> spark-shell –master spark://hduser:7077
      -> spark-submit –class com.df.SparkWordCount SparkWC.jar spark://hduser:7077

      3) Yarn mode (Client/Cluster mode):
      -> spark-shell –master yarn or
      ->spark-shell –master yarn –deploy-mode client

      Above both commands are same.
      To launch spark application in cluster mode, we have to use spark-submit command. We cannot run yarn-cluster mode via spark-shell because when we run spark application, driver program will be running as part application master container/process. So it is not possible to run cluster mode via spark-shell.
      -> spark-submit –class com.df.SparkWordCount SparkWC.jar yarn-client
      -> spark-submit –class com.df.SparkWordCount SparkWC.jar yarn-cluster

      4) Mesos mode:
      -> spark-shell –master mesos://HOST:5050

Viewing 2 reply threads
  • You must be logged in to reply to this topic.