Different Running Modes of Apache Spark

[DataFlair Team]

Different Running Modes of Apache Spark

[DataFlair Team]

Apache Spark can be run in following three mode :

(1) Local mode
(2) Standalone mode
(3) Cluster mode

To study in detail about Spark running mode follow http://data-flair.training/blogs/apache-spark-cluster-managers-tutorial/

[DataFlair Team]

We can launch spark application in four modes:

1) Local Mode (local[*],local,local[2]…etc)
-> When you launch spark-shell without control/configuration argument, It will launch in local mode
spark-shell –master local[1]
-> spark-submit –class com.df.SparkWordCount SparkWC.jar local[1]

2) Spark Standalone cluster manger:
-> spark-shell –master spark://hduser:7077
-> spark-submit –class com.df.SparkWordCount SparkWC.jar spark://hduser:7077

3) Yarn mode (Client/Cluster mode):
-> spark-shell –master yarn or
(or)
->spark-shell –master yarn –deploy-mode client

Above both commands are same.
To launch spark application in cluster mode, we have to use spark-submit command. We cannot run yarn-cluster mode via spark-shell because when we run spark application, driver program will be running as part application master container/process. So it is not possible to run cluster mode via spark-shell.
-> spark-submit –class com.df.SparkWordCount SparkWC.jar yarn-client
-> spark-submit –class com.df.SparkWordCount SparkWC.jar yarn-cluster

4) Mesos mode:
-> spark-shell –master mesos://HOST:5050

[Author]

Posts