![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/Apache-Spark-Compatibility-with-Hadoop.jpg\")
<\/a>Apache Spark Compatibility with Hadoop<\/p><\/div>\n
2.\u00a0How is Spark compatible with Hadoop?<\/h2>\n
It is always mistaken that Spark replaces\u00a0Hadoop<\/a><\/strong>, rather it influences the functionality of Hadoop<\/a><\/strong>. Right from the starting Spark\u00a0read data from and write data to HDFS<\/a> (Hadoop Distributed File System<\/a><\/strong>). Thus we can say that Apache Spark is Hadoop-based data processing engine; it can take over batch and streaming data overheads. Hence, running Spark over Hadoop provides enhanced and more functionality.<\/p>\n Spark Hadoop Compatibility<\/p><\/div>\n In three ways we can use Spark over Hadoop:<\/p>\n Let’s discuss these three ways of Apache Spark compatibility with Hadoop one by one in detail.<\/p>\n Before Launching Spark application in standalone mode refer this guide to learn\u00a0how to install Apache Spark in Standalone Mode(single node cluster)?<\/a> If we launch the application through Spark submit<\/strong>, It automatically distributes the application jar to all worker nodes. For any additional jar specify it through –jars<\/em> flag, use comma as a delimiter. If the application exits with non-zero exit code, the standalone cluster mode will restart your application.<\/p>\n If we want to run Spark application in standalone mode by taking input from HDFS<\/a><\/strong> use the code: Apache Spark running on YARN was added in version 0.6.0 and was improved in later releases. When we run in cluster mode, the driver runs on a different machine as that of the client, so to make available the files that are on the client to SparkContext<\/a>.addJar<\/strong>, add those files with –jars option in the launch command. In YARN, both the application master and executors run inside the \u201ccontainers<\/strong>\u201d. Once the application has completed YARN has two modes to handle container log.<\/p>\n In the case where we turn the log aggregation on using the YARN.log-aggregation-enable\u00a0config, the container logs will copy to HDFS<\/strong> and will delete from the local machine. And later if we want to view this file from anywhere on cluster use the command: In this case, logs keep locally on each machine under\u00a0yarn_app_logs_dir<\/em>, which generally configures to \/tmp\/logs\u00a0or\u00a0$HADOOP_HOME\/logs\/userlogs depending on the Hadoop version and installation. It is an easy way for Hadoop MapReduce 1 user to use Apache Spark. Using this we can run Spark job and Spark Shell without installing Spark<\/a><\/strong> or Scala<\/a><\/strong>, or have administrative rights. The only pre-requisites are HDFS access and MapReduce v1. SIMR is open-sourced and is a joint work of Databricks and UC Berkeley\u00a0AMPLab. Once you download the SIMR, we can try it by typing SIMR<\/strong> allows the user to interact with the driver program. On the master mapper, the SIMR runs the relay server and the relay client is run on the machine that launches SIMR. The input to the client and the output from the driver goes to and fro between the client and the master mapper. Hence, to achieve all this we extensively use HDFS.<\/p>\n In conclusion to Apache Spark compatibility with Hadoop, we can say that Spark is a Hadoop-based data processing framework; it can take over batch and streaming data overheads. Hence, running Spark over Hadoop provides enhanced and extra functionality. After studying Hadoop Spark compatibility follow this guide to learn how Apache Spark works?<\/a><\/strong> <\/span><\/p>\n","protected":false},"excerpt":{"rendered":" 1. Objective In this tutorial on Apache Spark compatibility with Hadoop, we will discuss how Spark is compatible with Hadoop? This tutorial covers three ways to use Apache Spark over Hadoop i.e. Standalone, YARN,...<\/p>\n","protected":false},"author":6,"featured_media":42457,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[2787,13023,13042],"class_list":["post-2595","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-compatibility-of-spark-with-hadoop","tag-spark-and-hadoop-compatibility","tag-spark-compatibility-with-hadoop"],"yoast_head":"\n3. Apache Spark Compatibility with Hadoop<\/h2>\n
![\"Spark](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/08\/apache-spark-compatibility-with-hadoop-2.jpg\")
<\/a>\n
3.1. Launching Spark Application in Standalone Mode<\/h3>\n
\nSpark support two deployment modes for standalone cluster namely the cluster mode<\/strong> and the client mode<\/strong>. In client mode, the driver launch in the same process in which client submits the application. In cluster mode, the driver launch from one of worker node process inside the cluster, the client process exit as it submits the application without waiting for the application to finish.<\/p>\na. Adding the jar<\/h4>\n
b. Running application in Standalone Mode<\/h4>\n
\n[php]$ .\/bin\/spark-submit –class MyApp.class \u00a0–master MyApp.jar –input hdfs:\/\/localhost:9000\/input-file-path –output output-file-path[\/php]<\/p>\n3.2. Launching Spark on YARN<\/h3>\n
\nIf we want to run Spark job on Hadoop YARN cluster(Learn to configure Hadoop with yarn in pseudo distributed mode<\/a>) we first need to merge Spark JAR. It merges all the essential and required dependencies. We can achieve this by setting Hadoop version and SPARK_YARN\u00a0environment variable as:
\nSPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt\/sbt assembly.<\/em>
\nMake sure YARN_CONF_DIR or HADOOP_CONF_DIR indicates those directories which have a configuration file for Hadoop cluster<\/strong><\/a>. Using these configurations we write to HDFS and connect to YARN Resource Manager<\/a><\/strong>. The configurations that are contained in this directory are shared among YARN cluster so that all the containers<\/strong> that are used by applications use the same configuration.
\nTo launch Spark application on YARN there are two deployment modes namely: the cluster mode<\/em> and the client mode<\/em>.
\ni. Cluster Mode –<\/strong>\u00a0In cluster mode, the Spark driver<\/strong> runs inside Application Master Process and this is managed by YARN on the cluster.
\nii. Client Mode –<\/strong>\u00a0In this mode, the driver runs in client process and we use application master only for requesting a resource from YARN and providing it to the driver program.
\nIn YARN mode the address of ResourceManager is taken from Hadoop Configuration. So, here the \u00a0 \u00a0 \u00a0 –master parameter is yarn.
\nIf we want to launch Spark application in cluster mode use the command:
\n[php]$ .\/bin\/spark-submit –class path.to.your.Class –master yarn –deploy-mode cluster [options] <app jar> [app options][\/php]
\nIf we want to launch Spark application in client mode use the command (replace cluster in above by client)
\n[php]$ .\/bin\/spark-submit –class path.to.your.Class –master yarn –deploy-mode client[\/php]<\/p>\na.\u00a0Adding other JARs<\/h4>\n
\nFor Example:<\/strong>
\n[php]$ .\/bin\/spark-submit –class my.main.Class \\
\n–master yarn \\
\n–deploy-mode cluster \\
\n–jars my-other-jar.jar,my-other-other-jar.jar \\
\nmy-main-jar.jar \\
\napp_arg1 app_arg2[\/php]
\nIf we want that Spark runtime jars to be accessible from YARN side, specify-
\nspark.yarn.archive<\/em>\u00a0<\/em>or\u00a0<\/em>spark.yarn.jars<\/em>.<\/em>
\nIf we do not specify these then Spark will form a zip file with all jar under $SPARK_HOME\/jars and upload it to the distributed cache.<\/p>\nb. Debugging Application on YARN<\/h4>\n
i. If log aggregation is turned on<\/h5>\n
\n[php]yarn logs -applicationId <app ID>[\/php]
\nThis command will print out the content from all log files from all containers from given application.
\nWe can also see the container log files in HDFS directly by using HDFS Shell or API. If we want to find the directory where the log file is present, use this command:
\nyarn.nodemanager.remote-app-log-<\/em>dir\u00a0and \u00a0yarn.nodemanager.remote-app-log-dir-suffix<\/em><\/p>\nii. If log aggregation is not turned on<\/h5>\n
\nIf we want to view log from a container, we must first go to the host that contains it and look in the directory. Further, the sub-directory maintains log files by application ID and container ID.<\/p>\n3.3. Launching Spark in MapReduce (SIMR)<\/h3>\n
\n[php].\/simr –shell[\/php]
\nTo use this user only need to download package of SIMR that matches Hadoop cluster.<\/strong><\/a> The package of SIMR contains 3 files:
\nSIMR runtime script: simr
\nsimr-<hadoop-version>.jar
\nspark-assembly-<hadoop-version>.jar
\nTo get usage information place all three in the directory and execute SIMR. The job SIMR is to launch MapReduce jobs with required number of map slots and makes sure that Spark\/Scala and jobs are sent to all those nodes. One of the mappers<\/a><\/strong> is set as master and inside that mapper, Spark driver is made to run. On the remaining mapper SMIR launch the Spark executor, these executors will execute a task on behalf of the driver.
\nThe master is selected by leader election by writing to HDFS<\/a><\/strong>, the mapper which writes first in the HDFS is set as the master mapper. And the remaining mapper finds the driver URL by reading a specific file from HDFS<\/a><\/strong>. Thus, in place of cluster manager SIMR uses MapReduce and HDFS.<\/p>\na. How does SIMR work?<\/h4>\n
4. Conclusion<\/h2>\n
\nIf you feel any query about this post on Apache Spark Compatibility with Hadoop, please feel free to share with us. Hope we will solve them.
\nReference-<\/strong>
\nhttp:\/\/spark.apache.org\/<\/a>
\nSee Also-<\/strong><\/p>\n\n