{"id":864,"date":"2016-08-02T14:55:45","date_gmt":"2016-08-02T14:55:45","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=864"},"modified":"2018-11-19T16:13:14","modified_gmt":"2018-11-19T10:43:14","slug":"spark-installation-standalone-mode","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/spark-installation-standalone-mode\/","title":{"rendered":"Spark Installation in Standalone Mode | Install Apache Spark"},"content":{"rendered":"

1. Objective – Apache Spark Installation<\/h2>\n

This tutorial contains steps for\u00a0Apache Spark<\/strong> <\/a>Installation in Standalone Mode on Ubuntu. The Spark\u00a0standalone mode <\/strong>sets\u00a0the system\u00a0without any existing cluster management software. For example\u00a0Yarn Resource Manager<\/a><\/strong>\u00a0\/ Mesos<\/a><\/strong>. We have spark master and spark worker who divides driver and executors for Spark application in Standalone mode.<\/p>\n

So, let’s start Spark Installation in Standalone Mode.<\/p>\n

\"Spark<\/a>

Spark Installation in Standalone Mode | Install Apache Spark<\/p><\/div>\n

2. Steps to Apache Spark Installation in Standalone Mode<\/h2>\n

Let’s Follow the steps given below for Apache Spark Installation in Standalone Mode-<\/p>\n

i. Platform<\/h3>\n

a. Platform Requirements<\/h4>\n

Operating system:<\/strong> Ubuntu 14.04 or later, we can also use other Linux<\/a> flavors like CentOS, Redhat, etc.<\/p>\n

b. Configure & Setup Platform<\/h4>\n

If you are using Windows \/ Mac Operating System, so, you can\u00a0create a virtual machine and install Ubuntu using VMWare Player<\/a>, or you can create a virtual machine and install Ubuntu using Oracle Virtual Box<\/a>.<\/p>\n

ii. Software you need to install before installing Spark<\/h3>\n

a. Install Java<\/h4>\n

You need to install Java before Spark installation.\u00a0So, let\u2019s begin by installing Java. So, use the below command to download and install Java-
\n[php]$ sudo apt-get install python-software-properties[\/php]
\n[php]$ sudo apt-add-repository ppa:webupd8team\/java[\/php]
\n[php]$ sudo apt-get update[\/php]
\n[php]$ sudo apt-get install oracle-java7-installer[\/php]
\nOn executing this command Java gets start downloading and gets installed.
\nTo check whether installation procedure gets completed and a completely working Java is installed or not and to know the version of Java installed we have to use the below command-
\n[php] $ java -version[\/php]<\/p>\n

b. Installing Scala<\/h4>\n

Download Scala<\/strong><\/p>\n

Download the latest version of Scala<\/a> <\/strong>from http:\/\/www.scala-lang.org\/<\/a>
\nApache Spark<\/strong> is written in Scala, so we need to install Scala to built Spark.\u00a0Follow the steps given below for installing Scala.<\/p>\n

Untar the file<\/strong><\/p>\n

[php] $ sudo tar xvf scala-2.10.4.tgz [\/php]<\/p>\n

Edit <\/strong>Bashrc file<\/strong><\/p>\n

Make an entry for Scala in .bashrc file
\n[php] nano ~\/.bashrc[\/php]
\nAnd add the following path at the end of the file. It means adding the location, where the Scala software file are located to the PATH variable.
\n[php] export SCALA_HOME=Path-where-scala-file-is-located[\/php]
\n[php] export PATH=$PATH:$SCALA_HOME\/bin[\/php]
\nSource the changed .bashrc file by the command
\n[php] source ~\/.bashrc[\/php]<\/p>\n

Verifying Scala Installation<\/strong><\/p>\n

After installation, it is good to verify it. Use the following command for verifying Scala installation.
\n[php] $scala -version[\/php]<\/p>\n

iii.\u00a0Installing Spark<\/h3>\n

Install Spark in standalone mode on a Single node cluster – for Apache Spark Installation in Standalone Mode, simply place Spark setup on the node of the cluster and extract and configure it. Follow this guide If you are planning to install Spark on a multi-node cluster<\/a><\/strong>.<\/p>\n

a.\u00a0Download Spark<\/h4>\n

Download the latest version of Spark from http:\/\/spark.apache.org\/downloads.html<\/a> of\u00a0your choice from the Apache Spark website.
\nFollow the steps given below for installing Spark.<\/p>\n

b.\u00a0Extracting Spark tar<\/h4>\n

Use the following command for extracting the spark tar file.
\n[php] $ tar xvf spark-2.0.0-bin-hadoop2.6.tgz[\/php]<\/p>\n

c.\u00a0Setting up the environment for Spark<\/h4>\n

Make an entry for Spark in .bashrc file
\n[php]nano ~\/.bashrc[\/php]
\nAdd the following line to the ~\/.bashrc file. It means adding the location, where the spark software files are located to the PATH variable.
\n[php] export SPARK_HOME=\/home\/sapna\/spark-2.0.0-bin-hadoop2.6\/
\nexport PATH=$PATH:$SPARK_HOME\/bin[\/php]
\nUse the following command for sourcing the ~\/.bashrc file.
\n[php] $ source ~\/.bashrc[\/php]<\/p>\n

iv.\u00a0Start Spark Services<\/h3>\n

a. Starting a Cluster Manually<\/h4>\n

Now, start a standalone master server by executing-
\n[php].\/sbin\/start-master.sh [\/php]<\/p>\n

\"Spark<\/a>

Spark Installation – Starting a standalone master server<\/p><\/div>\n

\"Spark<\/a>

Spark Installation – Start a standalone master server<\/p><\/div>\n

\"Spark<\/a>

Apache Spark Installation in Standalone Mode<\/p><\/div>\n

After running, the master will print out a spark:\/\/HOST:PORT URL for itself,
\nwhich can be used to connect workers to it, or pass as the \u201cmaster\u201d argument to SparkContext<\/a><\/strong>.
\nYou will get this URL on the master\u2019s web UI, which is http:\/\/localhost:8080
\nPreview of web console of master using local http:\/\/localhost:8080<\/p>\n

\"Spark<\/a>

Spark Installation – Preview of web console of master<\/p><\/div>\n

Similarly, you can start one or more workers and connect them to the master via-<\/p>\n

.\/sbin\/start-slave.sh <master-spark-URL><\/pre>\n
[php].\/sbin\/start-slave.sh spark:\/\/sapna-All-Series:7077[\/php]<\/pre>\n
\"Spark<\/a>
Spark Installation – start one or more workers and connect them to the master<\/p><\/div>\n
\"Spark<\/a>
Spark Installation<\/p><\/div>\n
Note:<\/strong> You can copy master-spark-Url from master web console\u00a0(http:\/\/localhost:8080)<\/em><\/p>\n
b. Check whether Spark daemons are working<\/h4>\n
[php]jps
\n7698 Master
\n4582 Worker
\n[\/php]<\/p>\n
\"Spark<\/a>
Spark Installation – Check whether the Spark daemons are working or not<\/p><\/div>\n
v. Running sample\u00a0Spark\u00a0application<\/h3>\n
Once you have done Apache Spark Installation in Standalone Mode Let’s run Apache Spark Pi example (the jar for the example is shipped with Spark)
\n[php].\/bin\/spark-submit –class org.apache.spark.examples.SparkPi –master spark:\/\/sapna-All-Series:7077 –executor-memory 1G –total-executor-cores 1 \/home\/sapna\/spark-2.0.0-bin-hadoop2.6\/examples\/jars\/spark-examples_2.11-2.0.0.jar 10[\/php]
\n–class: The entry point for your application.
\n–master: The master URL for the cluster.
\n–executor-memory: Specify memory to be\u00a0allocated for the application.
\n–total-executor-cores:\u00a0Specify no. of CPU cores\u00a0to be\u00a0allocated for the application.<\/p>\n
\"Spark<\/a>
Spark Installation – Apache Spark Pi example<\/p><\/div>\n
Result:<\/strong>
\nPi is roughly 3.141803141803142<\/p>\n
vi. Starting the\u00a0Spark Shell<\/h3>\n
[php]$ bin\/spark-shell spark:\/\/sapna-All-Series:7077[\/php]<\/p>\n
\"Spark<\/a>
Spark Installation – Starting the Spark Shell<\/p><\/div>\n
Now to play with Spark, firstly\u00a0create RDD<\/a><\/strong> and perform various RDD operations using\u00a0this Spark shell commands tutorial.<\/a>
\nSee Also-<\/b><\/p>\n