How to Install Apache Spark on Multi-Node Cluster 27


1. Objective

This Spark tutorial explains how to install Apache Spark on a multi-node cluster. This guide provides step by step instructions to deploy and configure Apache Spark on the real multi-node cluster. Once the setup and installation are done you can play with Spark and process data.

Learn, How to Install Apache Spark 2.x on Multi-Node Cluster?

2. Steps to install Apache Spark on multi-node cluster

Follow the steps given below to easily install Apache Spark on a multi-node cluster.

2.1. Recommended Platform

  • OS – Linux is supported as a development and deployment platform. You can use Ubuntu 14.04 / 16.04 or later (you can also use other Linux flavors like CentOS, Redhat, etc.). Windows is supported as a dev platform. (If you are new to Linux follow this Linux commands manual).
  • Spark – Apache Spark 2.x

For Apache Spark Installation On Multi-Node Cluster, we will be needing multiple nodes, either you can use Amazon AWS or follow this guide to setup virtual platform using VMWare player.

2.2. Install Spark on Master

I. Prerequisites

a. Add Entries in hosts file

Edit hosts file

sudo nano /etc/hosts

Now add entries of master and slaves

MASTER-IP master
SLAVE01-IP slave01
SLAVE02-IP slave02

(NOTE: In place of MASTER-IP, SLAVE01-IP, SLAVE02-IP put the value of the corresponding IP)

b. Install Java 7 (Recommended Oracle Java)

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

c. Install Scala

sudo apt-get install scala

d. Configure SSH
i. Install Open SSH Server-Client

sudo apt-get install openssh-server openssh-client

ii. Generate Key Pairs

ssh-keygen -t rsa -P ""

iii. Configure passwordless SSH

Copy the content of .ssh/id_rsa.pub (of master) to .ssh/authorized_keys (of all the slaves as well as master)

iv. Check by SSH to all the Slaves

ssh slave01
ssh slave02

II. Install Spark

a. Download Spark

You can download the latest version of spark from http://spark.apache.org/downloads.html.

b. Untar Tarball

tar xzf spark-2.0.0-bin-hadoop2.6.tgz

(Note: All the scripts, jars, and configuration files are available in newly created directory “spark-2.0.0-bin-hadoop2.6”)

c. Setup Configuration
i. Edit .bashrc

Now edit .bashrc file located in user’s home directory and add following environment variables:

export JAVA_HOME=<path-of-Java-installation> (eg: /usr/lib/jvm/java-7-oracle/)
export SPARK_HOME=<path-to-the-root-of-your-spark-installation> (eg: /home/dataflair/spark-2.0.0-bin-hadoop2.6/)
export PATH=$PATH:$SPARK_HOME/bin

(Note: After above step restart the Terminal/Putty so that all the environment variables will come into effect)

ii. Edit spark-env.sh

Now edit configuration file spark-env.sh (in $SPARK_HOME/conf/) and set following parameters:

Note: Create a copy of template of spark-env.sh and rename it:

cp spark-env.sh.template spark-env.sh

export JAVA_HOME=<path-of-Java-installation> (eg: /usr/lib/jvm/java-7-oracle/)
export SPARK_WORKER_CORES=8

iii. Add Salves

Create configuration file slaves (in $SPARK_HOME/conf/) and add following entries:

slave01
slave02

“Apache Spark has been installed successfully on Master, now deploy Spark on all the Slaves”

2.3. Install Spark On Slaves

I. Setup Prerequisites on all the slaves

Run following steps on all the slaves (or worker nodes):

  • “1.1. Add Entries in hosts file”
  • “1.2. Install Java 7”
  • “1.3. Install Scala”

II. Copy setups from master to all the slaves

a. Create tarball of configured setup

tar czf spark.tar.gz spark-2.0.0-bin-hadoop2.6

NOTE: Run this command on Master

b. Copy the configured tarball on all the slaves

scp spark.tar.gz slave01:~

NOTE: Run this command on Master

scp spark.tar.gz slave02:~

NOTE: Run this command on Master

III. Un-tar configured spark setup on all the slaves

tar xzf spark.tar.gz

NOTE: Run this command on all the slaves

“Congratulations Apache Spark has been installed on all the Slaves. Now Start the daemons on the Cluster”

2.4. Start Spark Cluster

I. Start Spark Services

sbin/start-all.sh

Note: Run this command on Master

II. Check whether services have been started

a. Check daemons on Master

jps
Master

b. Check daemons on Slaves

jps
Worker

2.5. Spark Web UI

I. Spark Master UI

Browse the Spark UI to know about worker nodes, running application, cluster resources.

http://MASTER-IP:8080/

II. Spark application UI

http://MASTER-IP:4040/

2.6. Stop the Cluster

I. Stop Spark Services

Once all the applications have finished, you can stop the spark services (master and slaves daemons) running on the cluster

sbin/stop-all.sh

Note: Run this command on Master

After Apache Spark installation, I recommend learning Spark RDD, DataFrame, and Dataset. You can proceed further with Spark shell commands to play with Spark.

3. Conclusion

After installing the Apache Spark on the multi-node cluster you are now ready to work with Spark platform. Now you can play with the data, create an RDD, perform operations on those RDDs over multiple nodes and much more.

If you have any query to install Apache Spark, so, feel free to share with us. We will be happy to solve them.

See Also-

Reference:

http://spark.apache.org/


27 thoughts on “How to Install Apache Spark on Multi-Node Cluster

  • Ashish Garg

    Thanks for the this great tutorial

    Don’t we need to setup the HDFS to share the repository with master and all workers?
    Can you share the tutorial for this?

  • Mady

    Fantastic blog to install Spark 2 in easy steps. Please share some Spark practicals as well to start with.

  • Nitin

    Thanks for this lovely article. However, I am facing one problem when doing “jps Master” it is throwing “RMI Registry not available at Master:1099
    Connection refused to host: Master; nested exception is:
    java.net.ConnectException: Connection refused”
    this error. Can you help?

  • Krish Rajaram

    Thanks for this post. I followed these steps and successfully created the cluster with spark 2.1.0. While I was testing a simple dataframe writer, it fails to write the output file to the target path. This happens only when run through spark-submit. But when I run the commands from spark-shell the output file is successfully stored in the target path. Did anyone encounter this issue?

  • Abdel

    Hi ! thanks for this article it’s very helpful. however I did not undestand this part of your tutorial:
    2.3.3 Add salves:
    Create configuration file slaves (in $SPARK_HOME/conf/) and add following entries:
    1 slave01
    2 slave02
    Do we have to add this entries in the file spark-env.sh or what ?

    Thanks in advance

    • Ganesh

      Add these entries into a new slaves file like following:
      $cp slaves.template slaves (to copy the slaves.template file to another file named as slaves)

      $vim slaves
      master
      slave01
      slave02

  • Swaroop P

    I followed all your steps as you mentioned.

    I am unable to connect workers. Only master is acting as master and worker form me.

    Is the above process required hadoop installation? Because i didn’t install hadoop or yarn.

    Please help me ASAP

    • Dinesh Dev Pandey

      Hi,
      I was facing the same problem. I checked the log generated for master. I found –
      “Service MasterUI is started on port 8081”.
      I tried with http://Master_IP:8081 and it worked for me.

      You can also check logs once.

  • Ugur

    Thanks for your awesome sharing,
    However, I have a problem. I setup multi-node spark according to your guidance but i cannot access with ip of master node(x.y.z.t:8080). How can i solve the problem?

    • Emiliano Amendola

      You need to add these two lines in the ~/$SPARK_HOME/conf/spark-env.sh file, in your master and worker nodes:

      export SPARK_MASTER_HOST= YOUR.MASTER.IP.ADDRESS
      export SPARK_MASTER_WEBUI_PORT=8080

  • rama

    Thanks for your awesome sharing, I have installed Spark on multiple nodes successfully.

  • Djibrina B.

    Thx for this article.
    However i would like to know how to set up hdfs to enable all workers and master to share the same repository?
    I installed a Spark-Cluster with 3 workers and i would like to save a dataframe along all workers. I created on each worker the repository ” home/data/”. It saves but if i read it back, i am geting “lost files error: java.io.FileNotFoundException: file part XXXX does not exist”.
    I tried also using parquet and using partitions by column y but i still get the same kind of error “file footer not found ”

    Any suggestions please?
    Thx

  • Vaasir Nisaar

    I have installed MapR with 1-Control node and 2-Data nodes but now im going to install Apache Spark with all nodes using Python how im going to develop.

  • Begna

    Best tutorial, I have wasted my time on other alternatives. Really am happy and helped me a lot for my Project.
    Thank thank you

  • Lebin

    Very Nice article.
    I have a doubt, how to execute the job after configuring the cluster? is it necessary to copy the jar in all the nodes(master as well as in slave)? Will it work if i can do it by only keeping the jar in master node?

  • Umesh

    Hi,
    Thank you for the article . However when I am trying to submit job on master it is not sending it to the slave node. COuld you please help me here?
    detailed description:

    I am deploying prediction.io on a multinode cluster where training should happen on the worker node. The worker node has been successfully registered with the master.

    following are the logs of after starting slaves.sh
    Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties 18/05/22 06:01:44 INFO Worker: Started daemon with process name: 2208@ip-172-31-6-235 18/05/22 06:01:44 INFO SignalUtils: Registered signal handler for TERM 18/05/22 06:01:44 INFO SignalUtils: Registered signal handler for HUP 18/05/22 06:01:44 INFO SignalUtils: Registered signal handler for INT 18/05/22 06:01:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 18/05/22 06:01:44 INFO SecurityManager: Changing view acls to: ubuntu 18/05/22 06:01:44 INFO SecurityManager: Changing modify acls to: ubuntu 18/05/22 06:01:44 INFO SecurityManager: Changing view acls groups to: 18/05/22 06:01:44 INFO SecurityManager: Changing modify acls groups to: 18/05/22 06:01:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); groups with view permissions: Set(); users with modify permissions: Set(ubuntu); groups with modify permissions: Set() 18/05/22 06:01:44 INFO Utils: Successfully started service ‘sparkWorker’ on port 45057. 18/05/22 06:01:44 INFO Worker: Starting Spark worker 172.31.6.235:45057 with 8 cores, 24.0 GB RAM 18/05/22 06:01:44 INFO Worker: Running Spark version 2.1.1 18/05/22 06:01:44 INFO Worker: Spark home: /home/ubuntu/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6 18/05/22 06:01:45 INFO Utils: Successfully started service ‘WorkerUI’ on port 8081. 18/05/22 06:01:45 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://172.31.6.235:8081 18/05/22 06:01:45 INFO Worker: Connecting to master ip-172-31-5-119.ap-southeast-1.compute.internal:7077… 18/05/22 06:01:45 INFO TransportClientFactory: Successfully created connection to ip-172-31-5-119.ap-southeast-1.compute.internal/172.31.5.119:7077 after 19 ms (0 ms spent in bootstraps) 18/05/22 06:01:45 INFO Worker: Successfully registered with master spark://ip-172-31-5-119.ap-southeast-1.compute.internal:7077

    Now the issues:

    if I launch one slave on master and one slave my other node:
    1.1 if the slave of the master node is given fewer resources it will give some unable to re-shuffle error.
    1.2 if I give more resources to the worker on the master node the all the execution happens on master node, it does not send any execution to the slave node.
    If I do not start a slave on the master node:
    2.1 I get the following error:
    WARN] [TaskSchedulerImpl] Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

    I have assigned 24gb ram to the worker and 8 cores.

    However, while I start the process following are the logs I get on slave machine:
    18/05/22 06:16:00 INFO Worker: Asked to launch executor app-20180522061600-0001/0 for PredictionIO Training: com.actionml.RecommendationEngine 18/05/22 06:16:00 INFO SecurityManager: Changing view acls to: ubuntu 18/05/22 06:16:00 INFO SecurityManager: Changing modify acls to: ubuntu 18/05/22 06:16:00 INFO SecurityManager: Changing view acls groups to: 18/05/22 06:16:00 INFO SecurityManager: Changing modify acls groups to: 18/05/22 06:16:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); groups with view permissions: Set(); users with modify permissions: Set(ubuntu); groups with modify permissions: Set() 18/05/22 06:16:00 INFO ExecutorRunner: Launch command: “/usr/lib/jvm/java-8-oracle/bin/java” “-cp” “./:/home/ubuntu/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6/conf/:/home/ubuntu/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6/jars/*” “-Xmx4096M” “-Dspark.driver.port=45049” “org.apache.spark.executor.CoarseGrainedExecutorBackend” “–driver-url” “spark://CoarseGrainedScheduler@172.31.5.119:45049” “–executor-id” “0” “–hostname” “172.31.6.235” “–cores” “8” “–app-id” “app-20180522061600-0001” “–worker-url” “spark://Worker@172.31.6.235:45057” 18/05/22 06:16:50 INFO Worker: Asked to kill executor app-20180522061600-0001/0 18/05/22 06:16:50 INFO ExecutorRunner: Runner thread for executor app-20180522061600-0001/0 interrupted 18/05/22 06:16:50 INFO ExecutorRunner: Killing process! 18/05/22 06:16:51 INFO Worker: Executor app-20180522061600-0001/0 finished with state KILLED exitStatus 143 18/05/22 06:16:51 INFO Worker: Cleaning up local directories for application app-20180522061600-0001 18/05/22 06:16:51 INFO ExternalShuffleBlockResolver: Application app-20180522061600-0001 removed, cleanupLocalDirs = true

    Thanks!

  • Aman Agarwal

    Hi,
    I would like to ask how to install spark to use it as an execution engine for hive. I already have hive installed in a multi-node cluster and now wants to use spark as execution engine instead of MR.

  • Aman Agarwal

    Hi,
    I want to use spark as hive’s execution engine. I have hive installed on a cluster of 1000 nodes and now want to install spark to use hive on spark, how to install spark in order to use as hive’s execution engine.

Comments are closed.