Create Spark project in Scala with Eclipse without Maven 13


1. Objective

This step by step tutorial will explain how to create a Spark project in Scala with Eclipse without Maven and how to submit the application after the creation of jar. This Guide also briefs about the installation of Scala plugin in eclipse and setup spark environment in eclipse. Learn how to configure development environment for developing Spark applications in Scala in this tutorial.

If you are completely new to Apache Spark, I recommend you to read this Apache Spark Introduction Guide.

Develop your first Spark Project in Scala with Eclipse without Maven

2. Steps to create Spark project in Scala

To create Spark Project in Scala with Eclipse without Maven follow the steps given below-

2.1. Platform Used / Required

  • Operating System: Windows / Linux / Mac
  • Java: Oracle Java 7
  • Scala: 2.11
  • Eclipse: Eclipse Luna, Mars or later

2.2. Install Eclipse plugin for Scala

Open Eclipse Marketplace (Help >> Eclipse Marketplace) and search for “scala ide”. Now install the Scala IDE. Alternatively, you can download Eclipse for Scala.

 Install Eclipse plugin for Scala

2.3. Create a New Spark Scala Project

To create a new Spark Scala project, click on File >> New >> Other

Create a New Spark Scala Project

Select Scala Project:

Select Scala Project

Supply Project Name:

Supply Project Name

2.4. Create New Package

After creating the project, now create a new package.

 Create New Package

Supply Package Name:

Supply Package Name

2.5. Create a New Scala Object

Now create a new Scala Object to develop Scala program for Spark application

Create a new Scala Object to develop Scala program for Spark application

Select Scala Object:

Select Scala Object

Supply Object Name:

Supply Object Name:

2.6. New Scala Object in Editor

Scala object is ready now we can develop our Spark wordcount code in Scala-

New Scala Object in Editor to create Spark Application

2.7. Copy below Spark Scala Wordcount Code in Editor

package com.dataflair.spark

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object Wordcount {
 def main(args: Array[String]) {
 
 //Create conf object
 val conf = new SparkConf()
 .setAppName("WordCount")
 
 //create spark context object
 val sc = new SparkContext(conf)

//Check whether sufficient params are supplied
 if (args.length < 2) {
 println("Usage: ScalaWordCount <input> <output>")
 System.exit(1)
 }
 //Read file and create RDD
 val rawData = sc.textFile(args(0))
 
 //convert the lines into words using flatMap operation
 val words = rawData.flatMap(line => line.split(" "))
 
 //count the individual words using map and reduceByKey operation
 val wordCount = words.map(word => (word, 1)).reduceByKey(_ + _)
 
 //Save the result
 wordCount.saveAsTextFile(args(1))

//stop the spark context
 sc.stop
 }
}

 

Spark Scala WordCount Code in Editor

You will see lots of error due to missing libraries.

2.8. Add Spark Libraries

Configure Spark environment in Eclipse: Right click on project name >> build path >> Configure Build Path

Configure Spark environment in Eclipse

Add the External Jars:

Add the External Jars

2.9. Select Spark Jars and insert

You should have spark setup available in developing environment, it will be needed for spark libraries.

 Select Spark Jars and insert

Go to “Spark-Home >> jars” and select all the jars:

select all the jars

Import the selected jar:

Import the selected jar

2.10. Spark Scala Word Count Program

After importing the libraries all the errors will be removed.

Spark WordCount Program in Scala

We have successfully created Spark environment in Eclipse and developed Spark Scala program. Now let’s deploy the Spark job on Linux, before deploying/running the application you must have Spark Installed.

Follow this links to install Apache Spark on single node cluster or on the multi-node cluster.

2.11. Create the Spark Scala Program Jar File

Before running created Spark word count application we have to create a jar file. Right click on project >> export

Create the Spark Scala Program Jar File

Select Jar-file Option to Export:

Select Jar-file Option to Export

Create the Jar file:

Create the Jar file

The jar file for the Spark Scala application has been created, now we need to run it.

2.12. Go to Spark Home Directory

Login to Linux and open terminal. To run Spark Scala application we will be using Ubuntu Linux. Copy the jar file to Ubuntu and create one text file, which we will use as input for Spark Scala wordcount job.

Copy the jar file to Ubuntu and create one text file, to use as input for spark Scala wordcount job.

2.13. Submit Spark Application using spark-submit script

To submit the Spark application using below command:

 bin/spark-submit --class <Qualified-Class-Name> --master <Master> <Path-Of-Jar-File> <Input-Path> <Output-Path>
bin/spark-submit --class com.dataflair.spark.Wordcount --master local ../sparkJob.jar ../wc-data output

Let’s understand above command:

  • bin/spark-submit: To submit Spark Application
  • –class: To specify the class name to execute
  • –master: Master (local / <Spark-URI> / yarn)
  • <Jar-Path>: The jar file of application
  • <Input-Path>: Location from where input data will be read
  • <Output-Path>: Location where Spark application will write output

Submit Spark Application using spark-submit script

Submit Spark Application using spark-submit script

The application has been completed successfully, now browse the result.

2.14. Browse the result

Browse the output directory and open the file with name part-xxxxx which contains the output of the application.

Browse the output directory and open the file which contains the output file

We have successfully created Spark project in Scala and deployed on Ubuntu.

To play with Spark First learn RDD, DataFrame, DataSet in Apache Spark and then refer this Spark shell commands tutorial to practically implements Spark functionalities.

See Also-


Leave a comment

Your email address will not be published. Required fields are marked *

13 thoughts on “Create Spark project in Scala with Eclipse without Maven