Create Spark project in Scala with Eclipse without Maven 33


1. Objective

This step by step tutorial will explain how to create a Spark project in Scala with Eclipse without Maven and how to submit the application after the creation of jar. This Guide also briefs about the installation of Scala plugin in eclipse and setup spark environment in eclipse. Learn how to configure development environment for developing Spark applications in Scala in this tutorial.

If you are completely new to Apache Spark, I recommend you to read this Apache Spark Introduction Guide.

Develop your first Spark Project in Scala with Eclipse without Maven

2. Steps to create Spark project in Scala

To create Spark Project in Scala with Eclipse without Maven follow the steps given below-

2.1. Platform Used / Required

  • Operating System: Windows / Linux / Mac
  • Java: Oracle Java 7
  • Scala: 2.11
  • Eclipse: Eclipse Luna, Mars or later

2.2. Install Eclipse plugin for Scala

Open Eclipse Marketplace (Help >> Eclipse Marketplace) and search for “scala ide”. Now install the Scala IDE. Alternatively, you can download Eclipse for Scala.

 Install Eclipse plugin for Scala

2.3. Create a New Spark Scala Project

To create a new Spark Scala project, click on File >> New >> Other

Create a New Spark Scala Project

Select Scala Project:

Select Scala Project

Supply Project Name:

Supply Project Name

2.4. Create New Package

After creating the project, now create a new package.

 Create New Package

Supply Package Name:

Supply Package Name

2.5. Create a New Scala Object

Now create a new Scala Object to develop Scala program for Spark application

Create a new Scala Object to develop Scala program for Spark application

Select Scala Object:

Select Scala Object

Supply Object Name:

Supply Object Name:

2.6. New Scala Object in Editor

Scala object is ready now we can develop our Spark wordcount code in Scala-

New Scala Object in Editor to create Spark Application

2.7. Copy below Spark Scala Wordcount Code in Editor

package com.dataflair.spark

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object Wordcount {
 def main(args: Array[String]) {
 
 //Create conf object
 val conf = new SparkConf()
 .setAppName("WordCount")
 
 //create spark context object
 val sc = new SparkContext(conf)

//Check whether sufficient params are supplied
 if (args.length < 2) {
 println("Usage: ScalaWordCount <input> <output>")
 System.exit(1)
 }
 //Read file and create RDD
 val rawData = sc.textFile(args(0))
 
 //convert the lines into words using flatMap operation
 val words = rawData.flatMap(line => line.split(" "))
 
 //count the individual words using map and reduceByKey operation
 val wordCount = words.map(word => (word, 1)).reduceByKey(_ + _)
 
 //Save the result
 wordCount.saveAsTextFile(args(1))

//stop the spark context
 sc.stop
 }
}

 

Spark Scala WordCount Code in Editor

You will see lots of error due to missing libraries.

2.8. Add Spark Libraries

Configure Spark environment in Eclipse: Right click on project name >> build path >> Configure Build Path

Configure Spark environment in Eclipse

Add the External Jars:

Add the External Jars

2.9. Select Spark Jars and insert

You should have spark setup available in developing environment, it will be needed for spark libraries.

 Select Spark Jars and insert

Go to “Spark-Home >> jars” and select all the jars:

select all the jars

Import the selected jar:

Import the selected jar

2.10. Spark Scala Word Count Program

After importing the libraries all the errors will be removed.

Spark WordCount Program in Scala

We have successfully created Spark environment in Eclipse and developed Spark Scala program. Now let’s deploy the Spark job on Linux, before deploying/running the application you must have Spark Installed.

Follow this links to install Apache Spark on single node cluster or on the multi-node cluster.

2.11. Create the Spark Scala Program Jar File

Before running created Spark word count application we have to create a jar file. Right click on project >> export

Create the Spark Scala Program Jar File

Select Jar-file Option to Export:

Select Jar-file Option to Export

Create the Jar file:

Create the Jar file

The jar file for the Spark Scala application has been created, now we need to run it.

2.12. Go to Spark Home Directory

Login to Linux and open terminal. To run Spark Scala application we will be using Ubuntu Linux. Copy the jar file to Ubuntu and create one text file, which we will use as input for Spark Scala wordcount job.

Copy the jar file to Ubuntu and create one text file, to use as input for spark Scala wordcount job.

2.13. Submit Spark Application using spark-submit script

To submit the Spark application using below command:

bin/spark-submit --class <Qualified-Class-Name> --master <Master> <Path-Of-Jar-File> <Input-Path> <Output-Path>

bin/spark-submit --class com.dataflair.spark.Wordcount --master local ../sparkJob.jar ../wc-data output

Let’s understand above command:

  • bin/spark-submit: To submit Spark Application
  • –class: To specify the class name to execute
  • –master: Master (local / <Spark-URI> / yarn)
  • <Jar-Path>: The jar file of application
  • <Input-Path>: Location from where input data will be read
  • <Output-Path>: Location where Spark application will write output

Submit Spark Application using spark-submit script

Submit Spark Application using spark-submit script

The application has been completed successfully, now browse the result.

2.14. Browse the result

Browse the output directory and open the file with name part-xxxxx which contains the output of the application.

Browse the output directory and open the file which contains the output file

We have successfully created Spark project in Scala and deployed on Ubuntu.

To play with Spark First learn RDD, DataFrame, DataSet in Apache Spark and then refer this Spark shell commands tutorial to practically implements Spark functionalities.

See Also-


33 thoughts on “Create Spark project in Scala with Eclipse without Maven

  • Nirmalya Sengupta

    What I like about this article is that it has not missed a step (in my eyes, at least). If hand-holding the hesitant starters in the world of Apache Spark, has been the objective, then you have achieved it. Good job!

  • Becky

    I’m gonna say to my little brother, that he should also visit this web site on regular basis to obtain updates from hottest reports.

    • Data Flair Post author

      Hii Becky,
      Thanks for visiting Data Flair. We are happy to hit the mark for you and your brother. It seems that you liked and understood how to create the Spark Project in Scala. You should share this Spark Knowledge but not only with your brother, with all who wants to explore the career in Spark. Our Site Data Flair is continuously working for the sake of our loyal readers.

  • udit

    I am not able to create jar. it says
    JAR creation failed. See details for additional information.
    Class files on classpath not found or not accessible for: ‘SparkApplication/src/com/spark/employee/Maxwages.scala’

    • DF HD Team

      Dear Udit,
      It seems there is compilation error in your program. Spark is not able to create class file when compilation error is there.

    • Data Flair Post author

      Glad to see your review on Spark Project in Scala. Our team is continuously working for readers like you. You must read more Spark blogs on our website and let us know if the content helps you.

  • Sree

    getting error while running with spark-submit

    exception in thread main java.lang.nosuchmethodexception

    • Data Flair Post author

      Dileep thanks a lot, for taking time to post the review on Spark Scala Project. Your words are valuable to us. Update yourself with our new Spark blogs. Keep learning, keep sharing.

    • Data Flair Post author

      Hii Dileep
      Thank you for sharing such a positive experience. Keep learning and keep visiting Data Flair.

    • Data Flair Post author

      Hii Mohit,
      Thank you for catching this Maven query for Spark Scala Project.
      Maven is quite a popular way, will post another article detailing the steps: how to create a Spark Scala project using Maven. Till then keep visiting Data Flair and keep learning.

  • Aliaa

    This Spark scala Project process is very helpful and very detailed.
    Thanks very much.

    • Data Flair Post author

      Aliaa, Thank you so much for taking the time to write this excellent review. We regularly post the simply written helpful articles on Spark. You can select the spark category for more on Spark. Good luck with the site.

  • venu

    clearly explained how to create a jar in eclipse, if possible pls explain in intellij as well

    • Data Flair Post author

      Hii Venu,
      Glad you understand our explained tutorial of creating Jar in Spark Scala Project. Soon, we will post another tutorial about the project creation in IntelliJ for a Spark & Scala project. Get notified with our new blogs, keep checking the site. You can also check our new blog, hope this helps to clear your Spark Concept
      https://data-flair.training/blogs/scala-spark-shell-commands/

  • VAIBHAV ADLAKHA

    Do the same commands apply when running spark-shell on Windows cmd? If not how do I run this jar file using Spark-submit on Windows cmd?

  • Purna Manepalli

    Excellent post.
    Just a small correction for creation of conf object.
    val conf = new SparkConf().setAppName(“WordCount”).setMaster(“local[*]”)

  • alka

    Hello sir
    Your this tutorial is very good but in last step we past jar file in spark home dir then use spark submit command for run the program. Here i am facing the problem. sir plz can you this make this tutorial on video. By video we can understand where we doing something wrong.

    Please sir help the student. You Blog is very good

  • alka

    hello sir
    Your tutorials are very good.This is also a good tutorial. When i was creating project in scala without using maven in the last step where we using jar file does not conf. So please sir can you convert this tutorial in video in this way we can more understand how to conf spark scala through eclipse to ubuntu
    Thanks if you can help us

    • Data Flair Post author

      Hii Alka, you commented on this Spark Scala Project, so we are grateful for your loyal feedback telling us that our current approach on Spark requires a video tutorial to explain the things to our readers. We will work on it so that you can get an easy understanding of Spark Scala. I can see that you are very curious to learn more about Spark, just follow our site for best Spark Tutorials.

  • Vinyas

    Used below command for execution:
    spark-submit –class WordCount –master local /vinyas/Jars/WordCount.jar

    But getting error:
    java.lang.ClassNotFoundException: WordCount
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    What might be the reason?

    • Data Flair Post author

      Hii Vinyas, check the below solutions your error on Spark Scala Project
      There could be the following 2 issues:
      Either you are mentioning the wrong class name, make sure to mention the fully qualified class name. Or there are compile-time issues in your program. Check this and still if you are getting any error do let us know.

  • Rajat Saha

    This one is great. Could you please explain how to run the above spark program in hadoop? The same .jar file made, not using spark-shell.

    • Data Flair Post author

      Rajat Saha, thanks a lot for such loyal comment and good words.
      You can run the Spark program on Hadoop, you need to mention the input and output path of HDFS URI and mention the master as yarn. If you are satisfied with this, leave a remark.

Comments are closed.