- This topic has 1 reply, 1 voice, and was last updated 7 years, 10 months ago by
.
-
Posts
-
-
I am trying to create a wordcount program in Spark. I have installed a 3 node Spark standalone cluster. I have created a program called sparkword count in Eclipse. This program reads from a file on HDFS and counts number of times each word has repeated. When i try to execute the program, i am getting class not found exception. So, i removed the entire code, and when i execute the program with just the below logic, i am able to see the output.
val conf = new SparkConf().setMaster(spark://BD-HMF24:7077).setJobName(“test”)
val sc = new SparkContext(conf)
val a = sc.textFile(“hdfs://BD-HMF24:9000/a”)
a.collect().foreach(println)Apart from the above logic, i am getting class not found exception even if i include a map statement on RDD a.
Please help me identify the issue. Also, please help me know how to troubleshoot these type of issues.
Regards,
Srilakshmi -
It seams your program has compile time errors hence the class is not compiled while creation of jar and your getting ClassNotFound Exception. check for error in your program and resubmit it.
Regarding troubleshoot / debug: you can configure spark environment in eclipse and run programs directly in eclipse, Add following statement in your program:
Specify Hadoop home Dir:
System.setProperty(“hadoop.home.dir”, “E:\\setups\\hadoop-2.5.0-cdh5.3.2\\hadoop-2.5.0-cdh5.3.2\\”)
second argument is the path of Hadoop directory on local filesystemSet Master to local:
val conf = new SparkConf().setAppName(“WordCount”).setMaster(“local”)
val sc = new SparkContext(conf)While creation of RDD specify path of localFS:
val data = sc.textFile(“E:\\test\\Posts1.xml”)Print the final result:
result.foreach { println }
-
- You must be logged in to reply to this topic.
