{"id":2202,"date":"2017-04-24T08:52:48","date_gmt":"2017-04-24T08:52:48","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2202"},"modified":"2018-11-21T11:39:01","modified_gmt":"2018-11-21T06:09:01","slug":"learn-apache-spark-sparkcontext","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/learn-apache-spark-sparkcontext\/","title":{"rendered":"Learn SparkContext – Introduction and Functions"},"content":{"rendered":"

1. Objective<\/h2>\n

SparkContext<\/strong>\u00a0is the entry gate of\u00a0Apache Spark<\/a><\/strong>\u00a0functionality. The most important step of any Spark driver application is to generate SparkContext. It allows your Spark Application to access Spark Cluster with the help of Resource Manager (YARN\/Mesos<\/strong>). To create SparkContext, first\u00a0SparkConf<\/strong>\u00a0should be made. The SparkConf has a configuration parameter that our\u00a0Spark\u00a0driver application will pass to SparkContext.
\nIn this Apache Spark tutorial, we will deeply understand what is SparkContext in Spark. How to create SparkContext Class in Spark with the help of Spark-Scala word count program. We will also learn various tasks of SparkContext and how to stop SparkContext in Apache Spark.<\/p>\n

So, let’s start SparkContext tutorial.<\/p>\n

\"Learn<\/a>

Learn SparkContext – Introduction and Functions<\/p><\/div>\n

Learn\u00a0how to install Apache Spark in standalone mode<\/a>\u00a0and\u00a0Apache Spark installation in a\u00a0<\/a>multi-node cluster.<\/a><\/strong><\/p>\n

2. What is SparkContext in Apache Spark?<\/h2>\n

SparkContext is the entry point of Spark functionality. The most important step of any Spark driver application is to generate SparkContext. It allows your Spark Application to access Spark Cluster with the help of Resource Manager. The resource manager can be one of these three-\u00a0Spark Standalone<\/a><\/strong>, \u00a0YARN<\/strong><\/a>, Apache Mesos<\/strong>.<\/a><\/p>\n

3. How to Create SparkContext Class?<\/h2>\n

If you want to create SparkContext, first\u00a0SparkConf<\/strong>\u00a0should be made. The SparkConf has a configuration parameter that our Spark driver application will pass to SparkContext. Some of these parameter defines properties of Spark driver application. While some are used by Spark to allocate resources on the cluster, like the number, memory size, and cores used by executor running on the worker nodes.
\nIn short, it guides how to access the Spark cluster. After the creation of a SparkContext object, we can invoke functions such as\u00a0textFile, sequenceFile, parallelize<\/strong>\u00a0etc. The different contexts in which it can run are local, yarn-client, Mesos URL and Spark URL.
\nOnce the SparkContext is created, it can be used to\u00a0create RDDs<\/a><\/strong>, broadcast variable, and accumulator, ingress Spark service and run jobs. All these things can be carried out until SparkContext is stopped.<\/p>\n

4. Stopping SparkContext<\/h2>\n

Only one SparkContext may be active per JVM. You must stop the active it before creating a new one as below:
\nstop(): Unit
\nIt will display the following message:
\nINFO SparkContext: Successfully stopped SparkContext<\/em><\/strong><\/p>\n

5. Spark Scala Word Count Example<\/h2>\n

Let’s see how to create SparkContext using SparkConf with the help of Spark-Scala word count example-
\n[php]
\npackage com.dataflair.spark
\nimport org.apache.spark.SparkContext
\nimport org.apache.spark.SparkConf
\nobject Wordcount {
\ndef main(args: Array[String]) {
\n\/\/Create conf object
\nval conf = new SparkConf()
\n.setAppName(“WordCount”)
\n\/\/create spark context object
\nval sc = new SparkContext(conf)
\n\/\/Check whether sufficient params are supplied
\nif (args.length < 2) {
\nprintln(“Usage: ScalaWordCount <input> <output>”)
\nSystem.exit(1)
\n}
\n\/\/Read file and create RDD
\nval rawData = sc.textFile(args(0))
\n\/\/convert the lines into words using flatMap operation
\nval words = rawData.flatMap(line => line.split(” “))
\n\/\/count the individual words using map and reduceByKey operation
\nval wordCount = words.map(word => (word, 1)).reduceByKey(_ + _)
\n\/\/Save the result
\nwordCount.saveAsTextFile(args(1))
\n\/\/stop the spark context
\nsc.stop
\n}
\n}[\/php]<\/p>\n

6. Functions of SparkContext in Apache Spark<\/h2>\n
\"10<\/a>

10 Important Functions of SparkContext in Apache Spark<\/p><\/div>\n

i. To get the current status of Spark Application<\/h3>\n