{"id":1309,"date":"2016-12-23T16:10:20","date_gmt":"2016-12-23T16:10:20","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=1309"},"modified":"2018-11-20T15:11:58","modified_gmt":"2018-11-20T09:41:58","slug":"apache-spark-map-vs-flatmap","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-spark-map-vs-flatmap\/","title":{"rendered":"Apache Spark Map vs FlatMap Operation"},"content":{"rendered":"

1. Objective<\/h2>\n

In this Apache Spark<\/a><\/strong> tutorial, we will discuss the comparison between Spark Map vs FlatMap Operation. Map and FlatMap are\u00a0the transformation operations in Spark<\/a><\/strong>.\u00a0<\/strong>Map()<\/strong> operation applies to each element of RDD<\/strong><\/a> and it returns the result as new RDD. In the Map, operation developer can define his own custom business logic. While FlatMap()<\/strong> is similar to Map, but FlatMap allows returning 0, 1 or more elements from map function.<\/p>\n

In this blog, we will discuss how to perform map operation on RDD and how to process data using FlatMap operation. This tutorial also covers what is map operation, what is a flatMap operation, the difference between map() and flatMap() transformation in Apache Spark with examples. We will also see Spark map and flatMap example in Scala<\/strong> and Java<\/strong> in this Spark tutorial.<\/p>\n

So, let’s start Spark Map vs FlatMap function.<\/p>\n

\"Apache<\/a>

Apache Spark Map vs FlatMap Operation<\/p><\/div>\n

\u00a0Do you know How to install and configure Apache Spark<\/strong>?<\/a><\/p>\n

2. Difference between Spark Map vs FlatMap Operation<\/h2>\n

This section of the Spark tutorial provides the details of Map vs FlatMap operation in Apache Spark with examples in\u00a0Scala<\/a><\/strong> and Java<\/strong> programming languages.<\/p>\n

i. Spark Map Transformation<\/h3>\n

A map<\/strong> is a transformation operation in Apache Spark.<\/strong>\u00a0It\u00a0applies to each element of RDD<\/strong> and it returns the result as new RDD. In the Map, operation developer can define his own custom business logic. The same logic will be applied to all the elements of RDD.<\/p>\n

Spark Map function takes one element as input process it according to custom code (specified by the developer) and returns one element at a time.\u00a0Map transforms an RDD<\/strong> of length N into another RDD of length N. The input and output RDDs will typically have the same number of records.<\/p>\n

\"apache<\/a>

Apache Spark map transformation operation<\/p><\/div>\n

a. Map Transformation Scala Example<\/strong><\/h4>\n

Create RDD<\/strong><\/p>\n

val data = spark.read.textFile(\"INPUT-PATH\").rdd<\/pre>\n
Above statement will create an RDD with name data. Follow this guide to learn more ways to create RDDs<\/a> <\/strong>in Apache Spark.<\/strong><\/p>\n
Map Transformation-1<\/strong><\/p>\n
val newData = data.map (line => line.toUpperCase() )<\/pre>\n
Above the map, a transformation will convert each and every record of RDD to upper case.<\/p>\n
Map Transformation-2<\/strong><\/p>\n
val tag = data.map {line => {\r\nval xml = XML.loadString(line)\r\nxml.attribute(\"Tags\").get.toString()\r\n}\r\n}<\/pre>\n
Above the map, a transformation will parse XML and collect Tag attribute from the XML data. Overall the map operation is converting XML into a structured format.<\/p>\n
Follow this link to know about Java Programming Language<\/a><\/strong><\/p>\n
b. Map Transformation Java Example<\/h4>\n
Create RDD<\/strong><\/p>\n
JavaRDD<String> linesRDD = spark.read().textFile(\"INPUT-PATH\").javaRDD();<\/pre>\n
Above statement will create an RDD with name\u00a0lines RDD.<\/p>\n
Map Transformation<\/strong><\/p>\n
JavaRDD<String> newData = linesRDD.map(new Function<String, String>() {\r\npublic String call(String s) {\r\nString result = s.trim().toUpperCase();\r\nreturn result;\r\n}\r\n});<\/pre>\n
We recommend you to read –\u00a0Spark Shell Commands to Interact with Spark-Scala<\/a><\/strong><\/p>\n
ii. Spark FlatMap Transformation Operation<\/h3>\n
Let’s now discuss flatMap() operation in Apache Spark-<\/p>\n
\"apache<\/a>
Apache Spark flatMap transformation operation<\/p><\/div>\n
A flatMap<\/strong> is a transformation operation. It applies to each element of RDD and it returns the result as new RDD. It is similar to Map, but FlatMap allows returning 0, 1 or more elements\u00a0from map function. In the FlatMap operation, a developer can define his own custom business logic. The same logic will be applied to all the elements of the RDD.<\/p>\n
A FlatMap function takes one element as input process it according to custom code (specified by the developer) and returns 0\u00a0or more element at a time. flatMap() transforms an RDD of length N into another RDD of length M.<\/p>\n
a.\u00a0FlatMap Transformation Scala Example<\/strong><\/h4>\n
val result = data.flatMap (line => line.split(\" \") )<\/pre>\n
Above flatMap transformation will convert\u00a0a line into words. One word will be an individual\u00a0element of the newly created RDD.<\/p>\n
Learn to Create Spark project in Scala with Eclipse<\/a><\/strong><\/p>\n
b. FlatMap Transformation Java Example<\/strong><\/h4>\n
JavaRDD<String> result = data.flatMap(new FlatMapFunction<String, String>() {\r\npublic Iterator<String> call(String s) {\r\nreturn Arrays.asList(s.split(\" \")).iterator();\r\n} });<\/pre>\n
Above flatMap transformation will convert\u00a0a line into words. One word will be an individual\u00a0element of the newly created RDD.<\/p>\n
3. Conclusion<\/h2>\n
Hence, from the comparison between Spark map() vs flatMap(), it is clear that Spark map function expresses a one-to-one transformation. It transforms each element of a collection into one element of the resulting collection. While Spark flatMap function expresses a one-to-many transformation. It transforms each element to 0 or more elements.<\/p>\n
Please leave a comment if you like this post or have any query about Apache Spark map vs flatMap function.<\/p>\n
See Also-<\/strong><\/p>\n