{"id":2128,"date":"2017-04-21T07:04:55","date_gmt":"2017-04-21T07:04:55","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2128"},"modified":"2018-11-14T16:19:54","modified_gmt":"2018-11-14T10:49:54","slug":"hadoop-combiner-tutorial","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hadoop-combiner-tutorial\/","title":{"rendered":"Hadoop Combiner – Best Explanation to MapReduce Combiner"},"content":{"rendered":"

1. Hadoop Combiner \/ MapReduce Combiner<\/h2>\n

Hadoop<\/a> Combiner<\/strong> is also known as \u201cMini-Reducer<\/strong>\u201d that summarizes the Mapper output record with the same Key before passing to the Reducer. In this tutorial on MapReduce <\/strong><\/a>combiner we are going to answer what is a Hadoop combiner, MapReduce program with and without combiner, advantages of Hadoop combiner and disadvantages of the combiner in Hadoop.<\/p>\n

\"Hadoop<\/a>

Hadoop Combiner – Best Explanation to MapReduce Combiner<\/p><\/div>\n

2. What is Hadoop Combiner?<\/h2>\n

On a large dataset when we run MapReduce job<\/strong><\/a>, large chunks of intermediate data is generated by the Mapper and this intermediate data is passed on the Reducer for further processing, which leads to enormous network congestion. MapReduce framework provides a function known as Hadoop Combiner <\/strong>that plays a key role in reducing network congestion.<\/p>\n

We have already seen earlier what is mapper<\/a> <\/strong>and what is reducer<\/strong> <\/a>in Hadoop MapReduce. Now we in the next step to learn Hadoop MapReduce Combiner.<\/p>\n

The combiner in MapReduce is also known as \u2018Mini-reducer\u2019. The primary job of Combiner is to process the output data from the Mapper, before passing it to Reducer. It runs after the mapper and before the Reducer and its use is optional.<\/p>\n

Read: Key-value Pairs in MapReduce<\/a><\/strong><\/p>\n

3. How does MapReduce Combiner work?<\/h2>\n

Let us now see the working of the Hadoop combiner in MapReduce and how things change when combiner is used as compared to when combiner is not used in MapReduce?<\/p>\n

3.1. MapReduce program without Combiner\u00a0<\/strong><\/h3>\n
\"MapReduce<\/a>

MapReduce Combiner : MapReduce program without combiner<\/p><\/div>\n

In the above diagram, no combiner is used. Input is split into two mappers and 9 keys are generated from the mappers. Now we have (9 key\/value<\/strong><\/a>) intermediate data, the further mapper will send directly this data to reducer and while sending data to the reducer, it consumes some network bandwidth (bandwidth means time taken to transfer data between 2 machines). It will take more time to transfer data to reducer if the size of data is big.<\/p>\n

Now in between mapper and reducer if we use a\u00a0 hadoop combiner, then combiner shuffles intermediate data (9 key\/value) before sending it to the reducer and generates 4 key\/value pair as an output.
\nRead: Data Locality in MapReduce<\/a><\/strong><\/p>\n

3.2. MapReduce program with Combiner in between Mapper and Reducer<\/h3>\n
\"MapReduce<\/a>

MapReduce Combiner: MapReduce program with combiner<\/p><\/div>\n

Reducer now needs to process only 4 key\/value pair data which is generated from 2 combiners. Thus reducer gets executed only 4 times to produce final output, which increases the overall performance.<\/p>\n

4. Advantages of MapReduce Combiner<\/h2>\n

As we have discussed what is Hadoop MapReduce Combiner in detail, now we will discuss some advantages of Mapreduce Combiner.<\/p>\n