{"id":2108,"date":"2017-04-19T14:31:54","date_gmt":"2017-04-19T14:31:54","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2108"},"modified":"2018-11-21T11:22:19","modified_gmt":"2018-11-21T05:52:19","slug":"hadoop-reducer","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hadoop-reducer\/","title":{"rendered":"Hadoop Reducer – 3 Steps learning for MapReduce Reducer"},"content":{"rendered":"

1. Hadoop Reducer Tutorial – Objective<\/h2>\n

In Hadoop<\/strong><\/a>, Reducer<\/strong> takes the output of the Mapper<\/strong><\/a> (intermediate key-value pair) process each of them to generate the output. The output of the reducer is the final output, which is stored in HDFS. Usually, in the Hadoop Reducer, we do aggregation or summation sort of computation.<\/p>\n

In this Hadoop Reducer tutorial, we will answer what is Reducer in Hadoop MapReduce, what are the different phases of Hadoop MapReduce<\/strong><\/a> Reducer, shuffling and sorting in Hadoop, Hadoop reduce phase, functioning of Hadoop reducer class. We will also discuss how many reducers are required in Hadoop and how to change the number of reducers in Hadoop MapReduce.<\/p>\n

\"Hadoop<\/a>

Hadoop Reducer – 3 Steps learning for MapReduce Reducer<\/p><\/div>\n

2. What is Hadoop Reducer?<\/h2>\n

Let’s now discuss what is Reducer in MapReduce first.<\/p>\n

The Reducer\u00a0process the output of the mapper. After processing the data, it produces a new set of output. At last\u00a0HDFS <\/strong><\/a>stores this output data.<\/p>\n

Hadoop Reducer takes a set of an intermediate key-value pair<\/strong><\/a> produced by the mapper as the input and runs a Reducer function on each of them. One can aggregate, filter, and combine this data (key, value) in a number of ways for a wide range of processing. Reducer first processes the intermediate values for particular key generated by the map function and then generates the output (zero or more key-value pair).<\/p>\n

One-one mapping takes place between keys and reducers. Reducers run in parallel since they are independent of one another. The user decides the number of reducers. By default number of reducers is 1.<\/p>\n

Read: HDFS Combiner Tutorial <\/a><\/strong><\/p>\n

3. Phases of MapReduce Reducer<\/h2>\n

As you can see in the diagram at the top, there are 3 phases of\u00a0Reducer\u00a0in Hadoop MapReduce. Let’s discuss each of them one by one-<\/p>\n

3.1. Shuffle Phase of MapReduce Reducer<\/h3>\n

In this phase, the sorted output from the mapper is the input to the Reducer. In Shuffle phase, with the help of HTTP, the framework fetches the relevant partition of the output of all the mappers.<\/p>\n

3.2. Sort Phase of MapReduce Reducer<\/h3>\n

In this phase, the input from different mappers is again sorted based on the similar keys in different Mappers. The shuffle and sort phases occur concurrently.<\/p>\n

Learn Mapreduce Shuffling and Sorting Phase<\/strong><\/a> in detail.<\/p>\n

Read: Features of HDFS<\/a><\/strong><\/p>\n

3.3. Reduce Phase<\/h3>\n

In this phase, after shuffling and sorting, reduce task aggregates the key-value pairs. The\u00a0OutputCollector.collect()<\/em>\u00a0method, writes the output of the reduce task to the Filesystem. Reducer output is not sorted.<\/p>\n

4. MapReduce Number of Reducers<\/h2>\n

In this section of Hadoop Reducer, we will discuss how many number of Mapreduce reducers are required in MapReduce and how to change the Hadoop reducer number in MapReduce?
\nWith the help of Job.setNumreduceTasks(int)<\/em> the user set the number of reducers for the job. The right number of reducers are 0.95 or 1.75 multiplied by (<no. of nodes> * <no. of the maximum container per node>).<\/p>\n

With 0.95, all reducers immediately launch and start transferring map outputs as the maps finish. With 1.75, the first round of reducers is finished by the faster nodes and second wave of reducers is launched doing a much better job of load balancing.<\/p>\n

Increasing the number of MapReduce reducers:<\/strong><\/p>\n