- This topic has 2 replies, 1 voice, and was last updated 5 years, 7 months ago by
.
-
Posts
-
-
-
In MapReduce job, Reducer takes intermediate key-value pairs generated by the Mapper as input i.e. output of mapper is the input to Reducer. Reducer runs reduce function on each of them and generate the output. Reducer output is the final output. Reducer does aggregation or summation sort of computation.
With the help of Job.setNumreduceTasks(int) the user set the number of reducers for the job. Hence the right number of reducers are set by the formula:
0.95 Or 1.75 multiplied by (<no. of nodes> * <no. of maximum container per node>)With 0.95, all the reducers can launch immediately. And start transferring map outputs as the map finish. With 1.75, faster node finishes the first round of reduces. Then launch the second wave of reduces.
With the increase of the number of reducers:
1) Load balancing increases.
2) Framework overhead increases.
3) Lowers the cost of failuresFollow the link to learn more about Reducer in Hadoop
-
The right number of Reducer seems to be 0.95 or 1.75 multiplied by (<no. of nodes> * <no. of maximum containers per node>).
With 0.95 all of the reduces can launch immediately and start transferring map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.
Increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.
The scaling factors above are slightly less than whole numbers to reserve a few reduce slots in the framework for speculative-tasks and failed tasks.
It is legal to set the number of reduce-tasks to zero if no reduction is desired
The default number of reducers for any job is 1. The number of reducers can be set in the job configuration.
Follow the link to learn more about Reducer in Hadoop
-
- You must be logged in to reply to this topic.
