Live instructor-led & Self-paced Online Certification Training Courses (Big Data, Hadoop, Spark) › Forums › Hadoop › In MapReduce Flow, when Combiner is called ?
September 20, 2018 at 3:46 pm #5614
During MapReduce flow, when is combiner called ? is it called after mapper or after partitioner?
What is the exact order of execution of combiner in MapReduce flow?September 20, 2018 at 3:46 pm #5615
If we have a look at the high level flow of Map Reduce, this should be the order of execution :
The combiner runs after mapper and before partitioner.
The job of the combiner is to optimize the output of the mapper before its fed to the reducer in order to reduce the data size that is moved to the reducer.
It cannot run after partitioner as the partitioner after processing the data sends it to the reducer. Also, the output of the partitioner is key and collection of values which is not meant to be given as input to the combiner.
Follow the link to learn in deep about Data Flow in MapReduceSeptember 20, 2018 at 3:47 pm #5617
Combiner can be viewed as mini-reducers in map phase.
In Map Reduce framework, usually the output from the map tasks is large and data transfer between map and reduce tasks will be high. Since the data transfer across the network is expensive and to limit the volume of data transfer between map and reduce tasks.
Combiner functions summarize the map output records with the same key and output of combiner will be sent over network to actual reduce task as input.
The order of execution in map-reduce is,
Combiner is applied at the same machine as the map.It is called after mapper and before partitioner.
Check this for more info about combiner :Combiner in HadoopSeptember 20, 2018 at 3:47 pm #5619
Combiner is called after mapper.
Details: Combiner can be viewed as mini-reducers in the map phase. They perform a local-reduce on the mapper results before they are distributed further. Once the Combiner functionality is executed, it is then passed on to the Reducer for further work.
The partitioner comes into the picture when we are working one more than on reducer. So, the partitioner decides which reducer is responsible for a particular key. They basically take the Mapper Result(if Combiner is used then Combiner Result) and send it to the responsible Reducer based on the key.
You must be logged in to reply to this topic.