In which case one cannot implement a Combiner?
-
-
Combiner is used as an optimization technique. But what are the prerequisites to implement combiner ?
In which case one cannot implement a Combiner?
-
The combiner is as semi-reducer in MapReduce. It is an optional class which can be specified in MapReduce driver class to process the output of map tasks before submitting it to reducer tasks.
The main purpose of combine class is, usually the output from map task is huge and data transfer from map to reduce will be high. Since data transfer is expensive and to limit the volume of data transfer between map and reduce tasks Combiner
function summarizes the map output records with the same Key. The output of combiner will be sent over to reducer as input.
- Combiner does not have its own interface, it implements Reduce Interface and reduce () method.
- There is no guarantee on how many times a combiner function will be called for each map output key. At times it may not be executed at all.
Follow the link to learnj more about Combiner in Hadoop
- You must be logged in to reply to this topic.