{"id":2244,"date":"2017-04-25T09:29:08","date_gmt":"2017-04-25T09:29:08","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2244"},"modified":"2018-11-21T11:29:57","modified_gmt":"2018-11-21T05:59:57","slug":"hadoop-partitioner-tutorial","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hadoop-partitioner-tutorial\/","title":{"rendered":"Hadoop Partitioner – Internals of MapReduce Partitioner"},"content":{"rendered":"

1. Hadoop Partitioner \/ MapReduce Partitioner<\/h2>\n

In this MapReduce Tutorial<\/strong><\/a>, our objective is to discuss what is Hadoop<\/strong><\/a> Partitioner. The Partitioner<\/strong>\u00a0in MapReduce controls the partitioning of the key of the intermediate mapper output. By hash function, key (or a subset of the key) is used to derive the partition. A total number of partitions depends on the number of reduce task. Here\u00a0we will also learn what is the need of Hadoop partitioner, what is the default Hadoop partitioner, how many practitioners are required in Hadoop and what do you mean by poor partitioning in Hadoop along with ways to overcome MapReduce poor partitioning.<\/p>\n

\"Hadoop<\/a>

Hadoop Partitioner – Internals of MapReduce Partitioner<\/p><\/div>\n

2. What is Hadoop Partitioner?<\/h2>\n

Before we start with MapReduce partitioner, let us understand what is Hadoop mapper<\/a><\/strong>, Hadoop Reducer<\/a><\/strong>, and combiner\u00a0in Hadoop<\/a><\/strong>?<\/p>\n

Partitioning<\/strong> of the keys of the intermediate map output is controlled by the Partitioner. By hash function, key (or a subset of the key) is used to derive the partition. According to the key-value<\/strong><\/a> each mapper output is partitioned and records having the same key value go into the same partition (within each mapper), and then each partition is sent to a reducer. Partition class determines which partition a given (key, value) pair will go. Partition phase takes place after map phase and before reduce phase. Lets move ahead with need of Hadoop Partitioner and if you face any difficulty anywhere in Hadoop MapReduce tutorial, you can ask us in comments.<\/p>\n

Read: A Guide on Big Data Hadoop for beginners<\/a><\/strong><\/p>\n

3. Need of Hadoop MapReduce Partitioner?<\/h2>\n

Let’s now discuss what is the need of Mapreduce Partitioner in Hadoop?<\/p>\n

MapReduce job<\/a><\/strong> takes an input data set and produces the list of the key-value pair which is the result of map phase in which input data is split and each task processes the split and each map, output the list of key-value pairs. Then, the output from the map phase is sent to reduce task which processes the user-defined reduce function on map outputs. But before reduce phase, partitioning of the map output take place on the basis of the key and sorted.
\nThis partitioning specifies that all the values for each key are grouped together and make sure that all the values of a single key go to the same reducer, thus allows even distribution of the map output over the reducer.<\/p>\n

Partitioner in Hadoop MapReduce redirects the mapper output to the reducer by determining which reducer is responsible for the particular key.
\nRead: Hadoop MapReduce tutorial<\/a><\/strong><\/p>\n

4. Default MapReduce Partitioner<\/h2>\n

The Default Hadoop partitioner in Hadoop MapReduce is Hash Partitioner which computes a hash value for the key and assigns the partition based on this result.<\/p>\n

5. How many Partitioners are there in Hadoop?<\/h2>\n

The total number of Partitioners that run in Hadoop is equal to the number of reducers i.e. Partitioner will divide the data according to the number of reducers which is set by JobConf.setNumReduceTasks()<\/em> method. Thus, the data from single partitioner is processed by a single reducer. And partitioner is created only when there are multiple reducers.<\/p>\n

6. Poor Partitioning in Hadoop MapReduce<\/h2>\n

If in data input one key appears more than any other key. In such case, we use two mechanisms to send data to partitions.<\/p>\n