{"id":2343,"date":"2017-04-28T11:28:17","date_gmt":"2017-04-28T11:28:17","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2343"},"modified":"2018-11-21T11:33:18","modified_gmt":"2018-11-21T06:03:18","slug":"shuffling-and-sorting-in-hadoop","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/shuffling-and-sorting-in-hadoop\/","title":{"rendered":"Shuffling and Sorting in Hadoop MapReduce"},"content":{"rendered":"

1. Objective<\/h2>\n

In Hadoop<\/strong><\/a>, the process by which the intermediate output from mappers<\/strong> is transferred to the reducer<\/strong> is called Shuffling. Reducer gets 1 or more keys and associated values on the basis of reducers. Intermediated key-value<\/strong> generated by mapper is sorted automatically by key.\u00a0In this blog, we will discuss in detail about shuffling and Sorting in Hadoop<\/strong> MapReduce<\/strong>.<\/p>\n

Here we will learn what is sorting in Hadoop, what is shuffling in Hadoop, what is the purpose of Shuffling and sorting phase in MapReduce<\/a><\/strong>, how MapReduce shuffle works and how MapReduce sort works. We will also learn what is secondary sorting in MapReduce?<\/p>\n

To learn Hadoop Cloudera\u00a0CDH5 installation follow this installation guide.<\/a><\/p>\n

\"Shuffling<\/a>

Shuffling and Sorting in Hadoop MapReduce<\/p><\/div>\n

2. What is Shuffling and Sorting in Hadoop MapReduce?<\/h2>\n

Before we start with Shuffle and Sort in MapReduce, let us revise the other phases of MapReduce like Mapper<\/a><\/strong>, reducer<\/a><\/strong> in MapReduce, Combiner<\/a><\/strong>, partitioner in MapReduce<\/a> <\/strong>and inputFormat in MapReduce.<\/a><\/strong><\/p>\n

Shuffle<\/strong> phase<\/strong> in Hadoop transfers the map output from Mapper to a Reducer in MapReduce. Sort phase<\/strong> in MapReduce covers the merging and sorting of map outputs. Data from the mapper are grouped by the key, split among reducers and sorted by the key. Every reducer obtains all values associated with the same key. Shuffle and sort phase in Hadoop occur simultaneously and are done by the MapReduce framework.<\/p>\n

Let us now understand both these processes in details below:<\/p>\n

3. Shuffling in MapReduce<\/h2>\n

The process of transferring data from the mappers to reducers is known as shuffling i.e. the process by which the system performs the sort and transfers the map output to the reducer as input. So, MapReduce shuffle phase is necessary for the reducers, otherwise, they would not have any input (or input from every mapper). As shuffling can start even before the map phase has finished so this saves some time and completes the tasks in lesser time.<\/p>\n

4. Sorting in MapReduce<\/h2>\n

The keys generated by the mapper are automatically sorted by MapReduce Framework, i.e. Before starting of reducer, all intermediate key-value pairs<\/strong> <\/a>in MapReduce that are generated by mapper get sorted by key and not by value. Values passed to each reducer are not sorted; they can be in any order.<\/p>\n

Sorting in Hadoop helps reducer to easily distinguish when a new reduce task should start. This saves time for the reducer. Reducer starts a new reduce task when the next key in the sorted input data is different than the previous. Each reduce task takes key-value pairs as input and generates key-value pair as output.<\/p>\n

Note that shuffling and sorting in Hadoop MapReduce is not performed at all if you specify zero reducers (setNumReduceTasks(0)). Then, the MapReduce job stops at the map phase, and the map phase does not include any kind of sorting (so even the map phase is faster).<\/p>\n

5. Secondary Sorting in MapReduce<\/h2>\n

If we want to sort reducer\u2019s values, then the secondary sorting technique is used as it enables us to sort the values (in ascending or descending order) passed to each reducer.<\/p>\n

6. Conclusion<\/h2>\n

In conclusion, Shuffling-Sorting occurs simultaneously to summarize the Mapper intermediate output. Shuffling and sorting in Hadoop MapReduce are not performed at all if you specify zero reducers (setNumReduceTasks(0)).<\/p>\n

If you find this blog helpful, or you have any query in Shuffling and Sorting in Hadoop, so, please leave a comment. Hope we will solve your queries.
\nSee Also-<\/strong><\/p>\n