{"id":2359,"date":"2017-04-29T09:49:55","date_gmt":"2017-04-29T09:49:55","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2359"},"modified":"2018-11-21T11:28:31","modified_gmt":"2018-11-21T05:58:31","slug":"map-only-job-in-hadoop-mapreduce","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/map-only-job-in-hadoop-mapreduce\/","title":{"rendered":"Map Only Job in Hadoop MapReduce with example"},"content":{"rendered":"

1. Objective<\/h2>\n

In Hadoop<\/a><\/strong>, Map-Only job is the process in which mapper does all task, no task is done by the reducer and mapper\u2019s output is the final output.\u00a0In this tutorial on Map only job in Hadoop MapReduce, we will learn about MapReduce process, the need of map only job in Hadoop, how to set a number of reducers to 0 for Hadoop map only job.<\/p>\n

We will also learn what are the advantages of Map Only job in Hadoop MapReduce, processing in Hadoop without reducer along with MapReduce example with no reducer.<\/p>\n

\"Map<\/a>

Map Only Job in Hadoop MapReduce with example<\/p><\/div>\n

Learn how to install Hadoop 2 with Yarn on pseudo distributed mode.<\/a><\/p>\n

2. What is Map Only Job in Hadoop MapReduce?<\/h2>\n
\"Map-only\"<\/a>

Hadoop MapReduce – Map Only job<\/p><\/div>\n

MapReduce<\/strong><\/a> is a software framework for easily writing applications that process the vast amount of structured and unstructured data stored in the Hadoop Distributed Filesystem (HDFS)<\/a><\/strong>. Two important tasks done by MapReduce algorithm are: Map task<\/strong> and Reduce task.<\/strong> Hadoop Map phase takes a set of data and converts it into another set of data, where individual element are broken down into tuples (key\/value pairs<\/strong><\/a>). Hadoop Reduce phase takes the output from the map as input and combines those data tuples based on the key and accordingly modifies the value of the key.<\/p>\n

From the above word-count example, we can say that there are two sets of parallel process, map<\/strong> and reduce<\/strong>; in map process, the first input is split to distribute the work among all the map nodes as shown in a figure, and then each word is identified and mapped to the number 1. Thus the pairs called tuples (key-value) pairs.<\/p>\n

In the first mapper<\/a><\/strong> node three words lion, tiger, and river are passed. Thus the output of the node will be three key-value pairs with three different keys and value set to 1 and the same process repeated for all nodes. These tuples are then passed to the reducer <\/a><\/strong>nodes and partitioner<\/strong> <\/a>comes into action. It carries out shuffling so that all tuples with the same key are sent to the same node. Thus, in reduce process basically what happens is an aggregation of values or rather an operation on values that share the same key.<\/p>\n

Now, let us consider a scenario where we just need to perform the operation and no aggregation required, in such case, we will prefer \u2018Map-Only job\u2019 in Hadoop. In Hadoop Map-Only job, the map does all task with its InputSplit<\/strong><\/a> and no job is done by the reducer. Here map output is the final output.
\nRefer this guide to learn Hadoop features and design principles<\/a>.<\/p>\n

3. How to avoid Reduce Phase in Hadoop?<\/h2>\n

We can achieve this by setting job.setNumreduceTasks(0)<\/em> in the configuration in a driver. This will make a number of reducer as 0 and thus the only mapper will be doing the complete task.<\/p>\n

4. Advantages of Map only job in Hadoop<\/h2>\n

In between map and reduces phases there is key, sort<\/strong> and shuffle<\/strong> phase. Sort and shuffle are responsible for sorting the keys in ascending order and then grouping values based on same keys. This phase is very expensive and if reduce phase is not required we should avoid it, as avoiding reduce phase would eliminate sort and shuffle phase as well. This also saves network congestion as in shuffling, an output of mapper travels to reducer and when data size is huge, large data needs to travel to the reducer. Learn more about shuffling and sorting in Hadoop MapReduce<\/a>.<\/p>\n

The output of mapper is written to local disk before sending to reducer but in map only job, this output is directly written to HDFS<\/a>. This further saves time and reduces cost as well.<\/p>\n

Also, there is no need of partitioner<\/a><\/strong> and combiner<\/a><\/strong> in Hadoop Map Only job that makes the process fast.<\/p>\n

5. Conclusion<\/h2>\n

In conclusion, Map only job in Hadoop reduces the network congestion by avoiding shuffle, sort and reduce phase. Mapper takes care of overall processing and produces the output. We can achieve this by using the job.setNumreduceTasks(0). <\/em><\/p>\n

Hope you liked this blog. If you have any query related to this blog, so feel free to share with us. We will be happy to solve them.
\nSee Also-<\/strong><\/p>\n