{"id":431,"date":"2016-06-13T14:02:03","date_gmt":"2016-06-13T14:02:03","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=431"},"modified":"2018-11-19T16:19:07","modified_gmt":"2018-11-19T10:49:07","slug":"hadoop-mapreduce-flow","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hadoop-mapreduce-flow\/","title":{"rendered":"Hadoop MapReduce Flow \u2013 How data flows in MapReduce?"},"content":{"rendered":"

1. Objective<\/h2>\n

Hadoop<\/a><\/strong> MapReduce processes a huge amount of data in parallel by dividing the job into a set of independent tasks (sub-job). In Hadoop, MapReduce works by breaking the processing into phases: Map and Reduce. In this tutorial, will explain you the complete Hadoop MapReduce flow.<\/p>\n

This MapReduce tutorial, will cover an end to end Hadoop\u00a0MapReduce\u00a0flow. Hope this blog will give you the answer for how Hadoop MapReduce works, how data is processed when a map-reduce job is submitted. Step by step execution flow of MapReduce, what are the steps involved in MapReduce job execution, etc. Let’s get the answer to all these questions with the deep study of Hadoop MapReduce flow.<\/p>\n

\"Hadoop<\/a>

Hadoop MapReduce Flow \u2013 How data flows in MapReduce?<\/p><\/div>\n

2. What is MapReduce?<\/h2>\n

MapReduce<\/strong> is the data processing layer of Hadoop (other layers are HDFS<\/a><\/strong> – data processing layer, Yarn<\/a><\/strong> – resource management layer).\u00a0MapReduce is a programming paradigm designed for processing huge\u00a0volumes of data in parallel by dividing the job (submitted work)\u00a0into a set of independent tasks (sub-job). You just need to put the custom code (business logic) in the way map reduce works and rest things will be taken care by the engine.<\/p>\n

2.1. Hadoop MapReduce Flow – MapReduce Video Tutorial<\/h3>\n

We have also provided the video tutorial for more understanding of the internal of Hadoop MapReduce flow.<\/p>\n

3. How Hadoop MapReduce Works?<\/h2>\n
\"hadoop<\/a>

MapReduce Data Flow<\/p><\/div>\n

MapReduce<\/strong> is the heart of Hadoop<\/strong>. It is a programming model designed for processing huge\u00a0volumes of data (both structured as well as unstructured) in parallel by dividing the work into a set of independent sub-work (tasks). Let’s discuss How MapReduce works internally-<\/p>\n

3.1. MapReduce Internals<\/h3>\n

MapReduce is the combination of two different processing idioms called Map<\/strong> and Reduce<\/strong>, where we can specify our custom business logic. The map is the first phase of processing, where we specify all the complex logic\/business rules\/costly code. On the other hand, Reduce is the second phase of processing, where we specify light-weight processing. For example, aggregation\/summation.<\/p>\n

3.2. Step by step MapReduce Job Flow<\/h3>\n

The data processed by MapReduce should be stored in HDFS<\/strong>, which divides the data into blocks and store distributedly, for more details about HDFS\u00a0follow this HDFS comprehensive tutorial<\/a>. Below are the steps for MapReduce data flow:<\/p>\n