{"id":2240,"date":"2017-04-25T08:35:49","date_gmt":"2017-04-25T08:35:49","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2240"},"modified":"2018-11-21T11:36:44","modified_gmt":"2018-11-21T06:06:44","slug":"inputsplit-in-hadoop-mapreduce","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/inputsplit-in-hadoop-mapreduce\/","title":{"rendered":"InputSplit in Hadoop MapReduce – Hadoop MapReduce Tutorial"},"content":{"rendered":"

1. Objective<\/h2>\n

In this Hadoop<\/strong><\/a> MapReduce tutorial, we will provide you the detailed description of InputSplit in Hadoop. In this blog, we will try to answer What is Hadoop InputSplit, what is the need of inputSplit in MapReduce and how Hadoop performs InputSplit, How to change split size in Hadoop. We will also learn the difference between InputSplit vs Blocks in HDFS.<\/strong><\/a><\/p>\n

\"InputSplit<\/a>

InputSplit in Hadoop MapReduce – Hadoop MapReduce Tutorial<\/p><\/div>\n

2. What is InputSplit in Hadoop?<\/h2>\n

InputSplit<\/strong> in Hadoop MapReduce<\/strong> <\/a>is the logical representation of data. It describes a unit of work that contains a single map task in a MapReduce program.
\nHadoop InputSplit represents the data which is processed by an individual Mapper<\/a><\/strong>. The split is divided into records. Hence, the mapper process each record (which is a key-value pair<\/strong><\/a>).<\/p>\n

MapReduce InputSplit length is measured in bytes and every InputSplit has storage locations (hostname strings). MapReduce system use storage locations to place map tasks as close to split\u2019s data as possible. Map tasks are processed in the order of the size of the splits so that the largest one gets processed first (greedy approximation algorithm) and this is done to minimize the job runtime (Learn MapReduce job optimization techniques<\/a>)The important thing to notice is that Inputsplit does not contain the input data; it is just a reference to the data.<\/p>\n

As a user, we don\u2019t need to deal with InputSplit directly, because they are created by an InputFormat<\/a> <\/strong>(InputFormat creates the Inputsplit and divide into records). FileInputFormat, by default, breaks a file into 128MB<\/strong> chunks (same as blocks in HDFS) and by setting mapred.min.split.size<\/em> parameter in mapred-site.xml<\/em> we can control this value or by overriding the parameter in the Job object used to submit a particular MapReduce job<\/strong><\/a>. We can also control how the file is broken up into splits, by writing a custom InputFormat.<\/p>\n

3. How to change split size in Hadoop?<\/h2>\n

InputSplit in Hadoop is user defined. User can control split size according to the size of data in MapReduce program. Thus the number of map tasks is equal to the number of InputSplits.<\/p>\n

The client (running the job) can calculate the splits for a job by calling \u2018getSplit()<\/strong>\u2019, and then sent to the application master, which uses their storage locations to schedule map tasks that will process them on the cluster. Then, map task passes the split to the createRecordReader()<\/em> method on InputFormat to get\u00a0RecordReader<\/strong> <\/a>for the split and RecordReader generate record (key-value pair), which it passes to the map function.<\/p>\n

4. Conclusion<\/h2>\n

In conclusion, InputSplit doesn\u2019t contain actual data, but a reference to the data. It is used during data processing in MapReduce program or other processing techniques. Hence, the split is divided into records and each record (which is a key-value pair) is processed by the map. Hope this blog helped you. If you like this blog or have any query related to InputSplit in Hadoop, so, leave a comment in a section given below.<\/p>\n

Reference:<\/strong><\/p>\n

http:\/\/hadoop.apache.org\/<\/a><\/p>\n

See Also-<\/strong><\/p>\n