Why we use IntWritable instead of Int? Why we use LongWritable instead of Long?

[DataFlair Team]

Why Hadoop uses IntWritable instead of int ? Why Hadoop needs new data types ? What is difference between default java data types and corresponding Hadoop classes like int vs IntWritable, long vs LongWritable ?

[DataFlair Team]

IntLong are java datatypes, while IntWritableLongWritable are Hadoopdatatypes. We use these datatypes in MapReduce function.These are used to handle objects in hadoop way.The object is serialised into byte of stream and then deserialized later. Since hadoop processes huge amount of data,it needs special datatypes to handle these and provide easier abstraction.

[DataFlair Team]

Since data is processing over a network, in order the optimize the data, Hadoop has it’s own data types. For example, Hadoop uses Text but Java uses the String, due to some similarity. But in Hadoop, Text implements interfaces like Comparable, Writable and WritableComparable.

[DataFlair Team]

In Java, we have also Wrapper classes which are Serialized by default, and Hadoop datatypes are also Serialized but the main difference is that Java Serialized objects are too heavy which consume much network bandwidth while moving over the network. So Hadoop creates own data types to handle the network issues and Hadoop data types are much lighter than Java data types.

[Author]

Posts