What do you mean by metadata in Hadoop HDFS?

[DataFlair Team]

What do you mean by metadata in HDFS?
Where is metadata stored in Hadoop?

[DataFlair Team]

In Hadoop, HDFS(Hadoop distributed files system) is used for storing data. It has 2 components: Name node(master node) and Data node(Slave node). In Data node actual data is stored and name node stores the meta data that is the file location, block size, file permission. It also receives heart beats from live data nodes, so it is also responsible for sending a signal to replicate if data node is no more available.

Metadata is stored in memory of name node. So it is recommended to have large RAM for name node machines.

[DataFlair Team]

Metadata is the data about the data. Metadata is stored in namenode where it stores data about the data present in datanode like location about the data and their replicas.

[DataFlair Team]

NameNode stores the Metadata, this consists of fsimage and editlog.

Fsimage:
This contained serialized form of all directory and file in the file System.The FsImage is stored as a file in the NameNode’s local file system.

Edit Log:
This is a transaction log, which logs every change in the file system.

On check point, secondary NameNode reads the fsimage and the edit log from the disk.Then merges the edit log into fsimage in memory then flushes out the updated fsimage into the Active Name Node. A checkpoint can be triggered at a given time interval.

[Author]

Posts