- This topic has 3 replies, 1 voice, and was last updated 5 years, 7 months ago by
.
-
Posts
-
-
-
In Hadoop, Namenode has 2 types of metadata:
1) File System Metadata: It his having all the information about file i.e what are the permission to the particular file, time-stamp, size, replication factor, Data Blocks of that file. It further categorized into two categories:
a) fsimage: It is having the snapshot of the file system with which Namenode will start for very first time. It always persisted to our disk due to which we don’t loose data even after shut down of Namenode.
b) edits(Journal): Whatever operations i.e write, delete, update, etc, it all stores in Journal and hence it keeps on growing during the life of cluster.
Journal will be committed to fsimage after either Rebooting or using Checkpoints.Checkpoints: It commits the Journal to fsimage.
2) Bitmap: It is having information about all the Mapping between block and data node i.e which block on which node. This metadata will never persist on disk.
-
Namenode main responsibility is storing HDFS directories and files in a tree. It contains the directories and file attributes like replication factor, permissions, ownership, timestamp etc.It loads in memory, it also keeps a persistent copy in the hard disk.
Namenode stores filesystem metadata which is further divided in Fsimage and Editlog. Fsimage keeps the latest snapshot of the file system metadata.But unsuitable for small incremental updates so to avoid creation of new Fsimage everytime an Editlog records all the creation, update and deletion operations for durability.
So when NameNode crashes, it can restore its state by first loading the fsimage then replaying all the operations in the edit log to catch up to the most recent state of the namesystem.
It also stores file to block mapping i.e every Data Blocks is present on which slave node, these are block reports sent by Datanode to Namenode.
-
The NameNode consists two types of Metadata.
One is the FSImage and the other is the Edit logs.
FSImage- FSImagecontains all the file system properties and information about how the Data Blocks are mapped i.e. the information about each datanode and the mapped blocks, this is done by Hadoop which creates Block reports. The Block reports are created when the NameNode is started or restarted.
Edit logs- When a filesystem client performs any write operation (such as creating or moving a file), the transaction is first recorded in the edit log.
-
- You must be logged in to reply to this topic.
