![\"Hadoop](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/HDFS-Data-Read-and-Write-Operations.jpg\")
<\/a>Hadoop HDFS Data Read and Write Operations<\/p><\/div>\n
This blog also contains the videos to deeply understand the internals of HDFS file read and write operations.<\/p>\n
2. Hadoop HDFS Data Read and Write Operations<\/h2>\n
HDFS<\/strong> \u2013 Hadoop Distributed File System<\/em> <\/strong><\/a>is the storage layer of Hadoop<\/strong>. It is most reliable storage system on the planet. HDFS works in master-slave<\/em> fashion, NameNode<\/strong> is the master daemon which runs on the master node, DataNode<\/strong> is the slave daemon which runs on the slave node. Here, we are going to cover the HDFS data read and write operations. Let’s discuss HDFS file write operation first followed by HDFS file read operation-<\/p>\n To write a file in HDFS<\/a><\/strong>, a client needs to interact with master i.e. namenode\u00a0<\/strong>(master). Now namenode provides the address of the datanodes<\/strong> (slaves) on which client will start writing the data. Client directly writes data on the datanodes, now datanode will create data write pipeline. Now let\u2019s understand complete end to end <\/span>HDFS <\/a><\/strong>data write pipeline. As shown in the above figure the data write operation in HDFS is distributed, client copies the data distributedly on <\/span>datanodes, the steps by step explanation of data write operation is:<\/span>i)\u00a0<\/b>The HDFS client sends a create<\/strong> request on DistributedFileSystem<\/em> APIs. The namenode performs various checks to make sure that the file doesn\u2019t already exist and that the client has the permissions to create the file. When these checks pass, then only the namenode makes a record of the new file; otherwise, file creation fails and the client is thrown an IOException. <\/em>Also Learn Hadoop HDFS Architecture<\/a><\/strong> in Detail.<\/p>\n iii)\u00a0<\/b>The DistributedFileSystem<\/em> returns a FSDataOutputStream<\/em> for the client to start writing data to. As the client writes data, DFSOutputStream<\/em> splits it into packets, which it writes to an internal queue, called the data queue. The data queue is consumed by the DataStreamer, which<\/em>I is responsible for asking the namenode to allocate new blocks<\/strong><\/a> by picking a list of suitable datanodes to store the replicas.<\/p>\n iv)\u00a0<\/b>The list of datanodes form a pipeline, and here we\u2019ll assume the replication level is three, so there are three nodes in the pipeline. The DataStreamer<\/em> streams the packets to the first datanode in the pipeline, which stores the packet and forwards it to the second datanode in the pipeline. Similarly, the second datanode stores the packet and forwards it to the third (and last) datanode in the pipeline. Learn HDFS Data blocks in detail<\/strong>.<\/a><\/p>\n v)\u00a0<\/b>DFSOutputStream<\/em> also maintains an internal queue of packets that are waiting to be acknowledged by datanodes, called the ack queue<\/em>. A packet is removed from the ack queue only when it has been acknowledged by the datanodes in the pipeline. Datanode sends the acknowledgment once required replicas are created (3 by default). Similarly, all the blocks are stored and replicated on the different datanodes, the data blocks are copied in parallel.<\/p>\n vi)\u00a0<\/b>When the client has finished writing data, it calls close()<\/strong> on the stream.<\/p>\n vii)\u00a0<\/b>This action flushes all the remaining packets to the datanode pipeline and waits for acknowledgments before contacting the namenode to signal that the file is complete. The namenode already knows which blocks the file is made up of, so it only has to wait for blocks to be minimally replicated before returning successfully.<\/p>\n Learn: Hadoop HDFS Data Read and Write Operations<\/a><\/strong> Data Write Mechanism in HDFS Tutorial<\/p><\/div>\n A sample code to write a file to HDFS in Java is as follows (To interact with HDFS and perform various operations follow this HDFS command part – 1<\/a><\/strong>):<\/p>\n [php]FileSystem fileSystem = FileSystem.get(conf); Hadoop HDFS Commands with Examples and Usage \u2013 Part II<\/a><\/strong><\/p>\n <\/b>To read a file from HDFS<\/a><\/strong>, a client needs to interact with namenode (master) as namenode is the centerpiece of Hadoop cluster<\/a><\/strong> (it stores all the metadata i.e. data about the data). Now namenode checks for required privileges, if the client has sufficient privileges then namenode provides the address of the slaves where a file is stored. Now client will interact directly with the respective datanodes to read the data blocks.<\/p>\n Now let\u2019s understand complete end to end HDFS data read operation. As shown in the above figure the data read operation in HDFS is distributed, the client reads the data parallelly from datanodes, the steps by step explanation of data read cycle is:<\/p>\n HDFS data read and write operations<\/p><\/div>\n i)<\/strong> Client opens the file it wishes to read by calling open()<\/strong> on the FileSystem<\/em> object, which for HDFS is an instance of DistributedFileSystem<\/em>. See Data Read Operation<\/a><\/strong> in HDFS A sample code to read a file from HDFS is as follows (To perform HDFS read and write operations\u00a0follow this HDFS command part – 3<\/strong><\/a>): As we have discussed HDFS data read and write operations in detail, Now, what happens when one of the machines i.e. part of the pipeline which has a datanode process running fails. Hadoop has an inbuilt functionality to handle this scenario (HDFS is fault tolerant<\/a><\/strong>). When a datanode fails while data is being written to it, then the following actions are taken, which are transparent to the client writing the data.<\/p>\n It\u2019s possible, but unlikely, that multiple datanodes fail while client writes a block. As long as it writes\u00a0dfs.replication.min<\/em> replicas (which default to<\/strong> 1<\/strong>), the write will be successful, and the block will asynchronously replicate across the cluster until it achieves target replication factor \u00a0(dfs.replication<\/em>, which defaults to<\/strong> 3<\/strong>). Learn: NameNode High Availability in Hadoop HDFS<\/a><\/strong><\/p>\n In conclusion, This design allows HDFS to increase the\u00a0numbers of clients. This is because the data traffic spreads on all the Datanodes of clusters. It also provides High Availability<\/strong><\/a>, Rack Awareness<\/strong><\/a>, Erasure coding<\/strong><\/a> etc, As a result, it empowers Hadoop. References:<\/strong> 1. Objective HDFS follow Write once Read many models. So we cannot edit files already stored in HDFS, but we can append data by reopening the file. In Read-Write operation client first, interact with...<\/p>\n","protected":false},"author":7,"featured_media":30491,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[782,1971,5201,5204,5592,5599,11356],"class_list":["post-425","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hdfs","tag-apache-hadoop","tag-big-data-training","tag-hadoop-admin","tag-hadoop-administration","tag-hdfs-read-operation","tag-hdfs-write-operation","tag-read-and-write-operations-in-hdfs"],"yoast_head":"\n
\nBefore start using with HDFS, you should install Hadoop. I recommend you-<\/p>\n\n
2.1. Hadoop HDFS Data Write Operation<\/h3>\n
\nThe first datanode will copy the block to another datanode, which intern copy it to the third datanode. Once it creates the replicas of blocks, it sends back the acknowledgment.<\/p>\na. HDFS Data Write Pipeline Workflow<\/h4>\n
\nii)\u00a0<\/b>DistributedFileSystem<\/em> makes an RPC call to the namenode to create a new file in the file system’s namespace.<\/p>\n
\nWe can summarize the HDFS data write operation from the following diagram:<\/em><\/p>\n![\"Data](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/05\/Data-Write-Mechanism-in-HDFS.gif\")
<\/a>b. How to Write a file in HDFS – Java Program<\/h4>\n
\n\/\/ Check if the file already exists
\nPath path = new Path(“\/path\/to\/file.ext”);
\nif (fileSystem.exists(path)) {
\nSystem.out.println(“File ” + dest + ” already exists”);
\nreturn;
\n}
\n\/\/ Create a new file and write data to it.
\nFSDataOutputStream out = fileSystem.create(path);
\nInputStream in = new BufferedInputStream(new FileInputStream(
\nnew File(source)));
\nbyte[] b = new byte[1024];
\nint numBytes = 0;
\nwhile ((numBytes = in.read(b)) > 0) {
\nout.write(b, 0, numBytes);
\n}
\n\/\/ Close all the file descripters
\nin.close();
\nout.close();
\nfileSystem.close();
\n[\/php]<\/p>\n2.2. Hadoop HDFS Data Read Operation<\/h3>\n
a. HDFS File Read Workflow<\/h4>\n
![\"HDFS](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/05\/Data-Read-Mechanism-in-HDFS.gif\")
<\/a>
\nii)<\/strong> DistributedFileSystem<\/em> calls the namenode using RPC to determine the locations of the blocks for the first few blocks in the file. For each block, the namenode returns the addresses of the datanodes that have a copy of that block<\/a><\/strong> and datanode are sorted according to their proximity to the client.
\niii)<\/strong> DistributedFileSystem<\/em> returns a FSDataInputStream<\/em> to the client for it to read data from. FSDataInputStream<\/em>, thus, wraps the DFSInputStream<\/em> which manages the datanode and namenode I\/O. Client calls read()<\/strong> on the stream. DFSInputStream which has stored the datanode addresses then connects to the closest datanode for the first block in the file.
\niv)<\/strong> Data is streamed from the datanode back to the client, as a result client can call read()<\/strong> repeatedly on the stream. When the block ends, DFSInputStream will close the connection to the datanode and then finds the best datanode for the next block. Also learn about Data write operation <\/a><\/strong>in HDFS
\nv)\u00a0<\/b>If the DFSInputStream<\/em> encounters an error while communicating with a datanode, it will try the next closest one for that block. It will also remember datanodes that have failed so that it doesn\u2019t needlessly retry them for later blocks. The DFSInputStream<\/em> also verifies checksums for the data transferred to it from the datanode. If it finds a corrupt block, it reports this to the namenode before the DFSInputStream<\/em> attempts to read a replica of the block from another datanode.vi)\u00a0<\/b>When the client has finished reading the data, it calls close()<\/strong> on the stream.
\nWe can summarize the HDFS data read operation from the following diagram:<\/em><\/p>\nb. How to Read a file from HDFS – Java Program<\/h4>\n
\n[php]FileSystem fileSystem = FileSystem.get(conf);
\nPath path = new Path(“\/path\/to\/file.ext”);
\nif (!fileSystem.exists(path)) {
\nSystem.out.println(“File does not exists”);
\nreturn;
\n}
\nFSDataInputStream in = fileSystem.open(path);
\nint numBytes = 0;
\nwhile ((numBytes = in.read(b))> 0) {
\nSystem.out.prinln((char)numBytes));\/\/ code to manipulate the data which is read
\n}
\nin.close();
\nout.close();
\nfileSystem.close();[\/php]<\/p>\n3. Fault Tolerance in HDFS<\/h2>\n
\n
\nFailover process is same as data write operation, data read operation is also fault tolerant.<\/p>\n4. Conclusion<\/h2>\n
\nIf you like this post or have any query about\u00a0HDFS data read and write operations, so please leave a comment. We will be happy to solve them.
\nSee Also-<\/b><\/p>\n\n
\nHadoop – The definitive guide<\/a><\/strong>
\n hadoop.apache.org<\/a><\/strong><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"