Hadoop HDFS Commands Tutorial Part-III 1


1. Objective

In this Hadoop HDFS commands tutorial, we are going to learn the remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying a file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide.

hadoop hdfs commands tutorial

Looking to learn HDFS, follow these detailed tutorials: HDFS High AvailabilityHDFS Fault Tolerance

2. Hadoop HDFS Commands Tutorial

Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. In this Hadoop HDFS commands tutorial, we will discuss the frequently use HDFS commands with their usage and description. All the Hadoop file system shell commands are invoked by the bin/hdfs script.

2.1. touchz

Command Usage

 touchz <path> 

Command Example

 hdfs dfs -touchz /user/dataflair/dir2 

Description
It creates a file at the path containing the current time as a timestamp. Fails if a file already exists at a path, unless the file is already size 0.

2.2. test

Command Usage

 hdfs dfs -test -[ezd] URI 

Command Example

 "hdfs dfs -test -e sample
hdfs dfs -test -z sample
hdfs dfs -test -d sample" 

Description
The Hadoop test is used for file test operations.
It gives 1 output if a path exists; it has zero length, or it is a directory or otherwise 0.
Options:
-d: if the path given by the user is a directory, then it gives 0 output.
-e: if the path given by the user exists, then it gives 0 output.
-f: if the path given by the user is a file, then it gives 0 output.
-s: if the path given by the user is not empty, then it gives 0 output.
-z: if the file is zero length, then it gives 0 output.

2.3. text

Command Usage

 hdfs dfs -text <source> 

Command Example

 hdfs dfs -text /user/dataflair/dir1/sample 

Description
Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.

2.4. stat

Command Usage

 hdfs dfs -stat path 

Command Example

 hdfs dfs -stat /user/dataflair/dir1 

Description
Prints information about the path.

%b: If the format is a string which accepts file size in blocks.

%n: Filename

%o: Block size

%r: replication

%y, %Y: modification date.

2.5. tail

Command Usage

  hdfs dfs -tail [-f] <filename2> 

Command Example

 "hdfs dfs -tail /user/dataflair/dir1/sample
hdfs dfs -tail -f /user/dataflair/dir1/sample" 

Description
Shows the last 1KB of the file on console or stdout.

2.6. chown

Command Usage

 hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ] 

Command Example

 hdfs dfs -chown -R dataflair /opt/hadoop/logs 

Description
Changes the owner of files. With -R, makes the change according to the order of the directory structure recursively. The user must be the superuser.

2.7. chmod

Command Usage

 chmod [-R] mode,mode,... <path>... 

Command Example

 hdfs dfs -chmod 777 /user/dataflair/dir1/sample 

Description
Changes the permissions of files. With -R, makes the change recursively by way of the directory structure. The user must be the owner of the file or the superuser.

2.8. appendToFile

Command Usage

 hadoop fs -appendToFile <localsource> ... <dst> 

Command Example

 hadoop fs -appendToFile /home/dataflair/Desktop/sample /user/dataflair/dir1 

Description
Append single sources or multiple sources from local file system to the file system at the destination. It also reads input from standard input and adds it to destination file system.

2.9. checksum

Command Usage

 hadoop fs -checksum URI 

Command Example

 hadoop fs -checksum /user/dataflair/dir1/sample 

Description
Returns the checksum information of a file.

2.10. count

Command Usage

 hdfs dfs -count [-q] <paths> 

Command Example

 hdfs dfs -count /user/dataflair 

Description
Counts the number of directories, number of files present and bytes under the paths that match the specified file pattern.

3. Related Links


Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “Hadoop HDFS Commands Tutorial Part-III

  • Evelyn Serrell

    I came to your Play with HDFS using Commands Part-III – DataFlair Blogs page by searching on Google and I really like your blog. Nice explanation.