Hadoop HDFS Commands with Examples and Usage

In this Hadoop HDFS Commands tutorial, we are going to learn the remaining important and frequently used Hadoop commands with the help of which we will be able to perform HDFS file operations like moving a file, deleting a file, changing files permissions, setting replication factor, changing files ownership, etc.

Hadoop HDFS Commands

Hadoop file system shell commands are used to perform various operations on Hadoop HDFS. In this article, frequently used Hadoop File System Shell commands are discussed with their example and usage. All the Hadoop basic commands are invoked by the bin/hdfs script.

If you face any doubt in Hadoop HDFS Commands, Please Ask us in comments.

Let us now start with the HDFS commands.

1. moveFromLocal

HDFS moveFromLocal Command Usage:

hadoop fs -moveFromLocal <localsrc> <dest>

HDFS moveFromLocal Command Example:

Here we are trying to move the file ‘test1’ from the local filesystem to the Hadoop filesystem.

The file from the local gets deleted.

HDFS moveFromLocal Command Description:

The Hadoop fs shell command moveFromLocal moves the file or directory from the local filesystem to the destination in Hadoop HDFS.

2. moveToLocal

HDFS moveToLocal Command Usage:

hadoop fs -moveToLocal <src> <localdest>

HDFS moveToLocal Command Example:

This command is not yet implemented.

HDFS moveToLocal Command Description:

The Hadoop fs shell command moveToLocal moves the file or directory from the Hadoop filesystem to the destination in the local filesystem.

3. tail

HDFS tail Command Usage:

hadoop fs -tail [-f] <file>

HDFS tail Command Example:

Here using the tail command, we are trying to display the 1KB of file ‘test’ present in the dataflair directory on the HDFS filesystem.

HDFS tail Commnad Description:

The Hadoop fs shell tail command shows the last 1KB of a file on console or stdout.

The -f shows the append data as the file grows.

If you missed Top 10 Hadoop Hdfs Commands in Part-II then practice them as well.

4. rm

HDFS rm Command Usage:

hadoop fs –rm <path>

HDFS rm Command Example:

The below example deletes the file ‘new’ using rm command.

Here in the below example we are recursively deleting the DataFlair directory using -r with rm command.

HDFS rm Command Description:

The rm command removes the file present in the specified path.

HDFS rm Command Example:

Here in the below example we are recursively deleting the DataFlair directory using -r with rm command.

HDFS rm Command Description:
Recursive version of delete.

5. expunge

HDFS expunge Command Usage:

hadoop fs -expunge

HDFS expunge Command Example:

HDFS expunge Command Description:
HDFS expunge command makes the trash empty.

Learn few more frequently used Hadoop Commands with Examples and Usage in Part-III.

6. chown

HDFS chown Command Usage:

hadoop fs -chown [-R] [owner] [:[group]] <path>

HDFS chown Command Example:
Here we are changing the owner of a file name sample using the chown command.

HDFS chown Command Description:

The Hadoop fs shell command chown changes the owner of the file.

The -R option recursively changes files permissions through the directory structure. The user must be the owner of the file or superuser.

To learn more about the world’s most reliable storage layer follow this HDFS introductory guide.

7. chgrp

HDFS chgrp Command Usage:

hadoop fs -chgrp <group> <path>

HDFS chgrp Command Example:

In the below example, we are changing the group of ‘sample.zip’ file of the HDFS file system.

Use -R option to change recursively through the directory structure.

Here we are trying to change the group of all files present in the DataFlair directory on the HDFS filesystem.

HDFS chgrp Description:

The Hadoop fs shell command chgrp changes the group of the file specified in the path.

The user must be the owner of the file or superuser.

8. setrep

HDFS setrep Command Usage:

hadoop fs -setrep <rep> <path>

HDFS setrep Command Example:

Here we are trying to change the replication factor of the ‘sample’ file present in newDataFlair directory on the HDFS filesystem.

Here we are trying to change the replication factor of all files residing in the newDataFlair directory.

HDFS setrep Command Description:

setrep command changes the replication factor to a specific count instead of the default replication factor for the file specified in the path. If used for a directory, then it will recursively change the replication factor for all the files residing in the directory.

9. du

HDFS du Command Usage:

hadoop fs –du –s /directory/filename

HDFS du Command Example:

Here we are checking the size of file ‘file1’ in newDataFlair directory of HDFS.

HDFS du Description:

This Hadoop fs shell command du prints a summary of the amount of disk usage of all files/directories in the path.

10. df

HDFS df Command Usage:

hadoop fs -df [-h] <path>

HDFS df Command Example:

HDFS df Command Description:

The Hadoop fs shell command df shows the capacity, size, and free space available on the HDFS file system.

The -h option formats the file size in the human-readable format.

11. fsck

HDFS fsck Command Usage:

hadoop fsck <path> [ -move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]

HDFS fsck Command Example:

In this example, we are trying to check the health of the files in ‘dataflair’ directory present in HDFS using the fsck command.

HDFS fsck Command Description:

The fsck Hadoop command is used to check the health of the HDFS.

Options	Description
<path>	start checking from the path specified here
-move	It moves a corrupted file to the lost+found directory.
-delete	It deletes the corrupted files present in HDFS.
-openforwrite	It prints the files which are opened for write
-files	It prints the files being checked.
-blocks	It prints out all the blocks of the file while checking.
-locations	It prints the location of all the blocks of files while checking.
-racks	It displays the network topology for DataNode locations.

What’s Next

Play with few more HDFS Commands in Part-III and know everything to perform operations on HDFS files/directories.

If you find this Hadoop HDFS commands helpful or if you have any queries let us know in the comments and our support team will get back to you.

Hope You like the Hadoop HDFS Commands Tutorial.

Keep Executing!!