Hadoop HDFS Commands with Examples and Usage

Boost your career with Free Big Data Courses!!

In this Hadoop HDFS Commands tutorial, we are going to learn the remaining important and frequently used Hadoop commands with the help of which we will be able to perform HDFS file operations like moving a file, deleting a file, changing files permissions, setting replication factor, changing files ownership, etc.

Hadoop HDFS Commands

Hadoop file system shell commands are used to perform various operations on Hadoop HDFS. In this article, frequently used Hadoop File System Shell commands are discussed with their example and usage. All the Hadoop basic commands are invoked by the bin/hdfs script.

If you face any doubt in Hadoop HDFS Commands, Please Ask us in comments.

Let us now start with the HDFS commands.

hdfs commands

1. moveFromLocal

HDFS moveFromLocal Command Usage:

hadoop fs -moveFromLocal <localsrc> <dest>

HDFS moveFromLocal Command Example:

Here we are trying to move the file ‘test1’ from the local filesystem to the Hadoop filesystem.

moveFromLocal HDFS commands

checking for file movefromlocal - HDFS commands

The file from the local gets deleted.

HDFS moveFromLocal Command Description:

The Hadoop fs shell command moveFromLocal moves the file or directory from the local filesystem to the destination in Hadoop HDFS.

2. moveToLocal

HDFS moveToLocal Command Usage:

hadoop fs -moveToLocal <src> <localdest>

HDFS moveToLocal Command Example:

This command is not yet implemented.

HDFS moveToLocal Command Description:

The Hadoop fs shell command moveToLocal moves the file or directory from the Hadoop filesystem to the destination in the local filesystem.

3. tail

HDFS tail Command Usage:

hadoop fs -tail [-f] <file>

HDFS tail Command Example:

Here using the tail command, we are trying to display the 1KB of file ‘test’ present in the dataflair directory on the HDFS filesystem.

tail command - HDFS commands

tail with fcommand-HDFS commands

HDFS tail Commnad Description:

The Hadoop fs shell tail command shows the last 1KB of a file on console or stdout.

The -f shows the append data as the file grows.

If you missed Top 10 Hadoop Hdfs Commands in Part-II then practice them as well.

4. rm

HDFS rm Command Usage:

hadoop fs –rm <path>

HDFS rm Command Example:

The below example deletes the file ‘new’ using rm command.

rm command - HDFS commands

Here in the below example we are recursively deleting the DataFlair directory using -r with rm command.

HDFS rm Command Description:

The rm command removes the file present in the specified path.

HDFS rm Command Example:

Here in the below example we are recursively deleting the DataFlair directory using -r with rm command.

find command - HDFS commands

HDFS rm Command Description:
Recursive version of delete.

5. expunge

HDFS expunge Command Usage:

hadoop fs -expunge

HDFS expunge Command Example:

expunge command - HDFS commands

HDFS expunge Command Description:
HDFS expunge command makes the trash empty.

Learn few more frequently used Hadoop Commands with Examples and Usage in Part-III.

6. chown

HDFS chown Command Usage:

hadoop fs -chown [-R] [owner] [:[group]] <path>

HDFS chown Command Example:
Here we are changing the owner of a file name sample using the chown command.

chown command - HDFS commands

checking for owner - HDFScommands

HDFS chown Command Description:

The Hadoop fs shell command chown changes the owner of the file.

The -R option recursively changes files permissions through the directory structure. The user must be the owner of the file or superuser.

To learn more about the world’s most reliable storage layer follow this HDFS introductory guide.

7. chgrp

HDFS chgrp Command Usage:

hadoop fs -chgrp <group> <path>

HDFS chgrp Command Example:

In the below example, we are changing the group of ‘sample.zip’ file of the HDFS file system.

chgrp command - HDFS commands

checkingforgroup - HDFScommands

Use -R option to change recursively through the directory structure.

Here we are trying to change the group of all files present in the DataFlair directory on the HDFS filesystem.

chrgp with Rcommand - HDFS commands

HDFS chgrp Description:

The Hadoop fs shell command chgrp changes the group of the file specified in the path.

The user must be the owner of the file or superuser.

8. setrep

HDFS setrep Command Usage:

hadoop fs -setrep <rep> <path>

HDFS setrep Command Example:

Here we are trying to change the replication factor of the ‘sample’ file present in newDataFlair directory on the HDFS filesystem.

setrep command - HDFScommands

Here we are trying to change the replication factor of all files residing in the newDataFlair directory.

setrep command - HDFS commands

HDFS setrep Command Description:

setrep command changes the replication factor to a specific count instead of the default replication factor for the file specified in the path. If used for a directory, then it will recursively change the replication factor for all the files residing in the directory.

9. du

HDFS du Command Usage:

hadoop fs –du –s /directory/filename

HDFS du Command Example:

du command - HDFS commands

Here we are checking the size of file ‘file1’ in newDataFlair directory of HDFS.

HDFS du Description:

This Hadoop fs shell command du prints a summary of the amount of disk usage of all files/directories in the path.

10. df

HDFS df Command Usage:

hadoop fs -df [-h] <path>

HDFS df Command Example:df command - HDFS commands

HDFS df Command Description:

The Hadoop fs shell command df shows the capacity, size, and free space available on the HDFS file system.

The -h option formats the file size in the human-readable format.

11. fsck

HDFS fsck Command Usage:

hadoop fsck <path> [ -move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]

HDFS fsck Command Example:

In this example, we are trying to check the health of the files in ‘dataflair’ directory present in HDFS using the fsck command.

fsck command - HDFS commands

fsck command output - HDFS commands

HDFS fsck Command Description:

The fsck Hadoop command is used to check the health of the HDFS.

OptionsDescription
<path>start checking from the path specified here
-moveIt moves a corrupted file to the lost+found directory.
-deleteIt deletes the corrupted files present in HDFS.
-openforwriteIt prints the files which are opened for write
-filesIt prints the files being checked.
-blocksIt prints out all the blocks of the file while checking.
-locationsIt prints the location of all the blocks of files while checking.
-racksIt displays the network topology for DataNode locations.

What’s Next

Play with few more HDFS Commands in Part-III and know everything to perform operations on HDFS files/directories.

If you find this Hadoop HDFS commands helpful or if you have any queries let us know in the comments and our support team will get back to you.

Hope You like the Hadoop HDFS Commands Tutorial.

Keep Executing!!

Did you like this article? If Yes, please give DataFlair 5 Stars on Google

follow dataflair on YouTube

12 Responses

  1. Brinu says:

    Thank you for sharing HDFS commands. This really helped me a lot.

  2. Jenni says:

    Really appreciate your blog that has shared many details. I like your other blogs also and regularly visit your page. Keep it up!!

    • Data Flair says:

      Thank you, Jenni, for complimenting on Hadoop HDFS Commands tutorial. Each Hadoop HDFS tutorial from Data Flair is an effort for every reader to provide them with latest and researched information.
      It is nice to know that you regularly visit Data Flair. Still, if you want more articles and information, you can follow us our FB page also.

  3. Madhavi says:

    This blog is the source of knowledge to enhance the career in Bigdata for the lifetime.

  4. DEVASHISH KUMAR CHOUDHARY says:

    thank you

  5. Ram says:

    Really appreciate your blog and it has all details required for beginners with easy language.

    Thanks for writing such blog and keep doing it.

    • DataFlair Team says:

      Hello Ram,

      We are glad that you liked our blog. Refer to our sidebar, there you can find more HDFS tutorials for further knowledge.

  6. swathi says:

    Hi
    It is very helpful. I am not clear with the usage of -R command .
    eg : RM -r : what is mean by deleting recursively?
    difference between rm and rm -r

  7. LAURA says:

    why when using the -du flag there are two different figures?

Leave a Reply

Your email address will not be published. Required fields are marked *