Hdfs Commands Tutorial Part-II – Top 10 HDFS Commands 3


1. Objective

In this Hadoop HDFS Commands tutorial, we are going to learn the remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying a file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide.

HDFS commands tutorial

2. HDFS Commands Tutorial

Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. In this mostly used HDFS commands tutorial top HDFS commands are discussed below. All the Hadoop file system shell commands are invoked by the bin/hdfs script.

2.1. moveFromLocal

Command Usage

 moveFromLocal <localSrc> <dest> 

Command Example

 hdfs dfs -moveFromLocal /home/dataflair/Desktop/sample /user/dataflair/dir1 

Description
Copies the file or directory from the local file system identified by the local source to destination within HDFS, and then deletes the local copy on success.

Learn more about Internals of HDFS Data Write Pipeline and File write execution flow

2.2. moveToLocal

Command Usage

 moveToLocal <src> <localDest> 

Command Example

 hdfs dfs -moveToLocal /user/dataflair/dir2/sample /user/dataflair/Desktop 

Description
Works like -get, but deletes the HDFS copy on success.

2.3. tail

Command Usage

hdfs dfs -tail [-f] <filename> 

Command Example

 "hdfs dfs -tail /user/dataflair/dir2/purchases.txt
hdfs dfs -tail -f /user/dataflair/dir2/purchases.txt" 

Description
Shows the last 1KB of the file on console or stdout.

2.4. rm

Command Usage

 rm <path> 

Command Example

 hdfs dfs -rm /user/dataflair/dir2/sample 

Description
Removes the file or empty directory present on the path provided by the user.

Command Example

 hdfs dfs -rm -r /user/dataflair/dir2 

Description
Recursive version of delete.

2.5. expunge

Command Usage

 hdfs dfs -expunge 

Command Example

 hdfs dfs -expunge 

Description
Used to empty the trash.

2.6. chown

Command Usage

 hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ] 

Command Example

 hdfs dfs -chown -R dataflair /opt/hadoop/logs 

Description
Changes the owner of files. With -R, changes are made recursively by way of the structure of the directory. A user should be the superuser.

2.7. chgrp

Command Usage

 hdfs dfs -chgrp [-R] <NewGroupName> <file or directory name> 

Command Example

 hdfs dfs -chgrp [-R] New Group sample 

Description
Hadoop chgrp shell command is used to change the files group association. Also, you can try -R option to make changes recursively by the way of structure of the directory.

2.8. setrep

Command Usage

 setrep [-R] [-w] rep <path> 

Command Example

 hdfs dfs -setrep -w 3 /user/dataflair/dir1 

Description
It is used to change the replication factor of a file. If the entered path is a directory, then this command changes the replication factor of all the files present in the directory tree rooted at path provided by user recursively.
Options:
The -w flag requests that the command waits for the replication process to get completed. This may likely take a very long time to get completed.
The -R flag is accepted for backward compatibility. It does not make any changes.

2.9. du

Command Usage

 du <path> 

Command Example

 hdfs dfs -du /user/dataflair/dir1/sample 

Description
Shows disk usage, in bytes, for all the files present on the path provided by the user; reporting of filenames are done with the full HDFS protocol prefix.

Command Example

hdfs dfs -du -s /user/dataflair/dir1/sample

Description
Like -du, but it prints a summary of the amount of disk usage of all files/directories in the path.

2.10. df

Command Usage

 hdfs dfs -df [-h] URI [URI ...] 

Command Example

hdfs dfs -df -h

Description
Displays free space.

3. Related Links


Leave a comment

Your email address will not be published. Required fields are marked *

3 thoughts on “Hdfs Commands Tutorial Part-II – Top 10 HDFS Commands