Top 10 Hadoop HDFS Commands with Examples and Usage

1. Hadoop HDFS Commands

In this tutorial, we are going to learn the most important and frequently used Hadoop HDFS commands with the help of which we will be able to perform HDFS file operations like copying the file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. In this Hadoop Commands tutorial we have mentioned the most frequently used HDFS commands.

Top 10 Hadoop HDFS Commands with Examples and Usage

Top 10 Hadoop HDFS Commands with Examples and Usage

2. Hadoop HDFS Commands: Introduction

Hadoop HDFS is a distributed file system which provides redundant storage space for files having huge sizes. It is used for storing files which are in the range of terabytes to petabytes. To learn more about world’s most reliable storage layer follow this HDFS introductory guide. Lets continue with Hadoop HDFS Commands.

Before working with HDFS you need to Deploy Hadoop, follow this guide to Install and configure Hadoop.

Hadoop Quiz

Get the most demanding skills of IT Industry - Learn Hadoop

3. Hadoop HDFS Commands

In This section of Hadoop HDFS command tutorial top, 10 HDFS commands are discussed below along with their usage, description, and examples. Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop shell commands are invoked by the bin/hdfs script.

3.1. version

Hadoop HDFS version Command Usage

version

Hadoop HDFS version Command Example

hdfs dfs version

Hadoop HDFS version Command Description

This Hadoop command prints the Hadoop version

3.2. mkdir

Hadoop HDFS mkdir Command Usage

mkdir <path>

Hadoop HDFS mkdir Command Example

hdfs dfs -mkdir /user/dataflair/dir1

Hadoop HDFS mkdir Command Description

This HDFS command takes path URI’s as an argument and creates directories.

Creates any parent directories in path that are missing (e.g., mkdir -p in Linux).

Learn various features of Hadoop HDFS from this HDFS features guide.

3.3. ls

Hadoop HDFS ls Command Usage

ls <path>

Hadoop HDFS ls Command Example

hdfs dfs -ls /user/dataflair/dir1

Hadoop HDFS ls Commnad Description

This Hadoop HDFS ls command displays a list of the contents of a directory specified by path provided by the user, showing the names, permissions, owner, size and modification date for each entry.

Hadoop HDFS ls Command Example

hdfs dfs -ls -R

Hadoop HDFS ls Description

This Hadoop fs command behaves like -ls, but recursively displays entries in all subdirectories of a path.

3.4. put

Hadoop HDFS put Command Usage

put <localSrc> <dest>

Hadoop HDFS put Command Example

hdfs dfs -put /home/dataflair/Desktop/sample /user/dataflair/dir1

Hadoop HDFS put Command Description

This hadoop basic command copies the file or directory from the local file system to the destination within the DFS.

Learn Internals of HDFS Data Write Pipeline and File write execution flow.

3.5. copyFromLocal

Hadoop HDFS copyFromLocal Command Usage

copyFromLocal <localSrc> <dest>

Hadoop HDFS copyFromLocal Command Example

hdfs dfs -copyFromLocal /home/dataflair/Desktop/sample /user/dataflair/dir1

Hadoop HDFS copyFromLocal Command Description

This hadoop shell command is similar to put command, but the source is restricted to a local file reference.

Learn Internals of HDFS Data Read Operation, How Data flows in HDFS while reading the file.

Any Doubt yet in Hadoop HDFS Commands? Please Comment.

3.6. get

Hadoop HDFS get Command Usage

get [-crc] <src> <localDest>

Hadoop HDFS get Command Example

hdfs dfs -get /user/dataflair/dir2/sample /home/dataflair/Desktop

Hadoop HDFS get Command Description
This HDFS fs command copies the file or directory in HDFS identified by the source to the local file system path identified by local destination.

Hadoop HDFS get Command Example
hdfs dfs -getmerge /user/dataflair/dir2/sample /home/dataflair/Desktop

Hadoop HDFS get Command Description
This HDFS basic command retrieves all files that match to the source path entered by the user in HDFS, and creates a copy of them to one single, merged file in the local file system identified by local destination.

Hadoop HDFS get Command Example

hadoop fs -getfacl /user/dataflair/dir1/sample
hadoop fs -getfacl -R /user/dataflair/dir1

Hadoop HDFS get Command Description

This Apache Hadoop command shows the Access Control Lists (ACLs) of files and directories. If a directory contains a default ACL, then getfacl also displays the default ACL.

Options :

-R: It displays a list of all the ACLs of all files and directories recursively.
path: File or directory to list.

Hadoop HDFS get Command Example

hadoop fs -getfattr -d /user/dataflair/dir1/sample

Hadoop HDFS get Command Description

This HDFS file system command displays if there is any extended attribute names and values for a file or directory.

Options:

-R: It recursively lists the attributes for all files and directories.
-n name: It displays the named extended attribute value.
-d: It displays all the extended attribute values associated with the pathname.
-e encoding: Encodes values after extracting them. The valid converted coded forms are “text”, “hex”, and “base64”. All the values encoded as text strings are with double quotes (” “), and prefix 0x and 0s are used for all the values which are converted and coded as hexadecimal and base64.
path: The file or directory.

Learn: Rack Awareness, High Availability

3.7. copyToLocal

Hadoop HDFS copyToLocal Command Usage

copyToLocal <src> <localDest>

Hadoop HDFS copyToLocal Command Example

hdfs dfs -copyToLocal /user/dataflair/dir1/sample /home/dataflair/Desktop

Hadoop HDFS copyToLocal Description
Similar to get command, only the difference is that in this the destination is restricted to a local file reference.

3.8. cat

Hadoop HDFS cat Command Usage

cat <file-name>

Hadoop HDFS cat Command Example

hdfs dfs -cat /user/dataflair/dir1/sample

Hadoop HDFS cat Command Description

This Hadoop fs shell command displays the contents of the filename on console or stdout.

3.9. mv

Hadoop HDFS mv Command Usage

mv <src> <dest>

Hadoop HDFS mv Command Example

hadoop fs -mv /user/dataflair/dir1/purchases.txt /user/dataflair/dir2

Hadoop HDFS mv Command Description

This basic HDFS command moves the file or directory indicated by the source to destination, within HDFS.

Learn: HDFS Disk Balancer and Erasure coding

3.10. cp

Hadoop HDFS cp Command Usage

cp <src> <dest>

Hadoop HDFS cp Command Example

hadoop fs -cp /user/dataflair/dir2/purchases.txt /user/dataflair/dir1

Hadoop HDFS cp Command Description

This Hadoop File system shell command copies the file or directory identified by the source to destination, within HDFS.

So this was all on Hadoop HDFS Commands. Hope you like it.

4. What’s Next

In case of any queries or feedback regarding Hadoop HDFS Commands feel free to let us know it in the comment section and we will get back to you.

15 Responses

  1. Nandhini says:

    Comment part II III IV are not accesible. have you removed those pages

    • Ismail Basha says:

      No,
      Part II III IV are working…
      please check your browser

      • Data Flair says:

        Thank you, Ismail, for checking the links and helping Nandhini. I hope you read the complete Hadoop HDFS Command Tutorial. Try our more Hadoop articles for better learning and keep helping others.
        Good luck.

    • Data Flair says:

      Hii Nandhini,
      Please check your browser or internet connectivity once. All links are fine. Try again.
      Still, if you find any problem, please let us know.

  2. Jitendra says:

    what is the major difference between copy from local and copy to local…????

    • Raheem Mohammed says:

      copyFromLocal lets you copy Local File system to HDFS simlarly, copyToLocal lets you copy from HDFS to LFS. Hope that answers

    • pooja says:

      the major diffrence is when you use copyFromLocal it will copy file from your local machine to HDFS architecture and when you use copyToLocal it will your file from HDFS architecture and place it on local machine

  3. rishabh says:

    diff b/w put and copyFromLocal in cammand

  4. rishabh says:

    What is the difference between hadoop hdfs put and copyFromLocal in command

    • lklk says:

      no difference,both are same

      • Babu Venkat says:

        If both are same means. why two commands?

        • Gopi says:

          we can mention multiple sourceDir in put command but in copyFromLocal we can not give more than one source dir ..

        • Ankur says:

          put command basically from linux origin and it is similar to the copyFromLocal , but there is small diff. suppose you copying any from from local to hdfs then and somehow during the copying the data task gets failed at 95% . if you are using the copyFromLocal, it 95% data will be available into the hdfs you only need to copy rest 5 % , while in put command you must copy entire data again.
          Hope this make sense to you

  5. Manoj Bhardwaj says:

    Need bit more description on each command. whatever is provided as of now is not sufficient for clarification.

  6. raju kumar says:

    how to install any package in hadoop

Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.