Top 10 Hadoop HDFS Commands with Examples and Usage 1


1. Hadoop HDFS Commands

In this tutorial, we are going to learn the most important and frequently used Hadoop HDFS commands with the help of which we will be able to perform HDFS file operations like copying the file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. In this Hadoop Commands tutorial we have mentioned the most frequently used HDFS commands.

2. Hadoop HDFS Commands: Introduction

Hadoop HDFS is a distributed file system which provides redundant storage space for files having huge sizes. It is used for storing files which are in the range of terabytes to petabytes. To learn more about world’s most reliable storage layer follow this HDFS introductory guide. Lets continue with Hadoop HDFS Commands.

Hadoop HDFS commands

Hadoop HDFS Commands

Before working with HDFS you need to Deploy Hadoop, follow this guide to Install and configure Hadoop.

3. Hadoop HDFS Commands

In This section of Hadoop HDFS command tutorial top, 10 HDFS commands are discussed below along with their usage, description, and examples. Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop shell commands are invoked by the bin/hdfs script.

3.1. version

Hadoop HDFS version Command Usage

version

Hadoop HDFS version Command Example

hdfs dfs version

Hadoop HDFS version Command Description
This Hadoop command prints the Hadoop version

3.2. mkdir

Hadoop HDFS mkdir Command Usage

mkdir <path>

Hadoop HDFS mkdir Command Example

hdfs dfs -mkdir /user/dataflair/dir1

Hadoop HDFS mkdir Command Description
This HDFS command takes path URI’s as an argument and creates directories.
Creates any parent directories in path that are missing (e.g., mkdir -p in Linux).

Learn various features of Hadoop HDFS from this HDFS features guide.

3.3. ls

Hadoop HDFS ls Command Usage

ls <path>

Hadoop HDFS ls Command Example

hdfs dfs -ls /user/dataflair/dir1

Hadoop HDFS ls Commnad Description
This Hadoop HDFS ls command displays a list of the contents of a directory specified by path provided by the user, showing the names, permissions, owner, size and modification date for each entry.

Hadoop HDFS ls Command Example

hdfs dfs -ls -R

Hadoop HDFS ls Description
This Hadoop fs command behaves like -ls, but recursively displays entries in all subdirectories of a path.

3.4. put

Hadoop HDFS put Command Usage

put <localSrc> <dest>

Hadoop HDFS put Command Example

hdfs dfs -put /home/dataflair/Desktop/sample /user/dataflair/dir1

Hadoop HDFS put Command Description
This hadoop basic command copies the file or directory from the local file system to the destination within the DFS.

Learn Internals of HDFS Data Write Pipeline and File write execution flow.

3.5. copyFromLocal

Hadoop HDFS copyFromLocal Command Usage

copyFromLocal <localSrc> <dest>

Hadoop HDFS copyFromLocal Command Example

hdfs dfs -copyFromLocal /home/dataflair/Desktop/sample /user/dataflair/dir1

Hadoop HDFS copyFromLocal Command Description
This hadoop shell command is similar to put command, but the source is restricted to a local file reference.

Learn Internals of HDFS Data Read Operation, How Data flows in HDFS while reading the file.

Any Doubt yet in Hadoop HDFS Commands? Please Comment.

3.6. get

Hadoop HDFS get Command Usage

get [-crc] <src> <localDest>

Hadoop HDFS get Command Example

hdfs dfs -get /user/dataflair/dir2/sample /home/dataflair/Desktop

Hadoop HDFS get Command Description
This HDFS fs command copies the file or directory in HDFS identified by the source to the local file system path identified by local destination.

Hadoop HDFS get Command Example

hdfs dfs -getmerge /user/dataflair/dir2/sample /home/dataflair/Desktop

Hadoop HDFS get Command Description
This HDFS basic command retrieves all files that match to the source path entered by the user in HDFS, and creates a copy of them to one single, merged file in the local file system identified by local destination.

Hadoop HDFS get Command Example

hadoop fs -getfacl /user/dataflair/dir1/sample
hadoop fs -getfacl -R /user/dataflair/dir1

Hadoop HDFS get Command Description
This Apache Hadoop command shows the Access Control Lists (ACLs) of files and directories. If a directory contains a default ACL, then getfacl also displays the default ACL.
Options :
-R: It displays a list of all the ACLs of all files and directories recursively.
path: File or directory to list.

Hadoop HDFS get Command Example

hadoop fs -getfattr -d /user/dataflair/dir1/sample

Hadoop HDFS get Command Description
This HDFS file system command displays if there is any extended attribute names and values for a file or directory.
Options:
-R: It recursively lists the attributes for all files and directories.
-n name: It displays the named extended attribute value.
-d: It displays all the extended attribute values associated with the pathname.
-e encoding: Encodes values after extracting them. The valid converted coded forms are “text”, “hex”, and “base64”. All the values encoded as text strings are with double quotes (” “), and prefix 0x and 0s are used for all the values which are converted and coded as hexadecimal and base64.
path: The file or directory.

Learn: Rack Awareness, High Availability

3.7. copyToLocal

Hadoop HDFS copyToLocal Command Usage

copyToLocal <src> <localDest>

Hadoop HDFS copyToLocal Command Example

hdfs dfs -copyToLocal /user/dataflair/dir1/sample /home/dataflair/Desktop

Hadoop HDFS copyToLocal Description
Similar to get command, only the difference is that in this the destination is restricted to a local file reference.

3.8. cat

Hadoop HDFS cat Command Usage

cat <file-name>

Hadoop HDFS cat Command Example

hdfs dfs -cat /user/dataflair/dir1/sample

Hadoop HDFS cat Command Description
This Hadoop fs shell command displays the contents of the filename on console or stdout.

3.9. mv

Hadoop HDFS mv Command Usage

mv <src> <dest>

Hadoop HDFS mv Command Example

hadoop fs -mv /user/dataflair/dir1/purchases.txt /user/dataflair/dir2

Hadoop HDFS mv Command Description
This basic HDFS command moves the file or directory indicated by the source to destination, within HDFS.

Learn: HDFS Disk Balancer and Erasure coding

3.10. cp

Hadoop HDFS cp Command Usage

cp <src> <dest>

Hadoop HDFS cp Command Example

hadoop fs -cp /user/dataflair/dir2/purchases.txt /user/dataflair/dir1

Hadoop HDFS cp Command Description
This Hadoop File system shell command copies the file or directory identified by the source to destination, within HDFS.

So this was all on Hadoop HDFS Commands. Hope you like it.

4. What’s Next

In case of any queries or feedback regarding Hadoop HDFS Commands feel free to let us know it in the comment section and we will get back to you.


Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “Top 10 Hadoop HDFS Commands with Examples and Usage