{"id":53192,"date":"2019-04-02T11:01:56","date_gmt":"2019-04-02T05:31:56","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=53192"},"modified":"2021-08-25T22:32:33","modified_gmt":"2021-08-25T17:02:33","slug":"commands-in-hadoop","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/","title":{"rendered":"10 Most Frequently Used Hadoop Commands With Examples"},"content":{"rendered":"

In this blog, we are going to explore most frequently used commands in Hadoop. These commands aid in performing various HDFS file operations<\/strong><\/a>. These include copying a file, moving a file, showing the contents of the file, creating directories, etc. So let us begin with an introduction and then we will see different commands in Hadoop with examples.<\/span><\/p>\n

\"10<\/a><\/p>\n

Top Hadoop Commands<\/span><\/h2>\n

Hadoop stores petabytes of data using HDFS. HDFS is a distributed file system<\/strong><\/a> which stores structured to unstructured data. It provides redundant storage for files having humongous size. There are various commands to perform different file operations. Let us take a look at some of the important Hadoop commands.<\/span><\/p>\n

List of Hadoop Commands<\/strong><\/p>\n

1. version<\/h3>\n

Command Name:<\/span> version<\/b><\/p>\n

Command Usage: version<\/span><\/p>\n

Example:<\/strong> <\/span><\/p>\n

hadoop version<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> Shows the version of hadoop installed.<\/span><\/p>\n
You must read about Hadoop Distributed Cache<\/strong><\/a><\/p>\n
2. mkdir<\/h3>\n
Command Name:<\/span> mkdir<\/b><\/p>\n
Command Usage: mkdir <path><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -mkdir \/user\/dataflair\/dir1<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This command takes the <path> as an argument and creates the directory.<\/span><\/p>\n
3. Is<\/h3>\n
Command Name:<\/span> ls<\/b><\/p>\n
Command Usage: ls <path><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -ls \/user\/dataflair<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This command displays the contents of the directory specified by <path>. It shows the name, permissions, owner, size and modification date of each entry.<\/span><\/p>\n
Do you know about Hadoop Automatic Failover?<\/strong><\/a><\/p>\n
Second Example:<\/strong> <\/span><\/p>\n
hdfs dfs -ls -R \/user<\/pre>\n
<insert image ls-R.png><\/span><\/p>\n
Description:<\/strong> This command behaves like ls but displays entries in all the sub-directories recursively<\/span><\/p>\n
4. put<\/h3>\n
Command Name:<\/span> put<\/b><\/p>\n
Command Usage: put <localsrc> <dest><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -put \/home\/sample.txt \/user\/dataflair\/dir1<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This command copies the file in the local filesystem to the file in DFS.<\/p>\n
5. copyFrom Local<\/h3>\n
Command Name:\u00a0<\/span>copyFrom Local<\/b><\/p>\n
Command Usage: copyFrom Local <localsrc> <dest><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -copyFromLocal \/home\/sample \/user\/dataflair\/dir1<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This command is similar to put command. But the source should refer to local file.<\/p>\n
6. get<\/h3>\n
Command Name:<\/span>get<\/b><\/p>\n
Command Usage: get <src> <localdest><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -get \/user\/dataflair\/dir1 \/home<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This Hadoop shell command<\/strong> copies the file in HDFS identified by <src> to file in local file system identified by <localdest><\/p>\n
Second Example:<\/strong> <\/span><\/p>\n
hdfs dfs -getmerge \/user\/dataflair\/dir1\/sample.txt \/user\/dataflair\/dir2\/sample2.txt \/home\/sample1.txt<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This HDFS command<\/a><\/strong> retrieves all files in the source path entered by the user in HDFS. And merges them into one single file created in the local file system identified by local destination.<\/p>\n
Third Example:<\/strong> <\/span><\/p>\n
hadoop fs \u2013getfacl \/user\/dataflair\/dir1<\/pre>\n
\"commands<\/a><\/p>\n
Fourth Example:<\/strong><\/span><\/p>\n
hadoop fs \u2013getfacl -R \/user\/dataflair\/dir1<\/pre>\n
\"commands<\/a><\/p>\n
Description:<\/strong> This Hadoop command shows the Access Control Lists (ACLs) of files and directories. This command displays default ACL if the directory contains the same.<\/p>\n
Options : -R: It recursively displays a list of all the ACLs of all files and directories.<\/span><\/p>\n
Fifth Example:<\/strong> <\/span><\/p>\n
hadoop fs \u2013getfattr \u2013d \/user\/dataflair\/dir1<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This HDFS command displays if there is any extended attribute names and values for the specified file or directory. <\/span><\/p>\n
Options:<\/span>-R: It lists the attributes for all files and directories recursively. -n name: It shows the named extended attribute value. -d: It shows all the extended attribute values associated with the pathname. -e encoding: Encodes values after extracting them. The valid coded forms that are \u201ctext\u201d, \u201chex\u201d, and \u201cbase64\u201d. The values which are encoded as text strings gets enclosed with double quotes (\u201d \u201c). It uses prefix 0x for hexadecimal conversion. And 0s for all the values which gets coded as base64.<\/span><\/p>\n
Don’t forget to learn about Hadoop Schedulers\u00a0<\/a><\/strong><\/p>\n
7. copyToLocal<\/h3>\n
Command Name:\u00a0<\/span>copyToLocal<\/b>
\n<\/span><\/p>\n
Command Usage: copyToLocal <src> <localdest><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -copyToLocal \/user\/dataflair\/dir1 \/home<\/pre>\n
\"commands<\/a><\/span><\/p>\n
Description:<\/strong> It is similar to get command. Only the difference is that in this the destination of copied file should refer to a local file.<\/span><\/p>\n
8. cat<\/h3>\n
Command Name:<\/span> cat<\/b><\/p>\n
Command Usage: cat <file-name><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -cat \/user\/dataflair\/dir1\/sample.txt<\/pre>\n
\"commands<\/a><\/span><\/p>\n
Description:<\/strong> This Hadoop shell command<\/a> displays the contents of file on console or stdout.<\/span><\/p>\n
9. mv<\/h3>\n
Command Name:<\/span> mv<\/b><\/p>\n
Command Usage: mv <src> <dest><\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -mv \/user\/dataflair\/dir1\/sample.txt \/user\/dataflair\/dir2<\/pre>\n
\"commands<\/a><\/p>\n
Description:<\/strong> This Hadoop shell command moves the file from the specified source to destination within HDFS.\u00a0<\/span><\/p>\n
Have a look at HDFS Disk Balancer<\/a><\/strong><\/p>\n
10. cp<\/h3>\n
Command Name:<\/span> cp<\/b><\/p>\n
Command Usage: cp <src> <dest><\/span>
\n<\/span><\/p>\n
Example:<\/strong> <\/span><\/p>\n
hdfs dfs -cp \/user\/dataflair\/dir2\/sample.txt \/user\/dataflair\/dir1<\/pre>\n
\"Commands<\/a><\/p>\n
Description:<\/strong> This Hadoop shell command copies the file or directory from given source to destination within HDFS.<\/span>
\n<\/span><\/p>\n
You must check 50 most asked Hadoop interview questions<\/strong><\/a><\/p>\n
Summary<\/span><\/h2>\n
There are many commands in \u201c$HADOOP_HOME\/bin\/hadoop fs\u201d other than what we have discussed in this tutorial. What we have covered are the frequently used basic commands to get started. If you are stuck somewhere then type the following: <\/span>
\n<\/span><\/p>\n
$HADOOP_HOME\/bin\/hadoop fs -help commandName<\/pre>\n
This will display a short usage summary of the command specified.<\/span><\/p>\n
Still, if you have any questions related to Hadoop Commands, ask in the comment section. We will definitely get back to you.
In this blog, we are going to explore most frequently used commands in Hadoop. These commands aid in performing various HDFS file operations. These include copying a file, moving a file, showing the contents...<\/p>\n","protected":false},"author":7,"featured_media":53231,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[19387,5231,19386],"class_list":["post-53192","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","tag-commands-in-hadoop","tag-hadoop-commands","tag-hadoop-commands-list"],"yoast_head":"\n10 Most Frequently Used Hadoop Commands With Examples - DataFlair<\/title>\n<meta name=\"description\" content=\"Explore the top 10 commands in Hadoop with their example and usage - version, copyfromLocal, copyToLocal, mkdir, Is, put, get etc\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"10 Most Frequently Used Hadoop Commands With Examples - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Explore the top 10 commands in Hadoop with their example and usage - version, copyfromLocal, copyToLocal, mkdir, Is, put, get etc\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-02T05:31:56+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-08-25T17:02:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2019\/03\/Commands-in-Hadoop.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"10 Most Frequently Used Hadoop Commands With Examples - DataFlair","description":"Explore the top 10 commands in Hadoop with their example and usage - version, copyfromLocal, copyToLocal, mkdir, Is, put, get etc","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/","og_locale":"en_US","og_type":"article","og_title":"10 Most Frequently Used Hadoop Commands With Examples - DataFlair","og_description":"Explore the top 10 commands in Hadoop with their example and usage - version, copyfromLocal, copyToLocal, mkdir, Is, put, get etc","og_url":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2019-04-02T05:31:56+00:00","article_modified_time":"2021-08-25T17:02:33+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2019\/03\/Commands-in-Hadoop.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"10 Most Frequently Used Hadoop Commands With Examples","datePublished":"2019-04-02T05:31:56+00:00","dateModified":"2021-08-25T17:02:33+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/"},"wordCount":745,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2019\/03\/Commands-in-Hadoop.jpg","keywords":["Commands in Hadoop","hadoop commands","hadoop Commands list"],"articleSection":["Hadoop Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/","url":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/","name":"10 Most Frequently Used Hadoop Commands With Examples - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2019\/03\/Commands-in-Hadoop.jpg","datePublished":"2019-04-02T05:31:56+00:00","dateModified":"2021-08-25T17:02:33+00:00","description":"Explore the top 10 commands in Hadoop with their example and usage - version, copyfromLocal, copyToLocal, mkdir, Is, put, get etc","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2019\/03\/Commands-in-Hadoop.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2019\/03\/Commands-in-Hadoop.jpg","width":1200,"height":628,"caption":"10 Most Frequently Used Hadoop Commands With Examples"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/commands-in-hadoop\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Hadoop Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hadoop\/"},{"@type":"ListItem","position":3,"name":"10 Most Frequently Used Hadoop Commands With Examples"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/53192","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=53192"}],"version-history":[{"count":4,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/53192\/revisions"}],"predecessor-version":[{"id":53258,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/53192\/revisions\/53258"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/53231"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=53192"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=53192"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=53192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}