\nTo learn more big data technologies like Apache Spark, follow this comprehensive guide.<\/a><\/p>\n
![\"Install](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg\")
<\/a>Install Hadoop 2 with YARN in Pseudo-Distributed Mode<\/p><\/div>\n
2. Setup and Install Hadoop 2 with Yarn<\/h2>\n
Follow the steps given below to install Hadoop 2 with Yarn on ubuntu os-<\/p>\n
2.1. Required Platforms to Install Hadoop 2 with Yarn on Ubuntu<\/h3>\n\n- Operating system:<\/strong> Ubuntu 14.04 or later, we can also use other Linux versions such as CentOS, Redhat, etc.<\/li>\n
- Hadoop:<\/strong> Cloudera Distribution for Apache Hadoop CDH5.x (we can also use Apache Hadoop 2.x)<\/li>\n<\/ul>\n
Note:\u00a0If you are using Windows\/Mac OS you can create a virtual machine and install Ubuntu using VMWare Player<\/a>, alternatively you can create a virtual machine and install Ubuntu using Oracle Virtual Box<\/a>.<\/p>\n2.2. Prerequisites to Install Hadoop 2 with Yarn<\/h3>\n
Necessary Software\u2019s to be installed before starting up to install Hadoop 2 with Yarn.<\/p>\n
I. Java 8 Installation<\/h4>\n
a. Install Python Software Properties<\/strong>
\nIn order to add the Java repositories, we need to download python-software-properties. Download and Installation of Python software properties can be easily executed by using the below command:<\/p>\n[php]sudo apt-get install python-software-properties[\/php]<\/p>\n
NOTE: After pressing \u201cEnter\u201d key. It will ask for your password because we are using \u201csudo\u201d command to provide root privileges for the installation. And in order to install or configure any software, we always need root privileges.<\/em><\/p>\nb. Add Repository<\/strong>
\nNow we will add a Repository. A repository is nothing but a collection of software for a Linux distribution on a server from where we can easily download and install software\u2019s using command \u201capt-get\u201d in Linux. So, now we will manually add a java repository so that Ubuntu can easily install the Java. Command to add a Java repository:<\/p>\n[php]sudo add-apt-repository ppa:webupd8team\/java[\/php]<\/p>\n
Press \u201cEnter\u201d.<\/p>\n
c. Update the source list<\/strong>
\nThe source list is basically a location where Ubuntu download and install the software and also updates the software. So whenever we install a new package\/software or update any previously installed software then it is recommended that we should update the source list periodically so that changes made can easily get saved and we can easily use the software. So in order to update source list, we need to use the following command:<\/p>\n[php]sudo apt-get update[\/php]<\/p>\n
When you run the above command Ubuntu will update its source list.<\/p>\n
d. Install Java<\/strong>
\nHadoop framework<\/a><\/strong> is written in Java. Hence it requires Java to be installed. So to download and install Java we need to use the below command:<\/p>\n[php]sudo apt-get install oracle-java8-installer[\/php]<\/p>\n
When you will press enter it will start downloading and installing Java.
\nTo check the whether the Java is installed successfully or not and to check the version of your Java type the below command terminal:<\/p>\n
[php]java -version[\/php]<\/p>\n
II. Configure SSH to Install Hadoop with Yarn<\/h4>\n
SSH means secured shell which is used for the remote login. Means we can login to a remote machine using SSH. Hence SSH setup is required to allow password-less login for the Hadoop<\/a> <\/strong>user from machines in the cluster. Password-less SSH setup is required for remote script invocation. Automatically remotely master will start the demons on slaves.<\/p>\na. Install Open SSH Server-Client<\/strong>
\nThese SSH tools will be used for the remote login.<\/p>\n[php]sudo apt-get install openssh-server openssh-client[\/php]<\/p>\n
b. Generate Key Pairs<\/strong>
\nNow will generate public and private key pair, which is used for authentication.<\/p>\n[php]ssh-keygen -t rsa -P “”[\/php]<\/p>\n
Note: It will ask \u201cEnter the file in which to save the key (\/home\/hdadmin\/.ssh\/id_rsa):\u201d You don\u2019t need to make any changes just press \u201cEnter\u201d. Now the file will be available in\u201c.ssh\u201d path. You can check the default path by \u201c$ls .ssh\/\u201d command and there you will see that two files are created \u201cid_rsa\u201d which is a private key and \u201cid_rsa.pub\u201d which is a public key.<\/p>\n
c. Configure passwordless SSH<\/strong>
\nWe need to copy the contents of \u201cid_rsa.pub\u201d into the \u201cauthorized_keys\u201d file. So this can be done by using the command:<\/p>\n[php]cat $HOME\/.ssh\/id_rsa.pub >> $HOME\/.ssh\/authorized_keys[\/php]<\/p>\n
d. Check by SSH to localhost<\/strong>
\n[php]ssh localhost[\/php]<\/p>\nAs previously we had configured passwordless SSH hence it will not ask for any password and thus we can easily get logged into localhost.<\/p>\n
2.3. Getting Started with Hadoop Installation<\/h3>\n
Step by step procedure for installation of Hadoop-2.6.2 in pseudo distributed mode is given below.<\/p>\n
I. Downloading Hadoop<\/h4>\n
We can download Hadoop from the below link.
\n[php]http:\/\/www.eu.apache.org\/dist\/hadoop\/common\/hadoop-2.6.2\/hadoop-2.6.2.tar.gz[\/php]
\nAfter downloading Hadoop, we need to copy it to our home directory.<\/p>\n
II. Untar Tar-ball<\/h4>\n
Now we need to extract all the contents of compressed Hadoop file. For this we will use Tar command:<\/p>\n
[php]tar xzf hadoop-2.6.2.tar.gz[\/php]<\/p>\n
Note: All the libraries, scripts, configuration files, etc. are available in HADOOP_HOME directory (the extracted directory is called as HADOOP_HOME. e.g. hadoop-2.6.0-cdh5.5.1)<\/em><\/p>\nIII. Setup Configuration to Install Hadoop with Yarn<\/h4>\n
Making changes in Setup Configuration files for successful installation and running of Hadoop services:<\/p>\n
a. Edit .bashrc<\/strong>
\nNow firstly we need to edit environment file \u201c.bashrc\u201d which is present in our home directory. We can identify and edit this file with the command: $ nano -\/.bashrc\u201d. In this file we need to add the below contents at the end of this file:<\/p>\n[php]export JAVA_HOME=\/usr\/lib\/jvm\/java-8-oracle\/
\nexport HADOOP_HOME=home\/dataflair\/hadoop-2.6.2
\nexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME\/lib\/native
\nexport HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME\/lib
\nexport HADOOP_PREFIX=”\/home\/hdadmin\/hadoop-2.6.2″
\nexport PATH=$PATH:$HADOOP_PREFIX\/bin
\nexport PATH=$PATH:$HADOOP_PREFIX\/sbin
\nexport HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
\nexport HADOOP_COMMON_HOME=${HADOOP_PREFIX}
\nexport HADOOP_HDFS_HOME=${HADOOP_PREFIX}
\nexport YARN_HOME=${HADOOP_PREFIX}
\n[\/php]<\/p>\n
Note: Make to enter the correct path. “\/home\/dataflair\/hadoop-2.6.2\u201d this is my home directory path. By \u201c$pwd\u201d command you can get your home directory path.<\/em><\/p>\nOn adding the above contents in the .bashrc file we need to save it. In order to save it press \u201cCtrl+X\u201d.
\nNote: After above step restarts the terminal so that all the environment variables will come into effect<\/em><\/p>\nb. Edit hadoop-env.sh<\/strong>
\nNow we need to edit configuration \u201chadoop-env.sh” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) and here we have to provide root path of Java installation. We have to provide Java installation path in \u201chadoop-env.sh” file under the heading \u201cJAVA_HOME\u201d. This can be done by executing steps:<\/p>\n[php]dataflair@ubuntu:~$cd hadoop-2.6.2\/
\ndataflair@ubuntu:~\/hadoop-2.6.2$ cd etc\/hadoop
\ndataflair@ubuntu:~\/hadoop-2.6.2\/etc\/hadoop$ nano hadoop-env.sh[\/php]<\/p>\n
Here set JAVA_HOME as:<\/p>\n
[php]export JAVA_HOME=<path-to-the-root-of-your-Java-installation> (eg: \/usr\/lib\/jvm\/java-8-oracle\/)[\/php]<\/p>\n
On adding the above contents in hadoop-env.sh file we need to save it. In order to save it press \u201cCtrl+X\u201d.
\nNote: \u201c\/usr\/lib\/jvm\/java-8-oracle\/\u201d is default Java path. If you had made any changes in your java path then enter that path here.<\/em><\/p>\nc. Edit core-site.xml<\/strong>
\nNow we need to edit configuration \u201dcore-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n[php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano core-site.xml[\/php]<\/p>\n
And here we have to add below entries between present at the end of this file:<\/p>\n
[php]<property>
\n<name>fs.defaultFS<\/name>
\n<value>hdfs:\/\/localhost:9000<\/value>
\n<\/property>
\n<property>
\n<name>hadoop.tmp.dir<\/name>
\n<value>\/home\/dataflair\/hdata<\/value>
\n<\/property>[\/php]<\/p>\n
Note: \u201c\/home\/dataflair\/hdata\u201d is my location where I have read-write privileges. So kindly enter the location of your path where you have Read Write privileges.<\/em><\/p>\nOn adding the above contents in core-site.xml file we need to save it. In order to save it press \u201cCtrl+X\u201d.<\/p>\n
d. Edit hdfs-site.xml<\/strong>
\nNow we need to edit configuration \u201dhdfs-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n[php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano hdfs-site.xml[\/php]<\/p>\n
And here we have to add below entries between present at the end of this file:<\/p>\n
[php]<property>
\n<name>dfs.replication<\/name>
\n<value>1<\/value>
\n<\/property>[\/php]<\/p>\n
On adding the above contents in hdfs-site.xml file we need to save it. In order to save it press \u201cCtrl+X\u201d.<\/p>\n
e. Edit mapred-site.xml<\/strong>
\nNow we need to edit configuration \u201dmapred-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n[php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano mapred-site.xml[\/php]<\/p>\n
After pressing \u201cEnter\u201d we will see that a new blank file named mapred-site.xml gets generate. This happens because there is no such file named mapred-site.xml is present in our home directory. Instead of that, there exists a template file available mapred-site.xml.template. So in order to edit mapred-site.xml file, firstly we need to create a copy of mapred-site.xml.template file.<\/p>\n
Command to create a copy of mapred-site.xml.template file is:<\/p>\n
[php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ cp mapred-site.xml.template mapred-site.xml[\/php]<\/p>\n
After creating a copy of mapred-site.xml.template file, we can now be able to edit mapred-site.xml file with the following command:<\/p>\n
[php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano mapred-site.xml[\/php]<\/p>\n
And here we need to add below entries between present at the end of this file:<\/p>\n
[php]<property>
\n<name>mapreduce.framework.name<\/name>
\n<value>yarn<\/value>
\n<\/property>[\/php]<\/p>\n
On adding the above contents in mapred-site.xml file we need to save it. In order to save it press \u201cCtrl+X\u201d.<\/p>\n
f. Edit yarn-site.xml<\/strong>
\nNow we need to edit configuration \u201dyarn-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n[php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano yarn-site.xml[\/php]<\/p>\n
And here we need to add below entries between present at the end of this file:<\/p>\n
[php]<property>
\n<name>yarn.nodemanager.aux-services<\/name>
\n<value>mapreduce_shuffle<\/value>
\n<\/property>
\n<property>
\n<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<\/name>
\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>
\n<\/property>[\/php]<\/p>\n
Now we are completed with the installation procedure of Hadoop-2.6.2 in pseudo distributed mode. Now we will start Hadoop services by following steps.<\/p>\n
For deep dive into Hadoop Yarn, resource manager follow this yarn tutorial.<\/a><\/p>\nIV. Start the Hadoop Cluster<\/h4>\n
a. Format the namenod<\/strong><\/p>\nBefore starting the Hadoop services we need to format namenode. In order to do so, we will use the below command:<\/p>\n
[php]dataflair@ubuntu:~$ hdfs namenode -format[\/php]<\/p>\n
NOTE: This activity should be done only once when you install Hadoop<\/a> if you repeat it you lose all your data and different namespaceId will be assigned to namenode. If namenode and datanode have different id datanode daemon will not be started when to start the daemons.<\/em><\/p>\nb. Start HDFS Daemons:<\/strong>
\n[php]dataflair@ubuntu:~$ start-dfs.sh[\/php]<\/p>\nc. Start YARN Daemons:<\/strong>
\n[php]dataflair@ubuntu:~$ start-yarn.sh[\/php]<\/p>\nd. Check the status of services:<\/strong>
\n[php]dataflair@ubuntu:~$ jps[\/php]<\/p>\nNameNode
\nDataNode
\nResourceManager
\nNodeManager<\/p>\n
If you are getting above output after running jps command, it means Hadoop has been installed successfully. Now you can perform HDFS<\/a><\/strong> operations (Follow HDFS commands guide<\/a>) and MapReduce<\/a> operations.<\/p>\nIf you like this tutorial on How to install Hadoop with Yran on Ubuntu or if you have any queries regarding the tutorial, do let us know it in the comment section below and we will get back to you.
\nSee Also-<\/strong><\/p>\n\n- Hadoop 2.x vs Hadoop 3.x<\/a><\/li>\n
- Hadoop vs Spark vs Flink<\/a><\/li>\n<\/ul>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
1.\u00a0Install Hadoop 2 with YARN Tutorial: Objective In this tutorial to install Hadoop 2 with Yarn, we will learn how to setup and run Apache Hadoop 2 with yarn on a single-node on Ubuntu...<\/p>\n","protected":false},"author":6,"featured_media":43201,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[782,1907,1971,2877,5186,5192,5201,5267,5329,5341,8140,8303,11664],"class_list":["post-353","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","tag-apache-hadoop","tag-big-data","tag-big-data-training","tag-configure-hadoop","tag-hadoop","tag-hadoop-2-x","tag-hadoop-admin","tag-hadoop-installation","tag-hadoop-setup","tag-hadoop-training","tag-learn-hadoop","tag-linux","tag-run-hadoop"],"yoast_head":"\n
Install Hadoop 2 with YARN in Pseudo-Distributed Mode - DataFlair<\/title>\n<meta name=\"description\" content=\"Install Hadoop 2 with Yarn in pseudo distributed mode Ubuntu, java installation,how to configure SSH, start Hadoop cluster,How to start Hadoop daemons\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Install Hadoop 2 with YARN in Pseudo-Distributed Mode - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Install Hadoop 2 with Yarn in pseudo distributed mode Ubuntu, java installation,how to configure SSH, start Hadoop cluster,How to start Hadoop daemons\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-06-13T07:32:33+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-11-21T11:03:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Install Hadoop 2 with YARN in Pseudo-Distributed Mode - DataFlair","description":"Install Hadoop 2 with Yarn in pseudo distributed mode Ubuntu, java installation,how to configure SSH, start Hadoop cluster,How to start Hadoop daemons","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/","og_locale":"en_US","og_type":"article","og_title":"Install Hadoop 2 with YARN in Pseudo-Distributed Mode - DataFlair","og_description":"Install Hadoop 2 with Yarn in pseudo distributed mode Ubuntu, java installation,how to configure SSH, start Hadoop cluster,How to start Hadoop daemons","og_url":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2016-06-13T07:32:33+00:00","article_modified_time":"2018-11-21T11:03:27+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Install Hadoop 2 with YARN in Pseudo-Distributed Mode","datePublished":"2016-06-13T07:32:33+00:00","dateModified":"2018-11-21T11:03:27+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/"},"wordCount":2121,"commentCount":6,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg","keywords":["apache hadoop","big data","big data training","configure hadoop","hadoop","Hadoop 2.x","hadoop admin","hadoop installation","hadoop setup","hadoop training","learn hadoop","linux","run hadoop"],"articleSection":["Hadoop Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/","url":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/","name":"Install Hadoop 2 with YARN in Pseudo-Distributed Mode - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg","datePublished":"2016-06-13T07:32:33+00:00","dateModified":"2018-11-21T11:03:27+00:00","description":"Install Hadoop 2 with Yarn in pseudo distributed mode Ubuntu, java installation,how to configure SSH, start Hadoop cluster,How to start Hadoop daemons","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Install-Hadoop-2-with-Yarn-01.jpg","width":1200,"height":628,"caption":"Install Hadoop 2 with YARN in Pseudo-Distributed Mode"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/hadoop-2-with-yarn\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Hadoop Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hadoop\/"},{"@type":"ListItem","position":3,"name":"Install Hadoop 2 with YARN in Pseudo-Distributed Mode"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/353","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=353"}],"version-history":[{"count":7,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/353\/revisions"}],"predecessor-version":[{"id":43202,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/353\/revisions\/43202"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/43201"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=353"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=353"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=353"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Note:\u00a0If you are using Windows\/Mac OS you can create a virtual machine and install Ubuntu using VMWare Player<\/a>, alternatively you can create a virtual machine and install Ubuntu using Oracle Virtual Box<\/a>.<\/p>\n Necessary Software\u2019s to be installed before starting up to install Hadoop 2 with Yarn.<\/p>\n a. Install Python Software Properties<\/strong> [php]sudo apt-get install python-software-properties[\/php]<\/p>\n NOTE: After pressing \u201cEnter\u201d key. It will ask for your password because we are using \u201csudo\u201d command to provide root privileges for the installation. And in order to install or configure any software, we always need root privileges.<\/em><\/p>\n b. Add Repository<\/strong> [php]sudo add-apt-repository ppa:webupd8team\/java[\/php]<\/p>\n Press \u201cEnter\u201d.<\/p>\n c. Update the source list<\/strong> [php]sudo apt-get update[\/php]<\/p>\n When you run the above command Ubuntu will update its source list.<\/p>\n d. Install Java<\/strong> [php]sudo apt-get install oracle-java8-installer[\/php]<\/p>\n When you will press enter it will start downloading and installing Java. [php]java -version[\/php]<\/p>\n SSH means secured shell which is used for the remote login. Means we can login to a remote machine using SSH. Hence SSH setup is required to allow password-less login for the Hadoop<\/a> <\/strong>user from machines in the cluster. Password-less SSH setup is required for remote script invocation. Automatically remotely master will start the demons on slaves.<\/p>\n a. Install Open SSH Server-Client<\/strong> [php]sudo apt-get install openssh-server openssh-client[\/php]<\/p>\n b. Generate Key Pairs<\/strong> [php]ssh-keygen -t rsa -P “”[\/php]<\/p>\n Note: It will ask \u201cEnter the file in which to save the key (\/home\/hdadmin\/.ssh\/id_rsa):\u201d You don\u2019t need to make any changes just press \u201cEnter\u201d. Now the file will be available in\u201c.ssh\u201d path. You can check the default path by \u201c$ls .ssh\/\u201d command and there you will see that two files are created \u201cid_rsa\u201d which is a private key and \u201cid_rsa.pub\u201d which is a public key.<\/p>\n c. Configure passwordless SSH<\/strong> [php]cat $HOME\/.ssh\/id_rsa.pub >> $HOME\/.ssh\/authorized_keys[\/php]<\/p>\n d. Check by SSH to localhost<\/strong> As previously we had configured passwordless SSH hence it will not ask for any password and thus we can easily get logged into localhost.<\/p>\n Step by step procedure for installation of Hadoop-2.6.2 in pseudo distributed mode is given below.<\/p>\n We can download Hadoop from the below link. Now we need to extract all the contents of compressed Hadoop file. For this we will use Tar command:<\/p>\n [php]tar xzf hadoop-2.6.2.tar.gz[\/php]<\/p>\n Note: All the libraries, scripts, configuration files, etc. are available in HADOOP_HOME directory (the extracted directory is called as HADOOP_HOME. e.g. hadoop-2.6.0-cdh5.5.1)<\/em><\/p>\n Making changes in Setup Configuration files for successful installation and running of Hadoop services:<\/p>\n a. Edit .bashrc<\/strong> [php]export JAVA_HOME=\/usr\/lib\/jvm\/java-8-oracle\/ Note: Make to enter the correct path. “\/home\/dataflair\/hadoop-2.6.2\u201d this is my home directory path. By \u201c$pwd\u201d command you can get your home directory path.<\/em><\/p>\n On adding the above contents in the .bashrc file we need to save it. In order to save it press \u201cCtrl+X\u201d. b. Edit hadoop-env.sh<\/strong> [php]dataflair@ubuntu:~$cd hadoop-2.6.2\/ Here set JAVA_HOME as:<\/p>\n [php]export JAVA_HOME=<path-to-the-root-of-your-Java-installation> (eg: \/usr\/lib\/jvm\/java-8-oracle\/)[\/php]<\/p>\n On adding the above contents in hadoop-env.sh file we need to save it. In order to save it press \u201cCtrl+X\u201d. c. Edit core-site.xml<\/strong> [php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano core-site.xml[\/php]<\/p>\n And here we have to add below entries between present at the end of this file:<\/p>\n [php]<property> Note: \u201c\/home\/dataflair\/hdata\u201d is my location where I have read-write privileges. So kindly enter the location of your path where you have Read Write privileges.<\/em><\/p>\n On adding the above contents in core-site.xml file we need to save it. In order to save it press \u201cCtrl+X\u201d.<\/p>\n d. Edit hdfs-site.xml<\/strong> [php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano hdfs-site.xml[\/php]<\/p>\n And here we have to add below entries between present at the end of this file:<\/p>\n [php]<property> On adding the above contents in hdfs-site.xml file we need to save it. In order to save it press \u201cCtrl+X\u201d.<\/p>\n e. Edit mapred-site.xml<\/strong> [php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano mapred-site.xml[\/php]<\/p>\n After pressing \u201cEnter\u201d we will see that a new blank file named mapred-site.xml gets generate. This happens because there is no such file named mapred-site.xml is present in our home directory. Instead of that, there exists a template file available mapred-site.xml.template. So in order to edit mapred-site.xml file, firstly we need to create a copy of mapred-site.xml.template file.<\/p>\n Command to create a copy of mapred-site.xml.template file is:<\/p>\n [php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ cp mapred-site.xml.template mapred-site.xml[\/php]<\/p>\n After creating a copy of mapred-site.xml.template file, we can now be able to edit mapred-site.xml file with the following command:<\/p>\n [php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano mapred-site.xml[\/php]<\/p>\n And here we need to add below entries between present at the end of this file:<\/p>\n [php]<property> On adding the above contents in mapred-site.xml file we need to save it. In order to save it press \u201cCtrl+X\u201d.<\/p>\n f. Edit yarn-site.xml<\/strong> [php]dataflair@ubuntu:~\/ hadoop-2.6.2\/etc\/hadoop$ nano yarn-site.xml[\/php]<\/p>\n And here we need to add below entries between present at the end of this file:<\/p>\n [php]<property> Now we are completed with the installation procedure of Hadoop-2.6.2 in pseudo distributed mode. Now we will start Hadoop services by following steps.<\/p>\n For deep dive into Hadoop Yarn, resource manager follow this yarn tutorial.<\/a><\/p>\n a. Format the namenod<\/strong><\/p>\n Before starting the Hadoop services we need to format namenode. In order to do so, we will use the below command:<\/p>\n [php]dataflair@ubuntu:~$ hdfs namenode -format[\/php]<\/p>\n NOTE: This activity should be done only once when you install Hadoop<\/a> if you repeat it you lose all your data and different namespaceId will be assigned to namenode. If namenode and datanode have different id datanode daemon will not be started when to start the daemons.<\/em><\/p>\n b. Start HDFS Daemons:<\/strong> c. Start YARN Daemons:<\/strong> d. Check the status of services:<\/strong> NameNode If you are getting above output after running jps command, it means Hadoop has been installed successfully. Now you can perform HDFS<\/a><\/strong> operations (Follow HDFS commands guide<\/a>) and MapReduce<\/a> operations.<\/p>\n If you like this tutorial on How to install Hadoop with Yran on Ubuntu or if you have any queries regarding the tutorial, do let us know it in the comment section below and we will get back to you. <\/p>\n","protected":false},"excerpt":{"rendered":" 1.\u00a0Install Hadoop 2 with YARN Tutorial: Objective In this tutorial to install Hadoop 2 with Yarn, we will learn how to setup and run Apache Hadoop 2 with yarn on a single-node on Ubuntu...<\/p>\n","protected":false},"author":6,"featured_media":43201,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[782,1907,1971,2877,5186,5192,5201,5267,5329,5341,8140,8303,11664],"class_list":["post-353","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","tag-apache-hadoop","tag-big-data","tag-big-data-training","tag-configure-hadoop","tag-hadoop","tag-hadoop-2-x","tag-hadoop-admin","tag-hadoop-installation","tag-hadoop-setup","tag-hadoop-training","tag-learn-hadoop","tag-linux","tag-run-hadoop"],"yoast_head":"\n2.2. Prerequisites to Install Hadoop 2 with Yarn<\/h3>\n
I. Java 8 Installation<\/h4>\n
\nIn order to add the Java repositories, we need to download python-software-properties. Download and Installation of Python software properties can be easily executed by using the below command:<\/p>\n
\nNow we will add a Repository. A repository is nothing but a collection of software for a Linux distribution on a server from where we can easily download and install software\u2019s using command \u201capt-get\u201d in Linux. So, now we will manually add a java repository so that Ubuntu can easily install the Java. Command to add a Java repository:<\/p>\n
\nThe source list is basically a location where Ubuntu download and install the software and also updates the software. So whenever we install a new package\/software or update any previously installed software then it is recommended that we should update the source list periodically so that changes made can easily get saved and we can easily use the software. So in order to update source list, we need to use the following command:<\/p>\n
\nHadoop framework<\/a><\/strong> is written in Java. Hence it requires Java to be installed. So to download and install Java we need to use the below command:<\/p>\n
\nTo check the whether the Java is installed successfully or not and to check the version of your Java type the below command terminal:<\/p>\nII. Configure SSH to Install Hadoop with Yarn<\/h4>\n
\nThese SSH tools will be used for the remote login.<\/p>\n
\nNow will generate public and private key pair, which is used for authentication.<\/p>\n
\nWe need to copy the contents of \u201cid_rsa.pub\u201d into the \u201cauthorized_keys\u201d file. So this can be done by using the command:<\/p>\n
\n[php]ssh localhost[\/php]<\/p>\n2.3. Getting Started with Hadoop Installation<\/h3>\n
I. Downloading Hadoop<\/h4>\n
\n[php]http:\/\/www.eu.apache.org\/dist\/hadoop\/common\/hadoop-2.6.2\/hadoop-2.6.2.tar.gz[\/php]
\nAfter downloading Hadoop, we need to copy it to our home directory.<\/p>\nII. Untar Tar-ball<\/h4>\n
III. Setup Configuration to Install Hadoop with Yarn<\/h4>\n
\nNow firstly we need to edit environment file \u201c.bashrc\u201d which is present in our home directory. We can identify and edit this file with the command: $ nano -\/.bashrc\u201d. In this file we need to add the below contents at the end of this file:<\/p>\n
\nexport HADOOP_HOME=home\/dataflair\/hadoop-2.6.2
\nexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME\/lib\/native
\nexport HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME\/lib
\nexport HADOOP_PREFIX=”\/home\/hdadmin\/hadoop-2.6.2″
\nexport PATH=$PATH:$HADOOP_PREFIX\/bin
\nexport PATH=$PATH:$HADOOP_PREFIX\/sbin
\nexport HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
\nexport HADOOP_COMMON_HOME=${HADOOP_PREFIX}
\nexport HADOOP_HDFS_HOME=${HADOOP_PREFIX}
\nexport YARN_HOME=${HADOOP_PREFIX}
\n[\/php]<\/p>\n
\nNote: After above step restarts the terminal so that all the environment variables will come into effect<\/em><\/p>\n
\nNow we need to edit configuration \u201chadoop-env.sh” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) and here we have to provide root path of Java installation. We have to provide Java installation path in \u201chadoop-env.sh” file under the heading \u201cJAVA_HOME\u201d. This can be done by executing steps:<\/p>\n
\ndataflair@ubuntu:~\/hadoop-2.6.2$ cd etc\/hadoop
\ndataflair@ubuntu:~\/hadoop-2.6.2\/etc\/hadoop$ nano hadoop-env.sh[\/php]<\/p>\n
\nNote: \u201c\/usr\/lib\/jvm\/java-8-oracle\/\u201d is default Java path. If you had made any changes in your java path then enter that path here.<\/em><\/p>\n
\nNow we need to edit configuration \u201dcore-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n
\n<name>fs.defaultFS<\/name>
\n<value>hdfs:\/\/localhost:9000<\/value>
\n<\/property>
\n<property>
\n<name>hadoop.tmp.dir<\/name>
\n<value>\/home\/dataflair\/hdata<\/value>
\n<\/property>[\/php]<\/p>\n
\nNow we need to edit configuration \u201dhdfs-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n
\n<name>dfs.replication<\/name>
\n<value>1<\/value>
\n<\/property>[\/php]<\/p>\n
\nNow we need to edit configuration \u201dmapred-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n
\n<name>mapreduce.framework.name<\/name>
\n<value>yarn<\/value>
\n<\/property>[\/php]<\/p>\n
\nNow we need to edit configuration \u201dyarn-site.xml” file which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) with the help of the following command:<\/p>\n
\n<name>yarn.nodemanager.aux-services<\/name>
\n<value>mapreduce_shuffle<\/value>
\n<\/property>
\n<property>
\n<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<\/name>
\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>
\n<\/property>[\/php]<\/p>\nIV. Start the Hadoop Cluster<\/h4>\n
\n[php]dataflair@ubuntu:~$ start-dfs.sh[\/php]<\/p>\n
\n[php]dataflair@ubuntu:~$ start-yarn.sh[\/php]<\/p>\n
\n[php]dataflair@ubuntu:~$ jps[\/php]<\/p>\n
\nDataNode
\nResourceManager
\nNodeManager<\/p>\n
\nSee Also-<\/strong><\/p>\n\n