![\"Install](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/11\/Install-Hadoop-2.8.jpg\")
<\/a>Install Hadoop on Ubuntu | Hadoop Installation Steps<\/p><\/div>\n
2. Hadoop Overview\u00a0\u00a0<\/b><\/h2>\n
Hadoop is a framework for running distributed computing programs. It comprises of HDFS<\/strong><\/a> and Map Reduce<\/strong><\/a> (Programming framework).<\/span> [php]$ java -version[\/php]<\/p>\n Lets now discuss the steps to install Hadoop single node cluster on Ubuntu-<\/p>\n We will now setup the passwordless ssh client with the following command. Install Hadoop on Ubuntu – Open SSH Server and Open SSH Client<\/p><\/div>\n [php]ssh-keygen -t rsa -P “”[\/php] The below command will add the public ssh-key to authorized_keys. Moreover, it will configure the passwordless ssh.<\/span> Install Hadoop on Ubuntu – Configure password-less SSH<\/p><\/div>\n Also see What is Hadoop Cluster | Hadoop Cluster Architecture<\/a><\/p>\n As we type the \u201cssh localhost\u201d it will prompt us to connect with it. Type \u2018yes\u2019 and press enter to proceed.<\/span><\/p>\n [php]$ sudo apt-get install rsync[\/php]<\/p>\n Install Hadoop on Ubuntu | Install rsync<\/p><\/div>\n Use the link given below to download the Hadoop 2.8.x from Apache mirrors. Then we will extract the Hadoop into the home directory how to install hadoop on Ubuntu – Untar the Tarball<\/p><\/div>\n We can add only the minimum property in the Hadoop configuration. The user can add more properties to it.<\/span><\/p>\n [php]nano bash.bashrc[\/php] Hadoop Installation – Edit .bashrc<\/p><\/div>\n [php]source ~\/.bashrc[\/php]<\/p>\n Also see HBase Compaction and Data Locality in Hadoop<\/a><\/p>\n Edit hadoop-env.sh file which is in etc\/hadoop inside the Hadoop installation directory. The user can set JAVA_HOME:<\/span> Edit hadoop-env.sh<\/p><\/div>\n Edit the core-site.xml with \u201cnano core-site.xml\u201d. The file is in the etc\/hadoop inside Hadoop directory. Then we will add following entries.<\/span> Hadoop 2 Installation – Edit core-site.xml<\/p><\/div>\n Then we will edit the hdfs-site.xml with \u201cnano hdfs-site.xml\u201d. This file is actually located in etc\/hadoop inside Hadoop installation directory. We will add the following entries:\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0<\/b><\/p>\n [php]<configuration> Install hadoop on Ubuntu – Edit hdfs-site.xml<\/p><\/div>\n We will create a copy of mapred-site.xml from mapred-site.xml.template using cp command (cp mapred-site.xml.template mapred-site.xml). Now edit the mapred-site.xml with \u201cnano command\u201d. This file is also located in etc\/hadoop inside Hadoop directory. We will copy the file with same name mapred-site.xml. This will add following entries:<\/span> Install hadoop on Ubuntu – Edit mapred-site.xml<\/p><\/div>\n We will now edit yarn-site.xml with \u201cnano yarn-site.xml\u201d. It is in etc\/hadoop inside Hadoop installation directory. Finally we add following entries:<\/span> Hadoop Installation – Edit yarn-site.xml file.<\/p><\/div>\n Also see Unbeatable Apache Hadoop Online Quiz \u2013 Part 10<\/a><\/p>\n We will now start the single node cluster with the following commands.<\/span><\/p>\n Moreover, we will format the namenode before using it the first time.<\/span> Install Hadoop 2 – Format the Namenode<\/p><\/div>\n We will start the hadoop cluster using the hadoop start-up script. <\/span> Hadoop start-up script<\/p><\/div>\n For starting the YARN we use <\/span> Apache Hadoop Installation on Ubuntu – Starting the YARN services<\/p><\/div>\n [php]jps Install Hadoop on Ubuntu<\/p><\/div>\n visit : (http:\/\/localhost:50070)<\/span><\/p>\n Web interface-For viewing Web UI of NameNode<\/p><\/div>\n Also see Hadoop Schedulers Tutorial \u2013 Job Scheduling in Hadoop<\/a><\/p>\n The web interface will display all running jobs on cluster information. Hence, this will help monitor the progress report.<\/span><\/p>\n Hadoop Installation – Resource Manager UI<\/p><\/div>\n To Stop the HDFS Services we use stop-dfs.sh. To Stop YARN Services we use<\/span> Install Hadoop on Ubuntu – Stopping the clusters<\/p><\/div>\n You have successfully installed Hadoop 2.8.x on Ubuntu. Now you can play with big data<\/strong><\/a> using Hadoop HDFS commands<\/strong><\/a>. For any queries on How to install Hadoop on Ubuntu just drop a comment and we will be back to you. <\/span><\/p>\n","protected":false},"excerpt":{"rendered":" 1. Install Hadoop on Ubuntu Tutorial: Objective This article explains how to install Hadoop on Ubuntu in simple steps. Moreover, we will deploy Hadoop on the single node cluster on Ubuntu Linux. We will...<\/p>\n","protected":false},"author":6,"featured_media":42878,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[782,1907,1918,5186,5191,5267],"class_list":["post-4741","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","tag-apache-hadoop","tag-big-data","tag-big-data-hadoop","tag-hadoop","tag-hadoop-2-8-x-installation","tag-hadoop-installation"],"yoast_head":"\n
\nThe user can run only the MapReduce program in the earlier versions of Hadoop. Therefore, it was fit for batch processing computations.<\/span>
\nThe YARN<\/strong><\/a> provides API for requesting and allocating resource in the cluster. So the YARN is available in later versions of Hadoop 2. Hence, the API facilitates application program to process large-scale data of HDFS. <\/span><\/p>\n3. Prerequisites to Install Hadoop on Ubuntu<\/h2>\n
\n
4. Easy Steps to install Hadoop on Ubuntu<\/h2>\n
4.1. Setup passwordless ssh<\/h3>\n
a) Install Open SSH Server and Open SSH Client<\/h4>\n
\n[php]sudo apt-get install openssh-server openssh-client[\/php]<\/p>\n<\/a>b) Generate Public & Private Key Pairs<\/h4>\n
\nThe terminal will prompt the user for entering the file name. Press enter and proceed. The location of a file will be in the home directory. Moreover, the extension will be the .ssh file.<\/span><\/p>\nc) Configure password-less SSH<\/h4>\n
\n[php]cat $HOME\/.ssh\/id_rsa.pub >> $HOME\/.ssh\/authorized_keys[\/php]<\/p>\n<\/a>d) Now verify the working of password-less ssh<\/h4>\n
e) Now install rsync with command<\/h4>\n
<\/a>4.2. Configure and Setup Hadoop<\/h3>\n
a) Download the Hadoop package 2.8.x<\/h4>\n
\nhttp:\/\/www-eu.apache.org\/dist\/hadoop\/common\/hadoop-2.8.2\/<\/span><\/a><\/p>\nb) Untar the Tarball<\/h4>\n
\n[php]tar xzf hadoop-2.8.2.tar.gz[\/php]<\/p>\n<\/a>4.3. Setup Configuration<\/h3>\n
a) Setting Up the environment variables<\/h4>\n
\n
\nAnd add the following path variables in it<\/span>
\n[php]export HADOOP_HOME=\/home\/hduser\/hadoop-2.8.2
\nexport HADOOP_INSTALL=$HADOOP_HOME
\nexport HADOOP_MAPRED_HOME=$HADOOP_HOME
\nexport HADOOP_COMMON_HOME=$HADOOP_HOME
\nexport HADOOP_HDFS_HOME=$HADOOP_HOME
\nexport YARN_HOME=$HADOOP_HOME
\nexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME\/lib\/native
\nexport PATH=$PATH:$HADOOP_HOME\/sbin:$HADOOP_HOME\/bin[\/php]<\/p>\n<\/a>\n
b)\u00a0Hadoop configuration file changes<\/h4>\n
\n
\n[php]export JAVA_HOME=<root directory of Java-installation> (eg: \/usr\/lib\/jvm\/jdk1.8.0_151\/)[\/php]<\/p>\n<\/a>\n
\n[php]<configuration>
\n<property>
\n<name>fs.defaultFS<\/name>
\n<value>hdfs:\/\/localhost:9000<\/value>
\n<\/property>
\n<property>
\n<name>hadoop.tmp.dir<\/name>
\n<value>\/home\/hdadmin\/hdata<\/value>
\n<\/property>
\n<\/configuration>[\/php]<\/p>\n<\/a>\n
\n<property>
\n<name>dfs.replication<\/name>
\n<value>1<\/value>
\n<\/property>
\n<\/configuration>[\/php]<\/p>\n<\/a>\n
\n[php]<configuration>
\n<property>
\n<name>mapreduce.framework.name<\/name>
\n<value>yarn<\/value>
\n<\/property>
\n<\/configuration>[\/php]<\/p>\n<\/a>\n
\n[php]<configuration>
\n<property>
\n<name>yarn.nodemanager.aux-services<\/name>
\n<value>mapreduce_shuffle<\/value>
\n<\/property>
\n<property>
\n<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<\/name>
\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>
\n<\/property>
\n<\/configuration>[\/php]<\/p>\n<\/a>4.4. Start the cluster<\/h3>\n
a) Format the namenode<\/h4>\n
\n[php]hdfs namenode -format[\/php]<\/p>\n<\/a>b) Start the HDFS<\/h4>\n
\n[php]start-dfs.sh[\/php]<\/p>\n<\/a>c) Starting the YARN services<\/h4>\n
\n[php]start-yarn.sh[\/php]<\/p>\n<\/a>d) Verify if all process started<\/h4>\n
\n6775 DataNode
\n7209 ResourceManager
\n7017 SecondaryNameNode
\n6651 NameNode
\n7339 NodeManager
\n7663 Jps[\/php]<\/p>\n<\/a>e) Web interface-For viewing Web UI of NameNode<\/h4>\n
<\/a>f) Resource Manager UI \u00a0(http:\/\/localhost:8088)<\/h4>\n
<\/a>4.5. Stopping the clusters<\/h3>\n
\n[php]stop-yarn.sh[\/php]<\/p>\n<\/a>
\nSee Also-<\/strong><\/p>\n\n