{"id":824,"date":"2016-08-01T18:29:32","date_gmt":"2016-08-01T18:29:32","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=824"},"modified":"2019-08-03T17:06:25","modified_gmt":"2019-08-03T11:36:25","slug":"install-hadoop-on-centos","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/install-hadoop-on-centos\/","title":{"rendered":"How to Install Hadoop on Centos | Cloudera Hadoop Installation"},"content":{"rendered":"

1. Install Hadoop on CentOS: Objective<\/h2>\n

This is\u00a0steps by\u00a0steps tutorial\u00a0to install Hadoop on CentOS,\u00a0configure and run Hadoop <\/a><\/strong>cluster on CentOS. This guide explains how to step by step install Hadoop on CentOS or we can say, deploy a single node cluster on CentOS, single node Hadoop cluster setup <\/a>is also called as pseudo-distributed mode installation.\u00a0Once the Cloudera installation is done you can play with Hadoop and its components<\/strong> <\/a>like MapReduce<\/a><\/strong> for data processing and HDFS<\/strong> <\/a>for data storage.<\/p>\n

\"Learn<\/a>

Learn how to install Cloudera Hadoop CDH5 on CentOS.<\/p><\/div>\n

2. Steps to Install Cloudera Hadoop on CentOS<\/h2>\n

Follow the steps given below to install Hadoop on CentOS-<\/p>\n

2.1. Install Java<\/h3>\n

Go\u00a0to your home directory and download the Java using the below commands:
\n[php]cd ~
\ncd \/home\/saurabhpandey\/Downloads
\ntar xzf jdk-8u101-linux-x64.tar.gz
\nmv \/home\/saurabhpandey\/Downloads\/jdk1.8.0_101 \/home\/saurabhpandey[\/php]<\/p>\n

I. Install Java with Alternatives<\/h4>\n

In order to install Java we will use alternatives command which is available in chkconfig package.
\n[php]cd ~
\ncd jdk1.8.0_101\/
\nsudo alternatives –install \/usr\/bin\/java java \/home\/saurabhpandey\/jdk1.8.0_101\/bin\/java 2
\n[sudo] password for saurabhpandey:
\nsudo alternatives –config java
\n[sudo] password for saurabhpandey:[\/php]
\nThere are 3 programs which provide ‘java’.<\/p>\n

Selection\u00a0\u00a0\u00a0 Command\r\n-----------------------------------------------\r\n*+ 1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\/usr\/lib\/jvm\/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64\/jre\/bin\/java\r\n2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\/usr\/lib\/jvm\/java-1.7.0-openjdk-1.7.0.95-2.6.4.0.el7_2.x86_64\/jre\/bin\/java\r\n3\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\/home\/saurabhpandey\/jdk1.8.0_101\/bin\/java<\/pre>\n
Type selection number: 3,\u00a0Hence Java\u00a08 gets successfully installed on your system.<\/p>\n
\"installed<\/a>
Install Java alternatives to Install Hadoop on CentOS<\/p><\/div>\n
II. Configuring Environment Variables<\/h4>\n
In order to configure Java environment variables we need to use the following commands:
\na. Setup JAVA_HOME Variable<\/strong>
\n[php]export JAVA_HOME=\/home\/saurabhpandey\/Downloads\/jdk1.8.0_101[\/php]
\nb. Setup JRE_HOME Variable<\/strong>
\n[php]export JRE_HOME=\/home\/saurabhpandey\/Downloads\/jdk1.8.0_101\/jre[\/php]
\nc. Setup PATH Variable<\/strong>
\n[php]export PATH=$PATH:\/home\/saurabhpandey\/Downloads\/jdk1.8.0_101\/bin:\/home\/saurabhpandey\/Downloads\/jdk1.8.0_101\/jre\/bin[\/php]<\/p>\n
2.2. Install Hadoop on CentOS<\/h3>\n
I. Configure Password Less SSH<\/h4>\n
In order to set up key based ssh, we need to execute following commands:
\n[php]ssh-keygen -t rsa[\/php]<\/p>\n
Generating public\/private rsa key pair.\r\nEnter file in which to save the key (\/home\/hadoop\/.ssh\/id_rsa): \u201cPress Enter\u201d\r\nCreated directory '\/home\/hadoop\/.ssh'.\r\nEnter passphrase (empty for no passphrase): \u00a0\u201cPress Enter\u201d\r\nEnter same passphrase again: \u201cPress Enter\u201d<\/pre>\n
[php]cat ~\/.ssh\/id_rsa.pub >> ~\/.ssh\/authorized_keys
\nchmod 0600 ~\/.ssh\/authorized_keys[\/php]<\/p>\n
\"ssh<\/a>
Install Hadoop on CentOS to Configure Password Less SSH<\/p><\/div>\n
In order to verify key based login, we need to execute below command. But, it should not ask for a password.
\n[php]ssh localhost[\/php]<\/p>\n
II. Download Hadoop<\/h4>\n
http:\/\/archive.cloudera.com\/cdh5\/cdh\/5\/hadoop-2.6.0-cdh5.5.0.tar.gz<\/a>
\nNow copy Hadoop to Home directory of your system once it gets downloaded.<\/p>\n
III. Untar Hadoop File<\/h4>\n
All the contents of compressed Hadoop file package can be extracted using the below command:
\n[php]tar xzf hadoop-2.6.0-cdh5.5.0.tar.gz[\/php]<\/p>\n
IV. Setup Configuration Files<\/h4>\n
a. Edit .bashrc<\/strong>
\nNow makes changes in environment file \u201c.bashrc\u201d present in your system home directory. We can make changes in this file by executing: \u201c$ nano -\/.bashrc\u201d command and there we need to insert following parameters at the end:
\n[php]export HADOOP_HOME=\/home\/saurabhpandey\/hadoop-2.6.0-cdh5.5.0
\nexport HADOOP_INSTALL=$HADOOP_HOME
\nexport HADOOP_MAPRED_HOME=$HADOOP_HOME
\nexport HADOOP_COMMON_HOME=$HADOOP_HOME
\nexport HADOOP_HDFS_HOME=$HADOOP_HOME
\nexport YARN_HOME=$HADOOP_HOME
\nexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME\/lib\/native
\nexport PATH=$PATH:$HADOOP_HOME\/sbin:$HADOOP_HOME\/bin[\/php]<\/p>\n
\"hadoop_path_bashrc\"<\/a>
Setup configuration files to Install Hadoop on CentOS<\/p><\/div>\n
Note: Don\u2019t forget to take care of entering the correct path. “\/home\/saurabhpandey\/hadoop-2.6.0-cdh5.5.0\u201d this is the path of my home directory. You can get your path by using \u201c$pwd\u201d command.
\nAfter adding above parameters save this file. Press \u201cCtrl+X\u201d on your keyboard.
\nIn order to make environment variables effective we need to execute below command:
\n[php]source .bashrc[\/php]
\nb. Edit hadoop-env.sh<\/strong>
\nNow makes changes in Hadoop configuration file \u201chadoop-env.sh” which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) and in that file we need to set JAVA_HOME.
\n[php]cd hadoop-2.6.0-cdh5.5.0\/
\ncd etc
\ncd hadoop
\nnano hadoop-env.sh[\/php]
\nWe need to change the path of JAVA_HOME on this file as:
\n[php]export JAVA_HOME=<path-to-the-root-of-your-Java-installation> (eg: \/home\/saurabhpandey\/Downloads\/jdk1.8.0_101\/)[\/php]<\/p>\n
\"Hadoop-env.sh_Java_Home\"<\/a>
Install Hadoop on CentOS – Edit hadoop-env.sh<\/p><\/div>\n
After changing the path of JAVA_HOME save this file. Press\u00a0 \u201cCtrl+X\u201d on your keyboard.
\nNote: \u201c\/home\/saurabhpandey\/Downloads\/jdk1.8.0_101\/\u201d is the path of Java on my system. Enter your path where your java file is present.
\nc. Edit core-site.xml<\/strong>
\nNow makes changes in Hadoop configuration file \u201ccore-site.xml” which is present in configuration directory (HADOOP_HOME\/etc\/hadoop) by executing the below command:
\n[php]nano core-site.xml[\/php]
\nAnd inside it add the following entries between <configuration> <\/configuration> present at the end of this file:
\n[php]
\n<property>
\n<name>fs.default.name<\/name>
\n<value>hdfs:\/\/localhost:9000<\/value>
\n<\/property>
\n[\/php]<\/p>\n
\"Hadoop-Core-site_edit\"<\/a>
Install Hadoop on CentOS – Edit core-site<\/p><\/div>\n
After adding above parameters save this file. Press \u201cCtrl+X\u201d on your keyboard.
\ne. Edit hdfs-site.xml<\/strong>
\nNow we need to make changes in Hadoop configuration file hdfs-site.xml (which is located in HADOOP_HOME\/etc\/hadoop) by executing the below command:
\n[php]nano hdfs-site.xml[\/php]
\nAnd inside it add the following entries between <configuration> <\/configuration> present at the end of this file:
\n[php]
\n<property>
\n<name>dfs.replication<\/name>
\n<value>1<\/value>
\n<\/property>[\/php]<\/p>\n
\"Hadoop-hdfs-site_edit\"<\/a>
Install Hadoop on CentOS – Edit hdfs-site.xml<\/p><\/div>\n
Here we are setting Replication factor to 1, as it is a single node cluster<\/a>.
\nAfter adding above parameters save this file. Press\u00a0 \u201cCtrl+X\u201d on your keyboard.
\nf. Edit mapred-site.xml<\/strong>
\nIn order to edit mapred-site.xml file we need to first create a copy of file mapred-site.xml.template.
\nA copy of this file can be created using the following command:
\n[php]cp mapred-site.xml.template mapred-site.xml[\/php]
\nWe will edit the mapred-site.xml file by using the following command:
\n[php]nano mapred-site.xml[\/php]
\nAnd inside it add the following entries between <configuration> <\/configuration> present at the end of this file:
\n[php]
\n<property>
\n<name>mapreduce.framework.name<\/name>
\n<value>yarn<\/value>
\n<\/property>[\/php]<\/p>\n
\"Mapred-site_edit\"<\/a>
Install Hadoop on Cent OS – Edit mapred-site.xml<\/p><\/div>\n
After adding above parameters save this file. Press\u00a0 \u201cCtrl+X\u201d on your keyboard.
\ng. Edit yarn-site.xml<\/strong>
\nNow we need to make changes in Hadoop configuration file yarn-site.xml (which is located in HADOOP_HOME\/etc\/hadoop) by executing the below command:
\n[php]nano yarn-site.xml[\/php]
\nAnd inside it add the following entries between <configuration> <\/configuration> present at the end of this file:
\n[php]
\n<property>
\n<name>yarn.nodemanager.aux-services<\/name>
\n<value>mapreduce_shuffle<\/value>
\n<\/property>
\n<property>
\n<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<\/name>
\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>
\n<\/property>[\/php]<\/p>\n
\"yarn-site_edit\"<\/a>
Hadoop installation Steps – Edit yarn-site.xml<\/p><\/div>\n
After adding above parameters save this file. Press\u00a0 \u201cCtrl+X\u201d on your keyboard.<\/p>\n
V. Starting the Hadoop Cluster<\/h4>\n
a. NameNode Formatting<\/strong>
\n[php]hdfs namenode -format[\/php]<\/p>\n
\"starting_hadoop\"<\/a>
Cloudera Hadoop Installation – NameNode formatting<\/p><\/div>\n
NOTE: After we install hadoop on CentOS, we perform this activity only once otherwise all the data will get deleted and namenode would be assigned some other namespaceId. In case namenode and datanode Id\u2019s doesn\u2019t matches, then the datanode daemons will not start while starting the daemons.
\nb. Start HDFS Daemons<\/strong>
\n[php]start-dfs.sh[\/php]
\nc. Start YARN Daemons<\/strong>
\n[php]start-yarn.sh[\/php]<\/p>\n
2.3. Checking Installation<\/h3>\n
In order to check whether Hadoop is running successfully or not open the below address on any of your browsers:
\nhttp:\/\/localhost:50070<\/strong><\/p>\n
\"Browser\"<\/a>
Checking Install Hadoop on CentOS<\/p><\/div>\n
If this output appears means you have\u00a0successfully installed Hadoop in CentOS. After Hadoop Installation studies this guide to learn how Hadoop works<\/a>.
\nSee Also-<\/strong><\/p>\n