Install Hadoop on Single Machine

1. Objective

The Agenda for this tutorial is to describe how to Deploy / Setup Hadoop on the Single node. The tutorial is very simple so that you can install Hadoop on single machine in 10 Min. Alternatively, you can watch the video tutorial which shows end to end steps to configure Hadoop. Single Node Hadoop cluster is also called as Hadoop Pseudo-Distributed Mode. Once Hadoop is installed you can use Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

Looking to Install the latest version of Hadoop, follow these tutorials: Install Cloudera Hadoop CDH5 on Ubuntu and Install & Configure Apache Hadoop 2.7.x on Ubuntu

Install Hadoop on Single Machine

Install Hadoop on Single Machine

Hadoop Quiz

2. Install Hadoop on Single Machine

2.1. Recommended Platform

  • OS: Ubuntu 12.04 or later (you can use other OS (centOS, Redhat, etc))
  • Hadoop: Cloudera distribution for Apache Hadoop CDH3U6 (you can use Apache Hadoop 1.X)

I. Setup Platform

If you are using Windows/Mac OS you can create a virtual machine and install Ubuntu using VMWare Player, alternatively you can create virtual machine and install Ubuntu using Oracle Virtual Box.

2.2. Prerequisites

  • Java (oracle java is recommended for production)
  • Password-less SSH setup (Hadoop need passwordless ssh from master to all the slaves, this is required for remote script invocations)

I. Install Java

Install Java 8 or later (either oracle / open jdk)
Update the source list:
sudo apt-get update
Install Open jdk:
sudo apt-get install openjdk-8-jdk

II. Configure Password-less SSH

Install Open SSH Server-Client:
sudo apt-get install openssh-server openssh-client
Generate key-value pairs:
ssh-keygen -t rsa -P ""
Configure password-less SSH:
cat $HOME/.ssh/id_rsa.pub>>$HOME/.ssh/authorized_keys
Check by SSH to localhost:
ssh localhost

2.3. Install Hadoop

I. Download Hadoop:

http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u6.tar.gz

II. Deploy Hadoop

Untar Tarball:
tar xzf hadoop-0.20.2-cdh3u5.tar.gz
Go to HADOOP_HOME DIR, All the commands will be run from Hadoop_Home Directory
cd hadoop-0.20.2-cdh3u5/

III. Setup Configuration

a. Edit hadoop-env.sh:

Edit configuration file conf/hadoop-env.sh and set JAVA_HOME
export JAVA_HOME=path to be the root of your Java installation(eg: /usr/lib/jvm/java-8-sun)

b. Edit core-site.xml:

Edit configuration file conf/core-site.xml and add following entries:

<configuration>
<property>
<name>fs.default.name</name> <value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value><PATH-WHERE-YOU-HAVE-READ-WRITE-PRIVILEGES></value>
</property>
</configuration>

c. Edit hdfs-site.xml:

Edit configuration file conf/hdfs-site.xml and add following entries:

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

d. Edit mapred-site.xml:

Edit configuration file conf/mapred-site.xml and add following entries:

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

IV. Start The Cluster

a. Format the name node:

bin/hadoop namenode -format
This activity should be done once when you install hadoop, else It will delete all your data from HDFS

b. Start Hadoop Services:

bin/start-all.sh

c. Check whether services have been started:

jps
NameNode
SecondaryNameNode
DataNode
JobTracker
TaskTracker

2.4. Stop Hadoop Services

Once your work has been done, You can stop the services
bin/stop-all.sh

If these professionals can make a switch to Big Data, so can you:
Rahul Doddamani Story - DataFlair
Rahul Doddamani
Java → Big Data Consultant, JDA
Follow on
Mritunjay Singh Success Story - DataFlair
Mritunjay Singh
PeopleSoft → Big Data Architect, Hexaware
Follow on
Rahul Doddamani Success Story - DataFlair
Rahul Doddamani
Big Data Consultant, JDA
Follow on
I got placed, scored 100% hike, and transformed my career with DataFlair
Enroll now
Deepika Khadri Success Story - DataFlair
Deepika Khadri
SQL → Big Data Engineer, IBM
Follow on
DataFlair Web Services
You could be next!
Enroll now

3. Related Links

Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.