![\"Install](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Pivotal-Hadoop-v-2-Cluster-in-Production-01.jpg\")
<\/a>Install Pivotal Hadoop v-2 Cluster in production<\/p><\/div>\n
2. Recommended Platform<\/h1>\n
During the installation we are using following platform:
\nA) CentOS 6.4 (You can use RHEL or other versions as well)
\nB) 14 node cluster with following details:<\/p>\n
- \n
- 1 master (or namenode) (it should be of higher configuration than slaves it is recommended to have more RAM (as it keeps all the meta-data in the memory), although less disk space is required on master)<\/li>\n
- 1 standby-master (configuration of master and standby-master must be same, when master crashes this standy-master will automatically become master)<\/li>\n
- 1 client node (where all the clients like Pig, Hive, HBase and Mahout libraries will be installed, we can access complete components (Hadoop and all the sub-projects of Hadoop) of the cluster from this client node. This would be useful also when you want to share your cluster with other teams, you do not need to share master instead of that share this client machine with them)<\/li>\n
- 10 slaves (DataNodes) (these servers can of less configuration (less RAM \/ CPU, but should have multiple disks with high capacity))<\/li>\n
- 1 admin node (we will install PCC (Pivotal command centre) and ICM (installation and configuration Manager) on this machine)<\/li>\n<\/ul>\n
C) PHD-2.X<\/p>\n
3. Prerequisite:<\/h1>\n
![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Apache-Hadoop.png\")
<\/a>Apache Hadoop<\/p><\/div>\n
3.1. Install Java<\/h2>\n
Ensure that you are running Oracle JAVA JDK version 1.7 on the Admin node. Version 1.7 is required; version 1.7u15 is recommended.
\nIf you are not running the correct JDK, download a supported version from the Oracle site at http:\/\/www.oracle.com\/technetwork\/java\/javase\/downloads\/index.html
\n[php]\/usr\/java\/default\/bin\/java \u2013version[\/php]
\nThe output of this command should contain 1.7 (version number) and JavaHotSpot(TM) (Java version). For example:
\n[php]java version \u201c1.7.0_45\u2033Java(TM) SE Runtime Environment (build 1.7.0_45-b18)Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)[\/php]
\nNOTE: Make sure you are not running OpenJDK as your default JDK. If you are running OpenJDK, we recommend you to remove it.<\/p>\n3.2. Update hosts file<\/h2>\n
Add entry of all the machines in the hosts file of all the nodes. Edit the hosts file (nano \/etc\/hosts) and add following entries:
\n[php]admin.domain.com admin
\nclient.domain.com client
\nhdnn1.domain.com hdnn1
\nhdnn2.domain.com hdnn2
\nhddn1.domain.com hddn1
\n(do this for all the slaves ie hddn1, hddn2, hddn3\u2026. hddn10)[\/php]<\/p>\n3.3. Change system parameters<\/h2>\n
3.3.1. Change hostname<\/h3>\n
If you have hostname in capital letters, change the hostname:
\nAvoid using hostnames that contain capital letters because Puppet has an issue generating certificates for domains with capital letters. Also avoid using underscores as they are invalid characters in hostnames.
\n[php]nano \/etc\/sysconfig\/networkHOSTNAME=admin.domain.com[\/php]
\nNOTE: Do this activity for all the machines in the cluster, make sure you provide corresponding hostname in the \/etc\/sysconfig\/network file<\/p>\n3.3.2. Disable SELinux:<\/h3>\n
[php]sestatus[\/php]
\nBut if the output is
\n[php]sestatus
\nSELinux status: enabled
\nSELinuxfs mount: \/selinux
\nCurrent mode: enforcing
\nMode from config file: enforcing
\nPolicy version: 24
\nPolicy from config file: targeted[\/php]
\nThen edit \/etc\/selinux\/config and add following entries:
\n[php]nano \/etc\/selinux\/configSELINUX=disabled[\/php]<\/p>\n3.3.3. Update Operating System Parameters<\/h3>\n
Add following configuration parameter of Operating system to get best performance:<\/p>\n
3.3.3.1. Edit \/etc\/sysconfig\/cpuspeed<\/h4>\n
Edit \/etc\/sysconfig\/cpuspeed and add following parameter:
\n[php]nano \/etc\/sysconfig\/cpuspeedGOVERNOR=performance[\/php]<\/p>\n3.3.3.2. Stop cpuspeed service<\/h4>\n
[php]service cpuspeed stopDisabling ondemand cpu frequency scaling: [ OK ][\/php]<\/p>\n
3.3.3.3. Stop iptables service<\/h4>\n
[php]service iptables stopiptables: Firewall is not running.[\/php]<\/p>\n
3.3.3.4. Stop ip6tables service<\/h4>\n
[php]service ip6tables stop[\/php]<\/p>\n
3.3.3.5. Stop iptables<\/h4>\n
[php]chkconfig iptables off[\/php]<\/p>\n
3.3.3.6. Stop ip6tables<\/h4>\n
[php]chkconfig ip6tables off[\/php]<\/p>\n
3.3.3.7. Stop cpuspeed<\/h4>\n
[php]chkconfig cpuspeed off[\/php]<\/p>\n
3.3.3.8. Edit \/etc\/sysctl.conf (OPTIONAL)<\/h4>\n
Edit \/etc\/sysctl.conf and add following parameters
\n[php]nano \/etc\/sysctl.conf
\n# COMMENT FOLLOWING PARAMETERES:
\n# Disable netfilter on bridges.
\n#net.bridge.bridge-nf-call-ip6tables = 0
\n#net.bridge.bridge-nf-call-iptables = 0
\n#net.bridge.bridge-nf-call-arptables = 0# ADD FOLLOWING PARAMETERS:kernel.shmmax = 500000000
\nkernel.shmmni = 4096
\nkernel.shmall = 4000000000
\nkernel.sem = 250 512000 100 2048
\nkernel.sysrq = 1
\nkernel.core_uses_pid = 1
\nkernel.msgmnb = 65536
\nkernel.msgmax = 65536
\nkernel.msgmni = 2048
\nnet.ipv4.tcp_syncookies = 1
\nnet.ipv4.ip_forward = 0
\nnet.ipv4.conf.default.accept_source_route = 0
\nnet.ipv4.tcp_tw_recycle = 1
\nnet.ipv4.tcp_max_syn_backlog = 4096
\nnet.ipv4.conf.all.arp_filter = 1
\nnet.ipv4.ip_local_port_range = 1025 65535
\nnet.core.netdev_max_backlog = 10000
\nnet.core.rmem_max = 2097152
\nnet.core.wmem_max = 2097152
\nvm.overcommit_memory = 2[\/php]<\/p>\n3.3.3.9. Edit \/etc\/security\/limits.conf<\/h4>\n
Edit \/etc\/security\/limits.conf and add following parameter
\n[php]nano \/etc\/security\/limits.conf
\n# COMMENT FOLLOWING PARAMETERS:
\n#* soft nofile 65536
\n#* hard nofile 65536
\n#* soft nproc 131072
\n#* hard nproc 131072[\/php]<\/p>\n3.3.3.10. Edit \/etc\/security\/limits.d\/90-nproc.conf<\/h4>\n
Edit \/etc\/security\/limits.d\/90-nproc.conf and add following parameter
\n[php]nano \/etc\/security\/limits.d\/90-nproc.conf
\n# COMMENT FOLLOWING PARAMETERES
\n#root soft nproc unlimited# ADD FOLLOWING PARAMETERS:
\n* soft nofile 65536
\n* hard nofile 65536
\n* soft nproc 131072
\n* hard nproc 131072[\/php]<\/p>\n3.3.3.11. Edit \/etc\/rc.d\/rc.local<\/h4>\n
Edit \/etc\/rc.d\/rc.local and add following parameter
\n[php]nano \/etc\/rc.d\/rc.local
\nblockdev \u2013setra 16384 \/dev\/sd*[\/php]<\/p>\n3.4. Install Following packages<\/h2>\n
[php]yum install httpd mod_ssl postgresql postgresql-devel postgresql-server postgresql-jdbc createrepo sigar sudo[\/php]<\/p>\n
4. Install PCC (Pivotal Command Centre)<\/h1>\n
NOTE: Configure FQDN on admin node by adding FQDN in \/etc\/hosts
\n[php]cat \/etc\/hosts | grep admin
\nadmin.domain.com admin
\n$hostname
\nadmin $hostname -f
\nadmin.domain.com[\/php]
\nUntar PCC:
\n[php]tar -xvf PCC-2.2.1-150.x86_64.tar[\/php]
\nInstall Pivotal Command Centre:
\n[php].\/install[\/php]
\nNOTE:
\n[php]facter \u2013puppet fqdn (output should not be blank)[\/php]
\nCheck whether the commander has started:
\n[php]service commander status[\/php]<\/p>\n5. Deploy Pivotal Hadoop<\/h1>\n
NOTE: RUN FOLLOWING COMMANDS AS \u201cgpadmin\u201d USER<\/em>
\nUntar Hadoop:
\n[php]tar xzf PHD-2.0.1.0-148.tar[\/php]
\nImport the following tarballs for Pivotal HD:
\n[php]icm_client import -s PHD-2.0.1.0-148[\/php]
\nFetch the default cluster Configuration template:
\n[php]icm_client fetch-template -o ~\/ClusterConfigDir[\/php]
\nEdit clusterConfig.xml and set all the required parameters:
\n[php]nano clusterConfig.xml
\n# It should look like:
\n<?xml version=\u201d1.0\u2033 encoding=\u201dUTF-8\u2033?>
\n<clusterConfig>
\n<!– Cluster Name. This will be used on command lines, so avoid spaces and special characters. –>
\n<clusterName>my-hd-cluster<\/clusterName>
\n<gphdStackVer>PHD-2.0.1.0<\/gphdStackVer>
\n<!– Comma separated list. Choices are: hdfs,yarn,zookeeper,hbase,hive,hawq,gpxf,pig,mahout –>
\n<!– services not meant to be installed can be deleted from the following list. –>
\n<!– <services>hdfs,yarn,zookeeper,hbase,hive,hawq,gpxf,pig,mahout<\/services> –>
\n<services>hdfs,yarn,zookeeper,hbase,hive,pig,mahout<\/services>
\n<!– Hostname of the machine which will be used as the client –>
\n<!– ICM will install Pig, Hive, HBase and Mahout libraries on this machine. –>
\n<client>phdnn1.domain.com<\/client>
\n<!– Comma separated list of fully qualified hostnames should be associated with each service role –>
\n<!– services (and corresponding roles) not meant to be installed can be deleted from the following list –>
\n<hostRoleMapping>
\n<hdfs>
\n<!– HDFS, NameNode role (mandatory) –>
\n<namenode>phdnn1.domain.com<\/namenode>
\n<!– HDFS, DataNode role (mandatory) ** list of one or more, comma separated,
\nnames must be resolvable ** –>
\n<datanode>phddn1.domain.com,phddn2.domain.com,phddn3.domain.com,
\nphddn4.domain.com,phddn5.domain.com,phddn6.domain.com,phddn7.domain.com,
\nphddn8.domain.com,phddn9.domain.com,phddn10.domain.com<\/datanode>
\n<!– HDFS, Secondary NameNode role (optional) \u2013 ** comment out XML tag if not
\nrequired ** –>
\n<!– <secondarynamenode>host.yourdomain.com<\/secondarynamenode> –>
\n<!– HDFS, Standby NameNode that acts as a failover when HA is enabled –>
\n<standbynamenode>phdnn2.domain.com<\/standbynamenode>
\n<!–specify quorum journal nodes used as common storage by both active and
\nstand-by namenode. list of one or more comma separated –>
\n<journalnode>phdnn1.domain.com,phdnn2.domain.com,
\nphddn1.domain.com<\/journalnode>
\n<\/hdfs>
\n<yarn>
\n<!– YARN, ResourceManager role (mandatory) –>
\n<yarn-resourcemanager>phdnn1.domain.com<\/yarn-resourcemanager>
\n<!– YARN, NodeManager role (mandatory) ** list of one or more,
\ncomma separated, generally takes on the DataNode element ** –>
\n<yarn-nodemanager>phddn1.domain.com,phddn2.domain.com,phddn3.domain.com,
\nphddn4.domain.com,phddn5.domain.com,phddn6.domain.com,phddn7.domain.com,
\nphddn8.domain.com,phddn9.domain.com,phddn10.domain.com<\/yarn-nodemanager>
\n<!– Generally runs on the ResourceManager (mandatory) –>
\n<mapreduce-historyserver>phdnn1.domain.com<\/mapreduce-historyserver>
\n<\/yarn>
\n<zookeeper>
\n<!– ZOOKEEPER, zookeeper-server role (required if using HBase) **
\nlist an *ODD NUMBER*, comma separated ** –>
\n<!– Options to run zookeepers: Master nodes like Secondary NameNode,
\nResourceManager, HAWQ Standby node,etc EXCEPT NameNode–>
\n<zookeeper-server>phdnn1.domain.com,phdnn2.domain.com,
\nphddn1.domain.com<\/zookeeper-server>
\n<\/zookeeper>
\n<hbase>
\n<!– HBASE, master role –>
\n<hbase-master>phdnn1.domain.com<\/hbase-master>
\n<!– HBASE, region server role ** list of one or more, comma separated ** –>
\n<!– HBase region servers are generally deployed on the DataNode hosts –>
\n<hbase-regionserver>phddn1.domain.com,phddn2.domain.com,phddn3.domain.com,
\nphddn4.domain.com,phddn5.domain.com,phddn6.domain.com,phddn7.domain.com,
\nphddn8.domain.com,phddn9.domain.com,phddn10.domain.com<\/hbase-regionserver>
\n<\/hbase>
\n<hive>
\n<!– HIVE, server role –>
\n<!– Hive thrift server would be running on this machine. –>
\n<hive-server>phdnn1.domain.com<\/hive-server>
\n<!– HIVE, metastore role. This host will run an instance of PostgreSQL. –>
\n<!– Postgres database is installed on this node to maintain Hive metadata. –>
\n<hive-metastore>phdnn1.domain.com<\/hive-metastore>
\n<\/hive>
\n<hawq>
\n<!– HAWQ, hawq master role (mandatory). This should be a different host
\nthan the NameNode host. –>
\n<!– <hawq-master>host.yourdomain.com<\/hawq-master> –>
\n<!– HAWQ, hawq standbymaster role (optional) \u2013 ** comment out XML tag if not required **–>
\n<!– <hawq-standbymaster>host.yourdomain.com<\/hawq-standbymaster> –>
\n<!– HAWQ, hawq segments (mandatory) ** list of one or more, comma separated ** –>
\n<!– In many configurations, this will be the same list as the DataNode role. –>
\n<!– <hawq-segment>host.yourdomain.com<\/hawq-segment> –>
\n<\/hawq>
\n<\/hostRoleMapping>
\n<servicesConfigGlobals>
\n<!– List of one or more, comma separated, disk mount points on slave
\nnodes (datanode\/nodemanager\/regionservers) –>
\n<datanode.disk.mount.points>\/data1\/hdata,\/data2\/hdata,\/data3\/hdata,\/data4\/hdata,
\n\/data5\/hdata,\/data6\/hdata<\/datanode.disk.mount.points>
\n<!– List of one or more, comma separated, disk mount points on NameNode. More than one can be configured for NameNode local data redundancy. –>
\n<namenode.disk.mount.points>\/data1\/hdata<\/namenode.disk.mount.points>
\n<!– List of one or more, comma separated disk mount points on secondary-namenode. More than one can be configured for secondary-namenode local data redundancy. –>
\n<!– <secondary.namenode.disk.mount.points>\/data\/secondary_nn
\n<\/secondary.namenode.disk.mount.points> –>
\n<!– Max. memory on nodemanager collectively available to all the task containers
\nincluding application master. –>
\n<yarn.nodemanager.resource.memory-mb>49152<\/yarn.nodemanager.resource.memory-mb>
\n<!– Min memory required for the task or application manager container –>
\n<yarn.scheduler.minimum-allocation-mb>2048<\/yarn.scheduler.minimum-allocation-mb>
\n<!– THIS IS THE \u201cNameNode port\u201d. –>
\n<dfs.port>8020<\/dfs.port>
\n<!– This syntax correlates to a bash array data structure. There are no commas seperating the values. It\u2019s just whitespace. If you enter multiple entries with the exact same value; each entry will result in a HAWQ segment placed on a host. If you want two segments per hosts make two entries with the same values; which may or may not be placed on the same disk within the host. WHICH MAY OR MAY NOT BE ON THE SAME DISK. BY CONVENTION, THE LAST PART OF THIS STRING IS In the example below, two segments will be created on every HAWQ segment host –>
\n<!– <hawq.segment.directory>(\/data1\/primary \/data1\/primary)<\/hawq.segment.directory> –>
\n<!– THIS IS A SINGLE VALUE, AND IDEALLY THE DIRECTORY SPECIFIED
\nCORRELATES WITH A RAID VOLUME, AS THE
\nMASTER DATA DIRECTORY CONTAINS IMPORTANT METADATA. –>
\n<!– <hawq.master.directory>\/data1\/master<\/hawq.master.directory> –>
\n<!– IDEALLY, THIS DIRECTORY IS ON ITS OWN DEVICE, TO MINIMIZE I\/O
\nCONTENTION BETWEEN ZOOKEEPER
\nAND OTHER PROCESSES. –>
\n<zookeeper.data.dir>\/data1\/hdata\/zookeeper<\/zookeeper.data.dir>
\n<!– Zookeeper Client Port –>
\n<zookeeper.client.port>2181<\/zookeeper.client.port>
\n<!– JAVA_HOME defaults to this. Update if you have java installed elsewhere –>
\n<cluster_java_home>\/usr\/java\/latest\/<\/cluster_java_home>
\n<!–maximum amount of HEAP to use. Default is 1024–>
\n<dfs.namenode.heapsize.mb>4096<\/dfs.namenode.heapsize.mb>
\n<dfs.datanode.heapsize.mb>4096<\/dfs.datanode.heapsize.mb>
\n<yarn.resourcemanager.heapsize.mb>4096<\/yarn.resourcemanager.heapsize.mb>
\n<yarn.nodemanager.heapsize.mb>4096<\/yarn.nodemanager.heapsize.mb>
\n<hbase.heapsize.mb>4096<\/hbase.heapsize.mb>
\n<dfs.datanode.failed.volumes.tolerated>0<\/dfs.datanode.failed.volumes.tolerated>
\n<!– Choose a logical name for HA nameservice, for example \u201ctest\u201d. The name
\nyou choose will be used both for configuration
\nand as the authority component of absolute HDFS paths in the cluster –>
\n<nameservices>my-hd-cluster<\/nameservices>
\n<!– Choose ids for the two namenodes being used. These ids will be used in the configuration of properties related to these namenodes –>
\n<namenode1id>nn1<\/namenode1id>
\n<namenode2id>nn2<\/namenode2id>
\n<!–specify the path on quorum journal nodes where the shared data is written –>
\n<journalpath>\/data1\/hdata\/qjournal<\/journalpath>
\n<!–specify the port where quorum journal nodes should be run–>
\n<journalport>8485<\/journalport>
\n<\/servicesConfigGlobals>
\n<\/clusterConfig>[\/php]
\nCheck whether the xml formatting is correct:
\n[php]xmlwf clusterConfig.xml[\/php]<\/p>\n5.1. Enable namenode high-availability<\/h2>\n
To enable HA, you then need to make HA-specific edits to the following configuration files:<\/p>\n
- \n
- clusterConfig.xml<\/li>\n
- hdfs\/hdfs-site.xml<\/li>\n
- hdfs\/core-site.xml<\/li>\n
- hbase\/hbase-site.xml<\/li>\n
- yarn\/yarn-site.xml<\/li>\n<\/ul>\n
NOTE: When specifying the nameservices in the clusterConfig.xml, do not use underscores (\u2018_\u2019), for example, phd_cluster
\na. Edit clusterConfig.xml as follows:<\/p>\n- \n
- Comment out secondarynamenode role in hdfs service<\/li>\n
- Uncomment standbynamenode and journalnode roles in hdfs service<\/li>\n
- Uncomment nameservices, namenode1id, namenode2id, journalpath, and journalport entries in serviceConfigGlobals<\/li>\n<\/ul>\n
b. Edit hdfs\/hdfs-site.xml as follows:<\/p>\n
- \n
- Uncomment the following properties:<\/li>\n<\/ul>\n
[php]<property>
\n<name>dfs.nameservices<\/name>
\n<value>${nameservices}<\/value>
\n<\/property><property>
\n<name>dfs.ha.namenodes.${nameservices}<\/name>
\n<value>${namenode1id},${namenode2id}<\/value>
\n<\/property><property>
\n<name>dfs.namenode.rpc-address.${nameservices}.${namenode1id}<\/name>
\n<value>${namenode}:8020<\/value>
\n<\/property>
\n<property>
\n<name>dfs.namenode.rpc-address.${nameservices}.${namenode2id}<\/name>
\n<value>${standbynamenode}:8020<\/value>
\n<\/property>
\n<property>
\n<name>dfs.namenode.http-address.${nameservices}.${namenode1id}<\/name>
\n<value>${namenode}:50070<\/value>
\n<\/property>
\n<property>
\n<name>dfs.namenode.http-address.${nameservices}.${namenode2id}<\/name>
\n<value>${standbynamenode}:50070<\/value>
\n<\/property>
\n<property>
\n<name>dfs.namenode.shared.edits.dir<\/name>
\n<value>qjournal:\/\/${journalnode}\/${nameservices}<\/value>
\n<\/property>
\n<property>
\n<name>dfs.client.failover.proxy.provider.${nameservices}<\/name>
\n<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<\/value>
\n<\/property>
\n<property>
\n<name>dfs.ha.fencing.methods<\/name>
\n<value>sshfence
\nshell(\/bin\/true)
\n<\/value>
\n<\/property>
\n<property>
\n<name>dfs.ha.fencing.ssh.private-key-files<\/name>
\n<value>\/home\/hdfs\/.ssh\/id_rsa<\/value>
\n<\/property>
\n<property>
\n<name>dfs.journalnode.edits.dir<\/name>
\n<value>${journalpath}<\/value>
\n<\/property>
\n<!– Namenode Auto HA related properties –>
\n<property>
\n<name>dfs.ha.automatic-failover.enabled<\/name>
\n<value>true<\/value>
\n<\/property>
\n<!– END Namenode Auto HA related properties –>[\/php]<\/p>\n- \n
- Comment the following properties:<\/li>\n<\/ul>\n
[php]dfs.namenode.secondary.http-address
\n${secondarynamenode}:50090
\nThe secondary namenode http server address and port.[\/php]
\nc. Edit yarn\/yarn-site.xml:
\n[php]mapreduce.job.hdfs-servers
\nhdfs:\/\/${nameservices}[\/php]
\nd. Edit hdfs\/core-site.xml as follows:<\/p>\n- \n
- Set the following property key value:<\/li>\n<\/ul>\n
[php]fs.defaultFS
\nhdfs:\/\/${nameservices}
\nThe name of the default file system. A URI whosescheme and authority determine the FileSystem implementation. The uri\u2019s scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri\u2019s authority is used to determine the host, port, etc. for a filesystem.[\/php]<\/p>\n- \n
- Uncomment following property:<\/li>\n<\/ul>\n
[php]ha.zookeeper.quorum
\n${zookeeper-server}:${zookeeper.client.port}[\/php]
\ne. Edit hbase\/hbase-site.xml as follows:
\n[php]hbase.rootdir
\nhdfs:\/\/${nameservices}\/apps\/hbase\/data
\nThe directory shared by region servers and into which HBase persists. The URL should be \u2018fully-qualified\u2019 to include the filesystem scheme. For example, to specify the HDFS directory \u2018\/hbase\u2019 where the HDFS instance\u2019s namenode is running at namenode.example.org on port 9000, set this value to: hdfs:\/\/namenode.example.org:9000\/hbase. By default HBase writes into \/tmp. Change this configuration else all data will be lost on machine restart.[\/php]<\/p>\n6. Deploy the Cluster<\/h1>\n
Pivotal HD deploys clusters using input from the cluster configuration directory. This cluster configuration directory contains files that describes the topology and configuration for the cluster.
\nNOTE: Deploy the cluster as gpadmin.<\/em>
\nThe deploy command internally does following steps:<\/strong>
\nI. Prepares the cluster nodes with the prerequisites (internally runs preparehosts command)
\na) Creates the gpadmin user.
\nb) As gpadmin, sets up password-less SSH access from the Admin node.
\nc) Installs the provided Oracle Java JDK.
\nd) Disables SELinux across the cluster.
\ne) Optionally synchronizes the system clocks.
\nf) Installs Puppet version 2.7.20 (the one shipped with the PCC tarball, not the one from puppetlabs repo)
\ng) Installs sshpass.
\nII. Verifies the prerequisites (internally runs scanhosts command)
\nIII. Deploys the cluster<\/p>\n6.1. Troubleshooting<\/h2>\n
You can check the following log files to troubleshoot any failures:
\nOn Admin:
\n\/var\/log\/gphd\/gphdmgr\/GPHDClusterInstaller_XXX.log
\n\/var\/log\/gphd\/gphdmgr\/gphdmgr-webservices.log
\n\/var\/log\/messages
\n\/var\/log\/gphd\/gphdmgr\/installer.log
\nOn Cluster Nodes:
\n\/tmp\/GPHDNodeInstaller_XXX.log
\nDeploy Hadoop Cluster
\n[php]icm_client deploy -c ClusterConfigDir\/[\/php]<\/p>\n6.2. Start Hadoop Cluster<\/h2>\n
[php]icm_client start -l my-hd-cluster[\/php]
\nNow to check whether cluster is working correctly:
\nLogin to web-console:
\n[php]https:\/\/:5443
\nUserName: gpadmin
\nPassword: Gpadmin1[\/php]<\/p>\n6.3. Run sample program:<\/h2>\n
[php]# create input directory
\nhadoop fs -mkdir input# copy some file having text data to run word count
\nhadoop fs -copyFromLocal \/usr\/lib\/gphd\/hadoop\/CHANGES.txt input# run word count
\nhadoop jar \/usr\/lib\/gphd\/hadoop-mapreduce\/hadoop-mapreduce-examples.jar wordcount input output# dump output on console
\nhadoop fs -cat output\/part*[\/php]<\/p>\n7. Stop Hadoop Cluster:<\/h1>\n
[php]icm_client stop -l my-hd-cluster[\/php]<\/p>\n
8. Related Links:<\/h1>\n
- Uncomment following property:<\/li>\n<\/ul>\n
- Set the following property key value:<\/li>\n<\/ul>\n
- Comment the following properties:<\/li>\n<\/ul>\n
- Uncomment the following properties:<\/li>\n<\/ul>\n