Hadoop-2.6.0 集群的 安装与配置
1. 配置节点bonnie1 hadoop环境
(1) 下载hadoop- 2.6.0 并解压缩
[root@bonnie1 ~]# wget http://apache.fayea.com/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz -C /usr/local
[root@bonnie1 ~]# cd /usr/local/
[root@bonnie1 local]# tar -xvf hadoop-2.6.0.tar
(2) 配置环境变量
[root@bonnie1 local]# vi /etc/profile
export JAVA_HOME=/usr/local/jdk1.7.0_79
export HADOOP_HOME=/usr/local/hadoop-2.6.0
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.6
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[root@bonnie1 local]# cd hadoop-2.6.0/etc/hadoop/
[root@bonnie1 hadoop]# vi hadoop-env.sh
＃ 追加如下字段
export JAVA_HOME=/usr/local/jdk1.7.0_79
export HADOOP_PREFIX=/usr/local/hadoop-2.6.0
[root@bonnie1 hadoop]# vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.6.0/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>bonnie1:2181,bonnie2:2181,bonnie3:2181</value>
</property>
</configuration>
[root@bonnie1 hadoop]# vi hdfs-site.xml
<configuration>
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>bonnie1:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>bonnie1:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>bonnie2:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>bonnie2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://bonnie1:8485;bonnie2:8485;bonnie3:8485/ns1</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/hadoop-2.6.0/journal</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
[root@bonnie1 hadoop]# vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
[root@bonnie1 hadoop]# vi yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bonnie3</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
[root@bonnie1 hadoop]# vi slaves
bonnie1
bonnie2
bonnie3
2. 将配置好的hadoop文件拷贝到其他几个节点
[root@bonnie1 hadoop]# cd /usr/local
[root@bonnie1 local]# scp -r hadoop-2.6.0 bonnie2:/usr/local/
3. 集群初始化
(1) ＃启动 zookeeper ，zookeeper 安装见上一篇 zookeeper 安装与配置
[root@bonnie1 hadoop]# cd /usr/local/zookeeper-3.4.6/bin/
[root@bonnie1 bin]# ./zkServer.sh start
[root@bonnie2 hadoop]# cd /usr/local/zookeeper-3.4.6/bin/
[root@bonnie2 bin]# ./zkServer.sh start
[root@bonnie3 hadoop]# cd /usr/local/zookeeper-3.4.6/bin/
[root@bonnie3 bin]# ./zkServer.sh start
＃ 查看 zookeeper 状态
[root@bonnie1 bin]# ./zkServer.sh status
[root@bonnie2 bin]# ./zkServer.sh status
[root@bonnie3 bin]# ./zkServer.sh status
一个leader，两个follower
(2) 启动journalnode
* 分别在bonnie1、bonnie2、bonnie3上执行启动命令
[root@bonnie1 ~]# cd /usr/local/hadoop-2.6.0/sbin
[root@bonnie1 sbin]# ./hadoop-daemon.sh start journalnode
[root@bonnie2 sbin]# ./hadoop-daemon.sh start journalnode
[root@bonnie3 sbin]# ./hadoop-daemon.sh start journalnode
* 运行jps命令检验是否存在启动进程JournalNode
[root@bonnie1 sbin]# jps
5335 JournalNode
7303 Jps
3124 QuorumPeerMain
(3) 格式化HDFS
* 在节点bonnie1上执行格式化命令
[root@bonnie3 sbin]＃ hdfs namenode -format
＃ 出现如下日志表示成功
16/12/04 14:19:38 INFO common.Storage: Storage directory /usr/local/hadoop-2.6.0/tmp/dfs/name has been successfully formatted.
HA两节点保持同步，将invin01上的tmp目录拷贝到invin02节点
格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件，这里我配置的是/usr/local/hadoop-2.6.0/tmp，然后将/usr/local/hadoop-2.6.0/tmp拷贝到bonnie2的/usr/local/hadoop-2.6.0/下
[root@bonnie1 hadoop-2.6.0]# scp -r tmp/ bonnie2:/usr/local/hadoop-2.6.0/
(4) 格式化ZK
[root@bonnie1 hadoop-2.6.0]# hdfs zkfc -formatZK
# 出现如下日志表示成功
16/12/04 14:20:48 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
4. 启动集群
(1) jps （kill －9 ［进程号］）关闭（除QuorumPeerMain）所有进程
(2) 启动HDFS(在bonnie1上执行)
sbin/start-dfs.sh
(3) 启动YARN(在bonnie3上执行)
sbin/start-yarn.sh
5. hadoop-2.6.0集群配置完毕，通过浏览器访问验证
http://10.211.55.21:50070
NameNode 'bonnie1:9000' (active)
http://10.211.55.22:50070
NameNode 'bonnie2:9000' (standby)
http://10.211.55.23:8088
6. HDFS测试－上传下载文件
hadoop fs -mkdir /tmp/input ＃在HDFS上新建文件夹
hadoop fs -put input1.txt /tmp/input ＃把本地文件input1.txt传到HDFS的/tmp/input目录
hadoop fs -get input1.txt /tmp/input/input1.txt ＃把HDFS文件拉到本地
hadoop fs -ls /tmp/output ＃列出HDFS的某目录
hadoop fs -cat /tmp/ouput/output1.txt ＃查看HDFS上的文件
hadoop fs -rm -r /home/less/hadoop/tmp/output ＃删除HDFS上的目录
# 查看HDFS状态，比如有哪些datanode，每个datanode的情况
hadoop dfsadmin -report
hadoop dfsadmin -safemode leave # 离开安全模式
hadoop dfsadmin -safemode enter # 进入安全模式
7. YARN测试－WordCount
vi test.csv
hello tomcat
help yumily
cat bonnie
上传测试数据文件
hadoop fs -put test.csv /
运行wordcount
cd hadoop-2.6.0/share/hadoop/mapreduce/
hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /test.csv /out
8. Hadoop启动与关闭流程
(1) 启动流程
分别在bonnie1,bonnie2,bonnie3上执行启动命令
[root@bonnie1 ~]# cd /usr/local/zookeeper-3.4.6/bin/
[root@bonnie1 bin]# ./zkServer.sh start
(2) 查看各ZK节点状态
[root@bonnie1 bin]# ./zkServer.sh status
一个leader，两个follower
(3) 启动HDFS(在invin01上执行)
[root@bonnie1 hadoop-2.6.0]# sbin/start-dfs.sh
(4) 启动YARN
[root@bonnie1 hadoop-2.6.0]# sbin/start-yarn.sh
9 关闭流程
(1)关闭HDFS(在invin01上执行)
[root@bonnie1 hadoop-2.6.0]#sbin/stop-dfs.sh
(2)关闭YARN(在invin03上执行)
[root@bonnie1 hadoop-2.6.0]# sbin/stop-yarn.sh
(3)关闭ZK(在invin04，invin05，invin06上执行)
[root@bonnie1 hadoop-2.6.0]# cd /home/hduser/zookeeper-3.4.6/bin/
[root@bonnie1 bin]# ./zkServer.sh stop
