Hapoop 搭建 （八）YARN HA搭建

Hapoop 搭建 （三）hadoop集群搭建

Hapoop 搭建 （五）搭建zookeeper集群环境

Hapoop 搭建 （七）HDFS HA搭建

结构以及服务器角色

 

一、yarn-site.xml 文件配置

在centos01的hadoop安装目录的配置文件夹中 配置 yarn-site.xml (路径：/opt/modules/hadoop-2.8.2/etc/hadoop )，完整文件如下：

<configuration>
    <!--指定在YARN上面可以运行MapReduce程序 -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
   
    <!--  YARN HA 配置-->
    <!-- 开启ResourceManager HA高可用 -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <!-- 指定RM的cluster id -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yarncluster</value>
    </property>
    <!-- 集群中ResourceManager的Id列表 -->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

    <!-- 分别指定RM的地址 所在节点主机名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>centos01</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>centos02</value>
    </property>
   <!-- 分别指定web页面的访问地址 -->
   <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>centos01:8088</value>
    </property>
   <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>centos02:8088</value>
    </property>

    <!-- 指定zk集群地址 -->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>centos01:2181,centos02:2181,centos03:2181</value>
    </property>

    <!-- 启用自动恢复 -->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <!-- 指定resourcemanager的状态信息存储在zookeeper集群上 -->
    <property>
        <name>yarn.resourcemanager.store.class</name>
     <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>86400</value>
    </property>

</configuration>

 

二、同步到其它节点

将yarn-site.xml配置文件复制到另2个节点上

scp /opt/modules/hadoop-2.8.2/etc/hadoop/yarn-site.xml hadoop@centos02:/opt/modules/hadoop-2.8.2/etc/hadoop/
scp /opt/modules/hadoop-2.8.2/etc/hadoop/yarn-site.xml hadoop@centos03:/opt/modules/hadoop-2.8.2/etc/hadoop/

 

三、启动zookeeper集群

停止
sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh stop
启动
sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh start

[hadoop@centos01 ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: leader
[hadoop@centos01 ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
[hadoop@centos01 ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
[hadoop@centos01 ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[hadoop@centos01 ~]$ 

 

四、启动YARN集群

分别进入centos01、centos02的hadoop的安装目录启动ResourceManager

sh /opt/modules/hadoop-2.8.2/sbin/yarn-daemon.sh start resourcemanager

 进入3各节点的hadoop安装目录启动NodeManager

sh /opt/modules/hadoop-2.8.2/sbin/yarn-daemon.sh start nodemanager

 

五、查看节点进程

[hadoop@centos01 sbin]$ jps
96960 ResourceManager
64850 JournalNode
97427 NodeManager
64164 DataNode
34004 QuorumPeerMain
65305 DFSZKFailoverController
98876 Jps
36702 NameNode

[hadoop@centos02 ~]$ jps
59364 DataNode
84087 NodeManager
83686 ResourceManager
85798 Jps
61513 DFSZKFailoverController
20457 QuorumPeerMain
59980 JournalNode
82735 NameNode

[hadoop@centos03 hadoop]$ jps
62977 NodeManager
49509 JournalNode
65191 Jps
130810 QuorumPeerMain
48845 DataNode

 

六、测试YARN故障转移

访问centos01活动的ResourceManager

http://192.168.0.171:8088/cluster

 

 

访问centos02活动的ResourceManager，此时活动的是centos01所以会自动跳转到centos01上

http://192.168.0.172:8088/cluster

[hadoop@centos01 sbin]$ jps
96960 ResourceManager
64850 JournalNode
97427 NodeManager
64164 DataNode
34004 QuorumPeerMain
65305 DFSZKFailoverController
98876 Jps
36702 NameNode
[hadoop@centos01 sbin]$ kill -9 96960
[hadoop@centos01 sbin]$ 

自动切换到centos02上