Yarn使用ZK做高可用

1. 目前可用的YARN resourceManager状态存储实现类有如下三个：

org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore # a ZooKeeper based state-store implementation .
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore # a Hadoop FileSystem based state-store implementation like HDFS and local FS , default . 
org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore # a LevelDB based state-store implementation

 

2.  相关配置(最小配置，相关官网说明地址：https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html)

<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>cluster1</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>master1</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>master2</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>master1:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>master2:8088</value>
</property>
<property>
  <name>hadoop.zk.address</name>
  <value>zk1:2181,zk2:2181,zk3:2181</value>
</property>

 

3. 在zk中存储的信息如下：

get /yarn-leader-election/yarn-rm-cluster/ActiveBreadCrumb

yarn-rm-clusterrm1
cZxid = 0xca0e11dc4
ctime = Thu Sep 08 20:31:40 CST 2022
mZxid = 0xca0e11dc4
mtime = Thu Sep 08 20:31:40 CST 2022
pZxid = 0xca0e11dc4
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 22
numChildren = 0

 

get /yarn-leader-election/yarn-rm-cluster/ActiveStandbyElectorLock

yarn-rm-clusterrm1
cZxid = 0xca0e11dc3
ctime = Thu Sep 08 20:31:40 CST 2022
mZxid = 0xca0e11dc3
mtime = Thu Sep 08 20:31:40 CST 2022
pZxid = 0xca0e11dc3
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x48243f368178603
dataLength = 22
numChildren = 0

 

主备切换参考：

https://www.cnblogs.com/wttttt/p/7652852.html