导航：首页 > 互联网科技 >

hadoop官方文档学习笔记（1）——resource manager HA

发表于：2025-01-24 作者：千家信息网编辑

千家信息网最后更新 2025年01月24日，resource manager HA是hadoop自从2.4之后推出的功能，以Active/Standby的方式提供冗余，目的是为了消除单点失败的风险。1、总体架构：2、故障切换：有自动和手动两种形

千家信息网最后更新 2025年01月24日hadoop官方文档学习笔记（1）——resource manager HA

resource manager HA是hadoop自从2.4之后推出的功能，以Active/Standby的方式提供冗余，目的是为了消除单点失败的风险。

1、总体架构：

2、故障切换：有自动和手动两种形式。

手动：如果以手动形式切换，使用yarn haadmin命令首先将Active节点转为standby，再将standby节点转为active。

自动：RM有基于zookeeper的节点选举机制决定哪一个是活动节点。不需要像HDFS一样部署一个zkfc守护进程，因为RM内嵌了这样的功能。

做了rm的HA之后，所有节点和客户端都要列出所有RM节点，连接时会用轮询的方式遍历，直到找到一个active的节点。如果活动节点down了，它们会继续轮询。这一动作被实现为org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider类。可以通过重新实现该类，并在yarn.client.failover-proxy-provider 配置项中指定新的类名来改写这一行为逻辑。

3、配置

Configuration Properties	Description
yarn.resourcemanager.zk-address	Address of the ZK-quorum. Used both for the state-store and embedded leader-election.
yarn.resourcemanager.ha.enabled	Enable RM HA.
yarn.resourcemanager.ha.rm-ids	List of logical IDs for the RMs. e.g., "rm1,rm2".
yarn.resourcemanager.hostname.rm-id	For each rm-id, specify the hostname the RM corresponds to. Alternately, one could set each of the RM's service addresses.
yarn.resourcemanager.address.rm-id	For each rm-id, specify host:port for clients to submit jobs. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.scheduler.address.rm-id	For each rm-id, specify scheduler host:port for ApplicationMasters to obtain resources. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.resource-tracker.address.rm-id	For each rm-id, specify host:port for NodeManagers to connect. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.admin.address.rm-id	For each rm-id, specify host:port for administrative commands. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.webapp.address.rm-id	For each rm-id, specify host:port of the RM web application corresponds to. You do not need this if you set yarn.http.policy to HTTPS_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.webapp.https.address.rm-id	For each rm-id, specify host:port of the RM https web application corresponds to. You do not need this if you set yarn.http.policy to HTTP_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.ha.id	Identifies the RM in the ensemble. This is optional; however, if set, admins have to ensure that all the RMs have their own IDs in the config.
yarn.resourcemanager.ha.automatic-failover.enabled	Enable automatic failover; By default, it is enabled only when HA is enabled.
yarn.resourcemanager.ha.automatic-failover.embedded	Use embedded leader-elector to pick the Active RM, when automatic failover is enabled. By default, it is enabled only when HA is enabled.
yarn.resourcemanager.cluster-id	Identifies the cluster. Used by the elector to ensure an RM doesn't take over as Active for another cluster.
yarn.client.failover-proxy-provider	The class to be used by Clients, AMs and NMs to failover to the Active RM.
yarn.client.failover-max-attempts	The max number of times FailoverProxyProvider should attempt failover.
yarn.client.failover-sleep-base-ms	The sleep base (in milliseconds) to be used for calculating the exponential delay between failovers.
yarn.client.failover-sleep-max-ms	The maximum sleep time (in milliseconds) between failovers.
yarn.client.failover-retries	The number of retries per attempt to connect to a ResourceManager.
yarn.client.failover-retries-on-socket-timeouts	The number of retries per attempt to connect to a ResourceManager on socket timeouts.

4、示例（最小配置）


  yarn.resourcemanager.ha.enabled
  true


  yarn.resourcemanager.cluster-id
  cluster1


  yarn.resourcemanager.ha.rm-ids
  rm1,rm2


  yarn.resourcemanager.hostname.rm1
  master1


  yarn.resourcemanager.hostname.rm2
  master2


  yarn.resourcemanager.webapp.address.rm1
  master1:8088


  yarn.resourcemanager.webapp.address.rm2
  master2:8088


  yarn.resourcemanager.zk-address
  zk1:2181,zk2:2181,zk3:2181

5、管理命令


查看节点状态：
 $ yarn rmadmin -getServiceState rm1
 active

 $ yarn rmadmin -getServiceState rm2
 standby

故障切换：

 $ yarn rmadmin -transitionToStandby rm1
 $ yarn rmadmin -transitionToActive rm2

注意：
  开启自动故障切换后，系统为防止造成脑裂或其它不一致的状态，会拒绝人为管理HA状态。如果非常清楚自己的行为，可以在切换命令中指定-forcemanual选项。

很赞哦！