说说Keepalived的脑裂
1. 工作场景
Keepalived提供了Loadbalancing和High-Availability的功能, 本文说的是其为2个Mycat节点提供HA功能的场景.
2. 关键配置如下, 为主备非抢占模式.
! mycat01, 192.168.4.196
global_defs {
# 一个keepalived.conf, 对应一个router_id
router_id mycat01
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface eth0
# 一个组播内, 各节点该值相同
virtual_router_id 196
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass Zf4aqy
}
virtual_ipaddress {
192.168.4.200
}
}
! mycat02, 192.168.4.195
global_defs {
router_id mycat02
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface eth0
virtual_router_id 196
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass Zf4aqy
}
virtual_ipaddress {
192.168.4.200
}
}
3. Keepalived提供HA的工作原理
Keepalived的HA功能, 是通过VRRP(Virtual Router Redundancy Protocol, 虚拟路由冗余协议)来实现的. 其用IP组播的方式(默认组播地址: 224.0.0.18), 实现服务节点间的通信, 通过一种竞选机制来将路由任务交给某台VRRP路由器. 工作时主节点发送VRRP协议报文, 备节点接收报文, 若一段时间(默认3个报文发送时间)备节点接收不到主节点发送的报文, 就会启动接管程序接管主节点的资源. 备节点可以有多个, 通过优先级竞选.
4. Keepalived发生脑裂的情况
4.1 根据上述工作原理知道, 备节点接收不到报文时, 如两者间的网络不通了, 备节点就会启动接管程序接管主节点的资源, 对外提供服务, 表现形式就是备节点上出现了虚拟IP, 此时主节点也是持有虚拟IP的.
问题重现如下, 此时mycat01为主节点, mycat02为备节点, 在mycat02上drop掉组播数据包.
# iptables -A INPUT -m pkttype --pkt-type multicast -j DROP
然后观察message日志, 可见其也变成了主节点.
# tail -f /var/log/messages
Mar 17 00:13:58 mycat02 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Mar 17 00:14:00 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.4.200
Mar 17 00:14:01 mycat02 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.200 added
Mar 17 00:14:06 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.4.200
在mycat01上抓包, 看到mycat02也在向组播地址发送数据包.
# tcpdump -nnei eth0 | grep -i 'vrid 196'
00:13:58.735738 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20
00:13:59.736268 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20
00:14:00.736800 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20
00:14:00.955581 fa:16:3e:5d:23:10 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.195 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20
00:14:01.956340 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20
当把iptables停掉后, mycat01和mycat02扔分别持有虚拟IP 192.168.4.200, 这也说明了主备非抢占模式的一个弊端, 主备抢占模式是不会这样的, 下面是其主要配置.
! mycat01, 192.168.4.196
global_defs {
router_id mycat01
}
vrrp_instance VI_1 {
state MASTER
virtual_router_id 196
priority 200
}
! mycat02, 192.168.4.195
global_defs {
router_id mycat02
}
vrrp_instance VI_1 {
state BACKUP
virtual_router_id 196
priority 150
}
4.2 同一个组播内, virtual_router_id配置不相同, 也会发生脑裂现象, 这里就不在赘述了.
说到最后, 如何避免脑裂现象呢, 其实可以看到, 只要有虚拟IP存在, 就不可能完全规避这个问题. 也就是没有虚拟IP了, 就没有脑裂了, 那么节点又如何向外提供服务呢? 引入Zookeeper或Consul的服务发现机制, 这些又是另外的事情了...
若感兴趣可关注订阅号"数据库最佳实践"(DBBestPractice).
