MHA监控检测报错

MHA安装之后一直正常运行，中间也有出现过切换的情况，不过都正常进行，后有其他dba在mha自动切换后做过一些操作，但并未说做了哪些操作。我再查看状态时，发现监控没有在运行，于是手工启动，结果就报以下错误：

Mon Nov 20 17:15:11 2017 - [info] MHA::MasterMonitor version 0.56.Mon Nov 20 17:15:12 2017 - [info] GTID failover mode = 0Mon Nov 20 17:15:12 2017 - [info] Dead Servers:Mon Nov 20 17:15:12 2017 - [info] Alive Servers:Mon Nov 20 17:15:12 2017 - [info]   192.168.2.211(192.168.2.211:3206)Mon Nov 20 17:15:12 2017 - [info]   192.168.2.212(192.168.2.212:3206)Mon Nov 20 17:15:12 2017 - [info]   192.168.2.213(192.168.2.213:3206)Mon Nov 20 17:15:12 2017 - [info] Alive Slaves:Mon Nov 20 17:15:12 2017 - [info]   192.168.2.212(192.168.2.212:3206)  Version=5.6.24-log (oldest major version between slaves) log-bin:enabledMon Nov 20 17:15:12 2017 - [info]     Replicating from 192.168.2.211(192.168.2.211:3206)Mon Nov 20 17:15:12 2017 - [info]     Primary candidate for the new Master (candidate_master is set)Mon Nov 20 17:15:12 2017 - [info]   192.168.2.213(192.168.2.213:3206)  Version=5.6.24-log (oldest major version between slaves) log-bin:enabledMon Nov 20 17:15:12 2017 - [info]     Replicating from 192.168.2.211(192.168.2.211:3206)Mon Nov 20 17:15:12 2017 - [info]     Not candidate for the new Master (no_master is set)Mon Nov 20 17:15:12 2017 - [info] Current Alive Master: 192.168.2.211(192.168.2.211:3206)Mon Nov 20 17:15:12 2017 - [info] Checking slave configurations..Mon Nov 20 17:15:12 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.2.213(192.168.2.213:3206).Mon Nov 20 17:15:12 2017 - [info] Checking replication filtering settings..Mon Nov 20 17:15:12 2017 - [info]  binlog_do_db= , binlog_ignore_db=Mon Nov 20 17:15:12 2017 - [info]  Replication filtering check ok.Mon Nov 20 17:15:12 2017 - [info] GTID (with auto-pos) is not supportedMon Nov 20 17:15:12 2017 - [info] Starting SSH connection tests..Mon Nov 20 17:15:15 2017 - [info] All SSH connection tests passed successfully.Mon Nov 20 17:15:15 2017 - [info] Checking MHA Node version..Mon Nov 20 17:15:15 2017 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln122] Got error when getting node version. Error:Mon Nov 20 17:15:15 2017 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln123]bash: apply_diff_relay_logs: command not foundMon Nov 20 17:15:15 2017 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln150] node version on 192.168.2.212 not found! Is MHA Node package installed ? at /usr/local/share/perl5/MHA/MasterMonitor.pm line 374.Mon Nov 20 17:15:15 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. node version on 192.168.2.212 not found! Is MHA Node package installed ? at /usr/local/share/perl5/MHA/MasterMonitor.pm line 374.        ...propagated at /usr/local/share/perl5/MHA/ManagerUtil.pm line 151.Mon Nov 20 17:15:15 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.Mon Nov 20 17:15:15 2017 - [info] Got exit code 1 (Not master dead).

提示到不到命令，同时提示我是否节点2是否没有安装node包，我做过以下尝试：

1）尝试配置环境变量路径，错误依旧。

2）尝试重新安装node包，错误依旧。

网上也查找过报错信息，但都是千篇一律，一篇文章转来转去，没有对应的解决方法。

后来空了又仔细看了下，问题解决掉了，解决方法如下：

[root@kxt-mha-03 ~]# find / -name apply_diff_relay_logs/usr/local/bin/apply_diff_relay_logs[root@kxt-mha-03 ~]# ln -s /usr/local/bin/apply_diff_relay_logs  /usr/bin/[root@kxt-mha-03 ~]# ln -s /usr/local/bin/save_binary_logs   /usr/bin/[root@kxt-mha-02 ~]# ln -s /usr/local/bin/apply_diff_relay_logs  /usr/bin/[root@kxt-mha-02 ~]# ln -s /usr/local/bin/save_binary_logs   /usr/bin/[root@kxt-mha-01 ~]# ln -s /usr/local/bin/apply_diff_relay_logs  /usr/bin/[root@kxt-mha-01 ~]# ln -s /usr/local/bin/save_binary_logs   /usr/bin/再次做检测：Wed Feb 28 10:48:40 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Wed Feb 28 10:48:40 2018 - [info] Reading application default configuration from /etc/mha/conf/app1.cnf..Wed Feb 28 10:48:40 2018 - [info] Reading server configuration from /etc/mha/conf/app1.cnf..Wed Feb 28 10:48:40 2018 - [info] MHA::MasterMonitor version 0.56.Wed Feb 28 10:48:41 2018 - [info] GTID failover mode = 0Wed Feb 28 10:48:41 2018 - [info] Dead Servers:Wed Feb 28 10:48:41 2018 - [info] Alive Servers:Wed Feb 28 10:48:41 2018 - [info]   192.168.2.211(192.168.2.211:3206)Wed Feb 28 10:48:41 2018 - [info]   192.168.2.212(192.168.2.212:3206)Wed Feb 28 10:48:41 2018 - [info]   192.168.2.213(192.168.2.213:3206)Wed Feb 28 10:48:41 2018 - [info] Alive Slaves:Wed Feb 28 10:48:41 2018 - [info]   192.168.2.212(192.168.2.212:3206)  Version=5.6.24-log (oldest major version between slaves) log-bin:enabledWed Feb 28 10:48:41 2018 - [info]     Replicating from 192.168.2.211(192.168.2.211:3206)Wed Feb 28 10:48:41 2018 - [info]     Primary candidate for the new Master (candidate_master is set)Wed Feb 28 10:48:41 2018 - [info]   192.168.2.213(192.168.2.213:3206)  Version=5.6.24-log (oldest major version between slaves) log-bin:enabledWed Feb 28 10:48:41 2018 - [info]     Replicating from 192.168.2.211(192.168.2.211:3206)Wed Feb 28 10:48:41 2018 - [info]     Not candidate for the new Master (no_master is set)Wed Feb 28 10:48:41 2018 - [info] Current Alive Master: 192.168.2.211(192.168.2.211:3206)Wed Feb 28 10:48:41 2018 - [info] Checking slave configurations..Wed Feb 28 10:48:41 2018 - [warning]  relay_log_purge=0 is not set on slave 192.168.2.213(192.168.2.213:3206).Wed Feb 28 10:48:41 2018 - [info] Checking replication filtering settings..Wed Feb 28 10:48:41 2018 - [info]  binlog_do_db= , binlog_ignore_db= Wed Feb 28 10:48:41 2018 - [info]  Replication filtering check ok.Wed Feb 28 10:48:41 2018 - [info] GTID (with auto-pos) is not supportedWed Feb 28 10:48:41 2018 - [info] Starting SSH connection tests..Wed Feb 28 10:48:44 2018 - [info] All SSH connection tests passed successfully.Wed Feb 28 10:48:44 2018 - [info] Checking MHA Node version..Wed Feb 28 10:48:44 2018 - [info]  Version check ok.Wed Feb 28 10:48:44 2018 - [info] Checking SSH publickey authentication settings on the current master..Wed Feb 28 10:48:44 2018 - [info] HealthCheck: SSH to 192.168.2.211 is reachable.Wed Feb 28 10:48:45 2018 - [info] Master MHA Node version is 0.56.Wed Feb 28 10:48:45 2018 - [info] Checking recovery script configurations on 192.168.2.211(192.168.2.211:3206)..Wed Feb 28 10:48:45 2018 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data --output_file=/etc/mha/app1/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.013591 Wed Feb 28 10:48:45 2018 - [info]   Connecting to root@192.168.2.211(192.168.2.211:22)..   Creating /etc/mha/app1 if not exists..    ok.  Checking output directory is accessible or not..   ok.  Binlog found at /data/mysql/data, up to mysql-bin.013591Wed Feb 28 10:48:45 2018 - [info] Binlog setting check done.Wed Feb 28 10:48:45 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..Wed Feb 28 10:48:45 2018 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.2.212 --slave_ip=192.168.2.212 --slave_port=3206 --workdir=/etc/mha/app1 --target_version=5.6.24-log --manager_version=0.56 --relay_dir=/data/mysql/data --current_relay_log=mysql-relay-bin.000091  --slave_pass=xxxWed Feb 28 10:48:45 2018 - [info]   Connecting to root@192.168.2.212(192.168.2.212:22)..   Checking slave recovery environment settings..    Relay log found at /data/mysql/data, up to mysql-relay-bin.000091    Temporary relay log file is /data/mysql/data/mysql-relay-bin.000091    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done.    Testing mysqlbinlog output.. done.    Cleaning up test file(s).. done.Wed Feb 28 10:48:46 2018 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.2.213 --slave_ip=192.168.2.213 --slave_port=3206 --workdir=/etc/mha/app1 --target_version=5.6.24-log --manager_version=0.56 --relay_dir=/data/mysql/data --current_relay_log=mysql-relay-bin.020034  --slave_pass=xxxWed Feb 28 10:48:46 2018 - [info]   Connecting to root@192.168.2.213(192.168.2.213:22)..   Checking slave recovery environment settings..    Relay log found at /data/mysql/data, up to mysql-relay-bin.020034    Temporary relay log file is /data/mysql/data/mysql-relay-bin.020034    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done.    Testing mysqlbinlog output.. done.    Cleaning up test file(s).. done.Wed Feb 28 10:48:46 2018 - [info] Slaves settings check done.Wed Feb 28 10:48:46 2018 - [info] 192.168.2.211(192.168.2.211:3206) (current master) +--192.168.2.212(192.168.2.212:3206) +--192.168.2.213(192.168.2.213:3206)Wed Feb 28 10:48:46 2018 - [info] Checking replication health on 192.168.2.212..Wed Feb 28 10:48:46 2018 - [info]  ok.Wed Feb 28 10:48:46 2018 - [info] Checking replication health on 192.168.2.213..Wed Feb 28 10:48:46 2018 - [info]  ok.Wed Feb 28 10:48:46 2018 - [warning] master_ip_failover_script is not defined.Wed Feb 28 10:48:46 2018 - [warning] shutdown_script is not defined.Wed Feb 28 10:48:46 2018 - [info] Got exit code 0 (Not master dead).MySQL Replication Health is OK.

至此问题解决。