orchestrator 高可用yum快速安装过程

[root@mgr1 ~]# curl -s https://packagecloud.io/install/repositories/github/orchestrator/script.rpm.sh | sudo bashDetected operating system as centos/7.Checking for curl...Detected curl...Downloading repository file: https://packagecloud.io/install/repositories/github/orchestrator/config_file.repo?os=centos&dist=7&source=scriptdone.Installing pygpgme to verify GPG signatures...Loaded plugins: fastestmirrorDetermining fastest mirrors * base: mirror.dal.nexril.net * extras: mirrors.huaweicloud.com * updates: mirrors.tuna.tsinghua.edu.cnbase                                                                                                                                                                      | 3.6 kB  00:00:00     extras                                                                                                                                                                    | 3.4 kB  00:00:00     github_orchestrator-source/signature                                                                                                                                      |  819 B  00:00:00     Retrieving key from https://packagecloud.io/github/orchestrator/gpgkeyImporting GPG key 0x7AC40831: Userid     : "https://packagecloud.io/github/orchestrator (https://packagecloud.io/docs#gpg_signing) " Fingerprint: 1580 fbdf 6d61 7952 e2e5 e859 f3e4 3403 7ac4 0831 From       : https://packagecloud.io/github/orchestrator/gpgkeygithub_orchestrator-source/signature                                                                                                                                      |  951 B  00:00:00 !!! proxysql_repo                                                                                                                                                             | 2.9 kB  00:00:00     updates                                                                                                                                                                   | 3.4 kB  00:00:00     (1/3): extras/7/x86_64/primary_db                                                                                                                                         | 215 kB  00:00:01     (2/3): proxysql_repo/7/primary_db                                                                                                                                         | 7.5 kB  00:00:02     (3/3): updates/7/x86_64/primary_db                                                                                                                                        | 7.4 MB  00:00:19     github_orchestrator-source/primary                                                                                                                                        |  175 B  00:00:01     Package pygpgme-0.3-9.el7.x86_64 already installed and latest versionNothing to doInstalling yum-utils...Loaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirror.dal.nexril.net * extras: mirrors.huaweicloud.com * updates: mirrors.tuna.tsinghua.edu.cnResolving Dependencies--> Running transaction check---> Package yum-utils.noarch 0:1.1.31-50.el7 will be installed--> Processing Dependency: python-kitchen for package: yum-utils-1.1.31-50.el7.noarch--> Processing Dependency: libxml2-python for package: yum-utils-1.1.31-50.el7.noarch--> Running transaction check---> Package libxml2-python.x86_64 0:2.9.1-6.el7_2.3 will be installed---> Package python-kitchen.noarch 0:1.1.1-5.el7 will be installed--> Processing Dependency: python-chardet for package: python-kitchen-1.1.1-5.el7.noarch--> Running transaction check---> Package python-chardet.noarch 0:2.2.1-1.el7_1 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================================================================================================================================= Package                                           Arch                                      Version                                               Repository                               Size=================================================================================================================================================================================================Installing: yum-utils                                         noarch                                    1.1.31-50.el7                                         base                                    121 kInstalling for dependencies: libxml2-python                                    x86_64                                    2.9.1-6.el7_2.3                                       base                                    247 k python-chardet                                    noarch                                    2.2.1-1.el7_1                                         base                                    227 k python-kitchen                                    noarch                                    1.1.1-5.el7                                           base                                    267 kTransaction Summary=================================================================================================================================================================================================Install  1 Package (+3 Dependent packages)Total download size: 861 kInstalled size: 4.3 MDownloading packages:(1/4): libxml2-python-2.9.1-6.el7_2.3.x86_64.rpm                                                                                                                          | 247 kB  00:00:01     (2/4): python-chardet-2.2.1-1.el7_1.noarch.rpm                                                                                                                            | 227 kB  00:00:02     (3/4): yum-utils-1.1.31-50.el7.noarch.rpm                                                                                                                                 | 121 kB  00:00:03     (4/4): python-kitchen-1.1.1-5.el7.noarch.rpm                                                                                                                              | 267 kB  00:00:03     -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Total                                                                                                                                                            251 kB/s | 861 kB  00:00:03     Running transaction checkRunning transaction testTransaction test succeededRunning transaction  Installing : python-chardet-2.2.1-1.el7_1.noarch                                                                                                                                           1/4   Installing : python-kitchen-1.1.1-5.el7.noarch                                                                                                                                             2/4   Installing : libxml2-python-2.9.1-6.el7_2.3.x86_64                                                                                                                                         3/4   Installing : yum-utils-1.1.31-50.el7.noarch                                                                                                                                                4/4   Verifying  : libxml2-python-2.9.1-6.el7_2.3.x86_64                                                                                                                                         1/4   Verifying  : python-kitchen-1.1.1-5.el7.noarch                                                                                                                                             2/4   Verifying  : yum-utils-1.1.31-50.el7.noarch                                                                                                                                                3/4   Verifying  : python-chardet-2.2.1-1.el7_1.noarch                                                                                                                                           4/4 Installed:  yum-utils.noarch 0:1.1.31-50.el7                                                                                                                                                               Dependency Installed:  libxml2-python.x86_64 0:2.9.1-6.el7_2.3                           python-chardet.noarch 0:2.2.1-1.el7_1                           python-kitchen.noarch 0:1.1.1-5.el7                          Complete!Generating yum cache for github_orchestrator...Importing GPG key 0x7AC40831: Userid     : "https://packagecloud.io/github/orchestrator (https://packagecloud.io/docs#gpg_signing) " Fingerprint: 1580 fbdf 6d61 7952 e2e5 e859 f3e4 3403 7ac4 0831 From       : https://packagecloud.io/github/orchestrator/gpgkeyGenerating yum cache for github_orchestrator-source...The repository is setup! You can now install packages.[root@mgr1 ~]# yum install orchestrator*Loaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirror.dal.nexril.net * extras: mirrors.huaweicloud.com * updates: mirrors.tuna.tsinghua.edu.cnResolving Dependencies--> Running transaction check---> Package orchestrator.x86_64 1:3.1.1-1 will be installed--> Processing Dependency: jq >= 1.5 for package: 1:orchestrator-3.1.1-1.x86_64---> Package orchestrator-cli.x86_64 1:3.1.1-1 will be installed--> Processing Dependency: jq >= 1.5 for package: 1:orchestrator-cli-3.1.1-1.x86_64---> Package orchestrator-client.x86_64 1:3.1.1-1 will be installed--> Processing Dependency: jq >= 1.5 for package: 1:orchestrator-client-3.1.1-1.x86_64--> Finished Dependency ResolutionError: Package: 1:orchestrator-cli-3.1.1-1.x86_64 (github_orchestrator)           Requires: jq >= 1.5Error: Package: 1:orchestrator-3.1.1-1.x86_64 (github_orchestrator)           Requires: jq >= 1.5Error: Package: 1:orchestrator-client-3.1.1-1.x86_64 (github_orchestrator)           Requires: jq >= 1.5 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest[root@mgr1 ~]# yum -y install epel-releaseLoaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirror.dal.nexril.net * extras: mirrors.huaweicloud.com * updates: mirrors.tuna.tsinghua.edu.cnResolving Dependencies--> Running transaction check---> Package epel-release.noarch 0:7-11 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================================================================================================================================= Package                                            Arch                                         Version                                      Repository                                    Size=================================================================================================================================================================================================Installing: epel-release                                       noarch                                       7-11                                         extras                                        15 kTransaction Summary=================================================================================================================================================================================================Install  1 PackageTotal download size: 15 kInstalled size: 24 kDownloading packages:epel-release-7-11.noarch.rpm                                                                                                                                              |  15 kB  00:00:00     Running transaction checkRunning transaction testTransaction test succeededRunning transaction  Installing : epel-release-7-11.noarch                                                                                                                                                      1/1   Verifying  : epel-release-7-11.noarch                                                                                                                                                      1/1 Installed:  epel-release.noarch 0:7-11                                                                                                                                                                     Complete![root@mgr1 ~]# yum install orchestrator*   Loaded plugins: fastestmirrorLoading mirror speeds from cached hostfileepel/x86_64/metalink                                                                                                                                                      | 6.3 kB  00:00:00      * base: mirror.dal.nexril.net * epel: mirrors.tuna.tsinghua.edu.cn * extras: mirrors.huaweicloud.com * updates: mirrors.tuna.tsinghua.edu.cnepel                                                                                                                                                                      | 5.3 kB  00:00:00     (1/3): epel/x86_64/group_gz                                                                                                                                               |  88 kB  00:00:01     (2/3): epel/x86_64/updateinfo                                                                                                                                             | 993 kB  00:00:06     (3/3): epel/x86_64/primary_db                                                                                                                                             | 6.8 MB  00:00:40     Resolving Dependencies--> Running transaction check---> Package orchestrator.x86_64 1:3.1.1-1 will be installed--> Processing Dependency: jq >= 1.5 for package: 1:orchestrator-3.1.1-1.x86_64---> Package orchestrator-cli.x86_64 1:3.1.1-1 will be installed---> Package orchestrator-client.x86_64 1:3.1.1-1 will be installed--> Running transaction check---> Package jq.x86_64 0:1.5-1.el7 will be installed--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64--> Running transaction check---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================================================================================================================================= Package                                            Arch                                  Version                                       Repository                                          Size=================================================================================================================================================================================================Installing: orchestrator                                       x86_64                                1:3.1.1-1                                     github_orchestrator                                9.8 M orchestrator-cli                                   x86_64                                1:3.1.1-1                                     github_orchestrator                                9.4 M orchestrator-client                                x86_64                                1:3.1.1-1                                     github_orchestrator                                 15 kInstalling for dependencies: jq                                                 x86_64                                1.5-1.el7                                     epel                                               153 k oniguruma                                          x86_64                                5.9.5-3.el7                                   epel                                               129 kTransaction Summary=================================================================================================================================================================================================Install  3 Packages (+2 Dependent packages)Total download size: 19 MInstalled size: 40 MIs this ok [y/d/N]: yDownloading packages:warning: /var/cache/yum/x86_64/7/epel/packages/jq-1.5-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY                                 ]  0.0 B/s |    0 B  --:--:-- ETA Public key for jq-1.5-1.el7.x86_64.rpm is not installed(1/5): jq-1.5-1.el7.x86_64.rpm                                                                                                                                            | 153 kB  00:00:00     (2/5): oniguruma-5.9.5-3.el7.x86_64.rpm                                                                                                                                   | 129 kB  00:00:06     orchestrator-cli-3.1.1-1.x86_6 FAILED                                             14% [==========-                                                             ]  140 B/s | 2.9 MB  34:44:11 ETA https://packagecloud.io/github/orchestrator/el/7/x86_64/orchestrator-cli-3.1.1-1.x86_64.rpm: [Errno 12] Timeout on https://d28dx6y1hfq314.cloudfront.net/1358/4059/el/7/package_files/505827.rpm?t=1565775074_357146002ed4a0c21fdc1791b6355bf780e8d974: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')Trying other mirror.(3/5): orchestrator-client-3.1.1-1.x86_64.rpm                                                                                                                             |  15 kB  00:00:01     orchestrator-3.1.1-1.x86_64.rp FAILED                                              8% [======                                                                  ] 1.0 kB/s | 1.7 MB  04:50:53 ETA https://packagecloud.io/github/orchestrator/el/7/x86_64/orchestrator-3.1.1-1.x86_64.rpm: [Errno 12] Timeout on https://d28dx6y1hfq314.cloudfront.net/1358/4059/el/7/package_files/505831.rpm?t=1565775074_9cada29cb9634db102677cf02fd559fb1611af06: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')Trying other mirror.(4/5): orchestrator-cli-3.1.1-1.x86_64.rpm                                                                                                                                | 9.4 MB  00:00:21     (5/5): orchestrator-3.1.1-1.x86_64.rpm                                                                                                                                    | 9.8 MB  00:00:25     -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Total                                                                                                                                                            174 kB/s |  19 MB  00:01:54     Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7Importing GPG key 0x352C64E5: Userid     : "Fedora EPEL (7) " Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5 Package    : epel-release-7-11.noarch (@extras) From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7Is this ok [y/N]: yRunning transaction checkRunning transaction testTransaction test succeededRunning transaction  Installing : oniguruma-5.9.5-3.el7.x86_64                                                                                                                                                  1/5   Installing : jq-1.5-1.el7.x86_64                                                                                                                                                           2/5   Installing : 1:orchestrator-client-3.1.1-1.x86_64                                                                                                                                          3/5   Installing : 1:orchestrator-3.1.1-1.x86_64                                                                                                                                                 4/5   Installing : 1:orchestrator-cli-3.1.1-1.x86_64                                                                                                                                             5/5   Verifying  : 1:orchestrator-client-3.1.1-1.x86_64                                                                                                                                          1/5   Verifying  : 1:orchestrator-3.1.1-1.x86_64                                                                                                                                                 2/5   Verifying  : oniguruma-5.9.5-3.el7.x86_64                                                                                                                                                  3/5   Verifying  : jq-1.5-1.el7.x86_64                                                                                                                                                           4/5   Verifying  : 1:orchestrator-cli-3.1.1-1.x86_64                                                                                                                                             5/5 Installed:  orchestrator.x86_64 1:3.1.1-1                               orchestrator-cli.x86_64 1:3.1.1-1                               orchestrator-client.x86_64 1:3.1.1-1                              Dependency Installed:  jq.x86_64 0:1.5-1.el7                                                                      oniguruma.x86_64 0:5.9.5-3.el7                                                                     Complete![root@mgr1 ~]# vim /etc/hosts127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4::1         localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.56.13 mgr"/etc/hosts" 8L, 232C written                                                                                                                                                  [root@mgr1 ~]# cd /etc/[root@mgr1 etc]# rzrz waiting to receive.Starting zmodem transfer.  Press Ctrl+C to cancel.Transferring orchestrator.conf.json...  100%       4 KB       4 KB/sec    00:00:01       0 Errors  [root@mgr1 etc]# cat orchestrator.conf.json {  "Debug": true,  "EnableSyslog": false,  "ListenAddress": ":3000",  "MySQLTopologyUser": "orchestrator",  "MySQLTopologyPassword": "123456",  "MySQLTopologyCredentialsConfigFile": "",  "MySQLTopologySSLPrivateKeyFile": "",  "MySQLTopologySSLCertFile": "",  "MySQLTopologySSLCAFile": "",  "MySQLTopologySSLSkipVerify": true,  "MySQLTopologyUseMutualTLS": false,  "BackendDB": "sqlite",  "SQLite3DataFile": "/usr/local/orchestrator/orchestrator.sqlite3",  "MySQLConnectTimeoutSeconds": 1,  "DefaultInstancePort": 3306,  "DiscoverByShowSlaveHosts": true,  "InstancePollSeconds": 5,  "DiscoveryIgnoreReplicaHostnameFilters": [    "a_host_i_want_to_ignore[.]example[.]com",    ".*[.]ignore_all_hosts_from_this_domain[.]example[.]com"  ],  "UnseenInstanceForgetHours": 240,  "SnapshotTopologiesIntervalHours": 0,  "InstanceBulkOperationsWaitTimeoutSeconds": 10,  "HostnameResolveMethod": "default",  "MySQLHostnameResolveMethod": "@@hostname",  "SkipBinlogServerUnresolveCheck": true,  "ExpiryHostnameResolvesMinutes": 60,  "RejectHostnameResolvePattern": "",  "ReasonableReplicationLagSeconds": 10,  "ProblemIgnoreHostnameFilters": [],  "VerifyReplicationFilters": false,  "ReasonableMaintenanceReplicationLagSeconds": 20,  "CandidateInstanceExpireMinutes": 60,  "AuditLogFile": "",  "AuditToSyslog": false,  "RemoveTextFromHostnameDisplay": ".mydomain.com:3306",  "ReadOnly": false,  "AuthenticationMethod": "",  "HTTPAuthUser": "",  "HTTPAuthPassword": "",  "AuthUserHeader": "",  "PowerAuthUsers": [    "*"  ],  "ClusterNameToAlias": {    "127.0.0.1": "test suite"  },  "SlaveLagQuery": "",  "DetectClusterAliasQuery": "SELECT SUBSTRING_INDEX(@@hostname, '.', 1)",  "DetectClusterDomainQuery": "",  "DetectInstanceAliasQuery": "",  "DetectPromotionRuleQuery": "",  "DataCenterPattern": "[.]([^.]+)[.][^.]+[.]mydomain[.]com",  "PhysicalEnvironmentPattern": "[.]([^.]+[.][^.]+)[.]mydomain[.]com",  "PromotionIgnoreHostnameFilters": [],  "DetectSemiSyncEnforcedQuery": "",  "ServeAgentsHttp": false,  "AgentsServerPort": ":3001",  "AgentsUseSSL": false,  "AgentsUseMutualTLS": false,  "AgentSSLSkipVerify": false,  "AgentSSLPrivateKeyFile": "",  "AgentSSLCertFile": "",  "AgentSSLCAFile": "",  "AgentSSLValidOUs": [],  "UseSSL": false,  "UseMutualTLS": false,  "SSLSkipVerify": false,  "SSLPrivateKeyFile": "",  "SSLCertFile": "",  "SSLCAFile": "",  "SSLValidOUs": [],  "URLPrefix": "",  "StatusEndpoint": "/api/status",  "StatusSimpleHealth": true,  "StatusOUVerify": false,  "AgentPollMinutes": 60,  "UnseenAgentForgetHours": 6,  "StaleSeedFailMinutes": 60,  "SeedAcceptableBytesDiff": 8192,  "PseudoGTIDPattern": "",  "PseudoGTIDPatternIsFixedSubstring": false,  "PseudoGTIDMonotonicHint": "asc:",  "DetectPseudoGTIDQuery": "",  "BinlogEventsChunkSize": 10000,  "SkipBinlogEventsContaining": [],  "ReduceReplicationAnalysisCount": true,  "FailureDetectionPeriodBlockMinutes": 1,  "RecoveryPeriodBlockSeconds": 0,  "RecoveryIgnoreHostnameFilters": [],  "RecoverMasterClusterFilters": [    "*"  ],  "RecoverIntermediateMasterClusterFilters": [    "*"  ],  "OnFailureDetectionProcesses": [    "echo 'Detected {failureType} on {failureCluster}. Affected replicas: {countSlaves}' >> /tmp/recovery.log"  ],  "PreFailoverProcesses": [    "echo 'Will recover from {failureType} on {failureCluster}' >> /tmp/recovery.log"  ],  "PostFailoverProcesses": [        "echo '(for all types) Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log","/usr/local/bin/orch_hook.sh {failureType} {failureClusterAlias} {failedHost} {successorHost} >> /tmp/orch.log"  ],  "PostUnsuccessfulFailoverProcesses": [],  "PostMasterFailoverProcesses": [    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log"  ],  "PostIntermediateMasterFailoverProcesses": [    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log"  ],  "CoMasterRecoveryMustPromoteOtherCoMaster": true,  "DetachLostSlavesAfterMasterFailover": true,  "ApplyMySQLPromotionAfterMasterFailover": true,  "PreventCrossDataCenterMasterFailover": false,  "PreventCro***egionMasterFailover": false,  "MasterFailoverDetachSlaveMasterHost": false,  "MasterFailoverLostInstancesDowntimeMinutes": 0,  "PostponeSlaveRecoveryOnLagMinutes": 0,  "OSCIgnoreHostnameFilters": [],  "GraphiteAddr": "",  "GraphitePath": "",  "GraphiteConvertHostnameDotsToUnderscores": true}[root@mgr1 etc]# /etc/init.d/elasticsearch  netconsole     network        proxysql       [root@mgr1 etc]# netstat -nltpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    tcp        0      0 0.0.0.0:6032            0.0.0.0:*               LISTEN      3326/proxysql       tcp        0      0 0.0.0.0:6033            0.0.0.0:*               LISTEN      3326/proxysql       tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3065/sshd           tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3295/master         tcp6       0      0 :::22                   :::*                    LISTEN      3065/sshd           tcp6       0      0 ::1:25                  :::*                    LISTEN      3295/master         [root@mgr1 etc]# cd /usr/local/orchestrator/[root@mgr1 orchestrator]# lltotal 19436-rwxr-xr-x. 1 root root 19884352 Aug  4 14:47 orchestrator-rw-r--r--. 1 root root     5465 Aug  4 14:45 orchestrator-sample.conf.json-rw-r--r--. 1 root root     4668 Aug  4 14:45 orchestrator-sample-sqlite.conf.jsondrwxr-xr-x. 7 root root       82 Aug 14 17:28 resources[root@mgr1 orchestrator]# /usr/local/orchestrator/orchestrator  http2019-08-14 17:33:12 DEBUG Connected to orchestrator backend: sqlite on /usr/local/orchestrator/orchestrator.sqlite32019-08-14 17:33:12 DEBUG Initializing orchestrator2019-08-14 17:33:12 DEBUG Migrating database schema2019-08-14 17:33:12 DEBUG Migrated database schema to version [3.1.1]2019-08-14 17:33:12 INFO Connecting to backend :3306: maxConnections: 128, maxIdleConns: 322019-08-14 17:33:12 INFO Starting Discovery2019-08-14 17:33:12 INFO Registering endpoints2019-08-14 17:33:12 INFO continuous discovery: setting up2019-08-14 17:33:12 INFO continuous discovery: starting2019-08-14 17:33:12 INFO Starting HTTP listener on :30002019-08-14 17:33:12 DEBUG Queue.startMonitoring(DEFAULT)2019-08-14 17:33:13 INFO Not elected as active node; active node: ; polling2019-08-14 17:33:14 INFO Not elected as active node; active node: ; polling2019-08-14 17:33:15 INFO Not elected as active node; active node: ; polling2019-08-14 17:33:16 INFO Not elected as active node; active node: ; polling2019-08-14 17:33:17 INFO Not elected as active node; active node: ; polling2019-08-14 17:33:19 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:20 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:21 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:22 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:23 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:24 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:25 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery2019-08-14 17:33:26 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery[martini] Started GET /web/clusters for 192.168.56.1:54290[martini] Completed 200 OK in 8.284668ms[martini] Started GET /js/jquery.min.js for 192.168.56.1:54290[martini] [Static] Serving /js/jquery.min.js[martini] Started GET /js/common.js for 192.168.56.1:54296[martini] [Static] Serving /js/common.js[martini] Completed 200 OK in 2.074743ms[martini] Started GET /js/jquery.cookie-1.4.1.min.js for 192.168.56.1:54291[martini] [Static] Serving /js/jquery.cookie-1.4.1.min.js[martini] Completed 200 OK in 2.944014ms[martini] Completed 200 OK in 31.260301ms[martini] Started GET /js/corex.js for 192.168.56.1:54293[martini] [Static] Serving /js/corex.js[martini] Started GET /js/corex-jquery.js for 192.168.56.1:54294[martini] [Static] Serving /js/corex-jquery.js[martini] Completed 200 OK in 9.197442ms[martini] Started GET /js/md5.js for 192.168.56.1:54295[martini] [Static] Serving /js/md5.js[martini] Completed 200 OK in 2.743661ms[martini] Started GET /css/orchestrator.css for 192.168.56.1:54291[martini] [Static] Serving /css/orchestrator.css[martini] Completed 200 OK in 6.951469ms[martini] Started GET /bootstrap/css/bootstrap.min.css for 192.168.56.1:54296[martini] [Static] Serving /bootstrap/css/bootstrap.min.css[martini] Completed 200 OK in 64.795256ms[martini] Started GET /js/orchestrator.js for 192.168.56.1:54294[martini] [Static] Serving /js/orchestrator.js[martini] Completed 200 OK in 6.589851ms[martini] Started GET /js/cluster-analysis-shared.js for 192.168.56.1:54291[martini] [Static] Serving /js/cluster-analysis-shared.js[martini] Completed 200 OK in 3.819321ms[martini] Started GET /js/custom.js for 192.168.56.1:54295[martini] [Static] Serving /js/custom.js[martini] Completed 200 OK in 10.738787ms[martini] Started GET /css/custom.css for 192.168.56.1:54290[martini] [Static] Serving /css/custom.css[martini] Completed 200 OK in 5.566896ms[martini] Completed 200 OK in 64.566793ms[martini] Started GET /images/ajax-loader.gif for 192.168.56.1:54293[martini] [Static] Serving /images/ajax-loader.gif[martini] Completed 200 OK in 2.594018ms[martini] Started GET /js/clusters.js for 192.168.56.1:54291[martini] [Static] Serving /js/clusters.js[martini] Completed 200 OK in 2.649993ms[martini] Started GET /js/instance-problems.js for 192.168.56.1:54290[martini] [Static] Serving /js/instance-problems.js[martini] Started GET /js/bootbox.min.js for 192.168.56.1:54294[martini] [Static] Serving /js/bootbox.min.js[martini] Completed 200 OK in 2.448253ms[martini] Started GET /bootstrap/js/bootstrap.min.js for 192.168.56.1:54295[martini] [Static] Serving /bootstrap/js/bootstrap.min.js[martini] Completed 200 OK in 4.693863ms[martini] Completed 200 OK in 15.828717ms[martini] Started GET /bootstrap/fonts/glyphicons-halflings-regular.woff for 192.168.56.1:54290[martini] [Static] Serving /bootstrap/fonts/glyphicons-halflings-regular.woff[martini] Completed 200 OK in 5.705257ms[martini] Started GET /api/clusters-info for 192.168.56.1:54290[martini] Completed 200 OK in 2.441157ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54295[martini] Completed 200 OK in 3.624178ms[martini] Started GET /api/clusters-info for 192.168.56.1:54290[martini] Completed 200 OK in 6.241382ms[martini] Started GET /api/problems for 192.168.56.1:54294[martini] Started GET /images/orchestrator-logo-32.png for 192.168.56.1:54291[martini] [Static] Serving /images/orchestrator-logo-32.png[martini] Completed 200 OK in 1.467163ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54290[martini] Completed 200 OK in 37.670099ms[martini] Started GET /assets/ico/favicon.ico for 192.168.56.1:54291[martini] Completed 404 Not Found in 1.050475ms[martini] Started GET /api/maintenance for 192.168.56.1:54294[martini] Completed 200 OK in 3.329422ms[martini] Completed 200 OK in 57.505224ms[martini] Started GET /api/problems for 192.168.56.1:54290[martini] Completed 200 OK in 7.896307ms[martini] Started GET /web/discover for 192.168.56.1:54290[martini] Completed 200 OK in 5.045515ms[martini] Started GET /js/discover.js for 192.168.56.1:54290[martini] [Static] Serving /js/discover.js[martini] Completed 200 OK in 929.59µs[martini] Started GET /api/clusters-info for 192.168.56.1:54290[martini] Completed 200 OK in 6.435473ms[martini] Started GET /api/problems for 192.168.56.1:54291[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54294[martini] Completed 200 OK in 2.087242ms[martini] Completed 200 OK in 25.108654ms[martini] Started GET /api/maintenance for 192.168.56.1:54291[martini] Completed 200 OK in 3.5533ms[martini] Started GET /api/discover/es2/3306 for 192.168.56.1:542912019-08-14 17:33:46 DEBUG Hostname unresolved yet: es22019-08-14 17:33:46 DEBUG Cache hostname resolve es2 as es22019-08-14 17:33:46 DEBUG Hostname unresolved yet: es32019-08-14 17:33:46 DEBUG Cache hostname resolve es3 as es32019-08-14 17:33:46 DEBUG Hostname unresolved yet: es32019-08-14 17:33:46 DEBUG Cache hostname resolve es3 as es3[martini] Completed 200 OK in 57.266738ms[martini] Started GET /web/clusters/ for 192.168.56.1:54291[martini] Completed 200 OK in 4.124483ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 2.95734ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54294[martini] Completed 200 OK in 4.641513ms[martini] Started GET /api/problems for 192.168.56.1:54290[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 2.410554ms[martini] Completed 200 OK in 25.635704ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54291[martini] Started GET /api/maintenance for 192.168.56.1:54290[martini] Completed 200 OK in 1.96617ms[martini] Completed 200 OK in 54.309241ms[martini] Started GET /api/problems for 192.168.56.1:54291[martini] Completed 200 OK in 10.705548ms[martini] Started GET /web/clusters/ for 192.168.56.1:54291[martini] Completed 200 OK in 3.347282ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 2.215592ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54290[martini] Completed 200 OK in 6.173855ms[martini] Started GET /api/problems for 192.168.56.1:54294[martini] Completed 200 OK in 11.36747ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 3.588512ms[martini] Started GET /api/maintenance for 192.168.56.1:54294[martini] Completed 200 OK in 2.02093ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54291[martini] Completed 200 OK in 56.420034ms[martini] Started GET /api/problems for 192.168.56.1:54291[martini] Completed 200 OK in 10.632568ms2019-08-14 17:33:52 DEBUG Hostname unresolved yet: es12019-08-14 17:33:52 DEBUG Cache hostname resolve es1 as es1[martini] Started GET /web/clusters/ for 192.168.56.1:54291[martini] Completed 200 OK in 5.428885ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 2.176698ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54294[martini] Completed 200 OK in 1.426532ms[martini] Started GET /api/problems for 192.168.56.1:54290[martini] Completed 200 OK in 9.368997ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 4.887766ms[martini] Started GET /api/maintenance for 192.168.56.1:54290[martini] Completed 200 OK in 6.794784ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54291[martini] Completed 200 OK in 48.708861ms[martini] Started GET /api/problems for 192.168.56.1:54291[martini] Completed 200 OK in 7.038165ms[martini] Started GET /web/clusters/ for 192.168.56.1:54291[martini] Completed 200 OK in 4.246108ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 3.952856ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54290[martini] Completed 200 OK in 6.460721ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 4.537462ms[martini] Started GET /api/problems for 192.168.56.1:54294[martini] Completed 200 OK in 8.731843ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54291[martini] Started GET /api/maintenance for 192.168.56.1:54294[martini] Completed 200 OK in 1.999778ms[martini] Completed 200 OK in 86.678785ms[martini] Started GET /api/problems for 192.168.56.1:54291[martini] Completed 200 OK in 7.292489ms[martini] Started GET /web/clusters/ for 192.168.56.1:54291[martini] Completed 200 OK in 4.109349ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 3.927826ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 5.524413ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54294[martini] Completed 200 OK in 2.888945ms[martini] Started GET /api/problems for 192.168.56.1:54290[martini] Started GET /api/replication-analysis for 192.168.56.1:54291[martini] Completed 200 OK in 34.22444ms[martini] Completed 200 OK in 37.299671ms[martini] Started GET /api/maintenance for 192.168.56.1:54290[martini] Completed 200 OK in 3.180638ms[martini] Started GET /api/problems for 192.168.56.1:54291[martini] Completed 200 OK in 10.985714ms[martini] Started GET /web/clusters/ for 192.168.56.1:54291[martini] Completed 200 OK in 3.097257ms[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 2.137603ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54290[martini] Completed 200 OK in 3.914453ms[martini] Started GET /api/problems for 192.168.56.1:54294[martini] Started GET /api/clusters-info for 192.168.56.1:54291[martini] Completed 200 OK in 1.990911ms[martini] Completed 200 OK in 24.679852ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54294[martini] Started GET /api/maintenance for 192.168.56.1:54291[martini] Completed 200 OK in 1.934805ms[martini] Completed 200 OK in 45.120922ms[martini] Started GET /api/problems for 192.168.56.1:54294[martini] Completed 200 OK in 14.302921ms2019-08-14 17:34:12 INFO auditType:inject-unseen-masters instance::0 cluster: message:Operations: 02019-08-14 17:34:12 INFO auditType:forget-unseen-differently-resolved instance::0 cluster: message:Forgotten instances: 02019-08-14 17:34:12 INFO auditType:review-unseen-instances instance::0 cluster: message:Operations: 02019-08-14 17:34:12 DEBUG kv.SubmitMastersToKvStores, clusterName: , force: false: numPairs: 52019-08-14 17:34:12 INFO auditType:forget-unseen instance::0 cluster: message:Forgotten instances: 02019-08-14 17:34:12 INFO auditType:resolve-unknown-masters instance::0 cluster: message:Num resolved hostnames: 02019-08-14 17:34:12 DEBUG kv.SubmitMastersToKvStores: submitKvPairs: 5[martini] Started GET /web/clusters/ for 192.168.56.1:54294[martini] Completed 200 OK in 10.713799ms[martini] Started GET /api/clusters-info for 192.168.56.1:54294[martini] Completed 200 OK in 4.009104ms[martini] Started GET /api/check-global-recoveries for 192.168.56.1:54291[martini] Completed 200 OK in 5.498638ms[martini] Started GET /api/problems for 192.168.56.1:54290[martini] Started GET /api/clusters-info for 192.168.56.1:54294[martini] Completed 200 OK in 13.613024ms[martini] Completed 200 OK in 3.732243ms[martini] Started GET /api/maintenance for 192.168.56.1:54294[martini] Completed 200 OK in 1.999938ms[martini] Started GET /api/replication-analysis for 192.168.56.1:54290[martini] Completed 200 OK in 47.912159ms[martini] Started GET /api/problems for 192.168.56.1:54290[martini] Completed 200 OK in 18.358529ms2019-08-14 17:35:12 INFO auditType:review-unseen-instances instance::0 cluster: message:Operations: 02019-08-14 17:35:12 INFO auditType:inject-unseen-masters instance::0 cluster: message:Operations: 02019-08-14 17:35:12 INFO auditType:forget-unseen instance::0 cluster: message:Forgotten instances: 02019-08-14 17:35:12 INFO auditType:forget-unseen-differently-resolved instance::0 cluster: message:Forgotten instances: 02019-08-14 17:35:12 INFO auditType:resolve-unknown-masters instance::0 cluster: message:Num resolved hostnames: 02019-08-14 17:35:12 DEBUG kv.SubmitMastersToKvStores, clusterName: , force: false: numPairs: 52019-08-14 17:35:12 DEBUG kv.SubmitMastersToKvStores: submitKvPairs: 0^C