怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误
这篇文章主要介绍"怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误",在日常操作中,相信很多人在怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误问题上存在疑惑,小编查阅了各式资料,整理出简单好用的操作方法,希望对大家解答"怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误"的疑惑有所帮助!接下来,请跟着小编一起来学习吧!
一、环境说明:
操作系统:RHEL7.4
数据库版本:12.2.0.1
已打补丁:201910月的季度补丁
由于保密缘故,相关命名与IP已做变更
二、问题描述
在对GI打季度补丁时,节点1运行opatchauto apply后,CRS正常关闭,但是到自动拉起CRS时遇到ORA-12547: TNS:lost contact报错了,导致打补丁失败。
详细信息如下:
CRS-4123: Starting Oracle High Availability Services-managed resourcesCRS-2672: Attempting to start 'ora.mdnsd' on 'db1'CRS-2672: Attempting to start 'ora.evmd' on 'db1'CRS-2676: Start of 'ora.mdnsd' on 'db1' succeededCRS-2676: Start of 'ora.evmd' on 'db1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'db1'CRS-2676: Start of 'ora.gpnpd' on 'db1' succeededCRS-2672: Attempting to start 'ora.gipcd' on 'db1'CRS-2676: Start of 'ora.gipcd' on 'db1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'db1'CRS-2676: Start of 'ora.cssdmonitor' on 'db1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'db1'CRS-2672: Attempting to start 'ora.diskmon' on 'db1'CRS-2676: Start of 'ora.diskmon' on 'db1' succeededCRS-2676: Start of 'ora.cssd' on 'db1' succeededCRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'db1'CRS-2672: Attempting to start 'ora.ctssd' on 'db1'CRS-2676: Start of 'ora.ctssd' on 'db1' succeededCRS-2676: Start of 'ora.cluster_interconnect.haip' on 'db1' succeededCRS-2672: Attempting to start 'ora.asm' on 'db1'CRS-2676: Start of 'ora.asm' on 'db1' succeededCRS-2672: Attempting to start 'ora.storage' on 'db1'ORA-12547: TNS:lost contactORA-12547: TNS:lost contactORA-15077: could not locate ASM instance serving a required diskgroupCRS-2883: Resource 'ora.storage' failed during Clusterware stack start.CRS-4406: Oracle High Availability Services synchronous start failed.CRS-4000: Command Start failed, or completed with errors.2020/06/10 16:46:12 CLSRSC-117: Failed to start Oracle Clusterware stack After fixing the cause of failure Run opatchauto resume]OPATCHAUTO-68061: The orchestration engine failed.OPATCHAUTO-68061: The orchestration engine failed with return code 1OPATCHAUTO-68061: Check the log for more details.OPatchAuto failed.OPatchauto session completed at Wed Jun 10 16:46:13 2020Time taken to complete the session 17 minutes, 25 seconds opatchauto failed with error code 42
调用crsctl check crs查看集群服务,发现Cluster Ready Services、Event Manager异常。查看/u01/app/grid/diag/crs/db1/crs/trace/alert.log日志,发现如下信息:
2020-06-10 16:36:08.150 [ORAROOTAGENT(679524)]CRS-5019: All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc".2020-06-10 16:45:58.112 [ORAROOTAGENT(679524)]CRS-5818: Aborted command 'start' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:5:3} in /u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc.2020-06-10 16:46:00.427 [ORAROOTAGENT(679524)]CRS-5017: The resource action "ora.storage start" encountered the following error: 2020-06-10 16:46:00.427+Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc".2020-06-10 16:46:00.429 [OHASD(679429)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.storage'. Details at (:CRSPE00221:) {0:5:3} in /u01/app/grid/diag/crs/db1/crs/trace/ohasd.trc.2020-06-10 16:47:12.689 [OSYSMOND(684959)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 684959
/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc日志如下:
2020-06-09 16:34:08.611 :CLSDYNAM:3440035584: [ora.storage]{0:1:251} [check] StorageAgent::parsekgforetcodes retcode = 0, kgfoCheckMount(OCR), flag 42020-06-09 16:34:08.611 :CLSDYNAM:3440035584: [ora.storage]{0:1:251} [check] StorageAgent::check kgfo returncode 02020-06-09 16:34:11.243 :CLSDYNAM:3429529344: [ ora.crf]{0:5:3} [check] Check return = 0, state detail = NULL2020-06-09 16:34:15.926 :CLSDYNAM:2525263616: [ora.ctssd]{0:5:3} [check] translateReturnCodes, return = 0, state detail = OBSERVERCheckcb data [0x7f2369c293d0]: mode[0xee] offset[0 ms].2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Arg Value = check2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.2.0/grid2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.2.0/grid2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Utils::execCmd 1 USR_ORA_ENV: oracleHome:/u01/app/12.2.0/grid CrsHome:/u01/app/12.2.0/grid2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Adding Environment Variables ORA_DAEMON_LOGGING_LEVELS=GIPCLIB=2,GIPC=2,GIPCGEN=2,GIPCTRAC=2,GIPCWAIT=2,GIPCXCPT=2,GIPCOSD=2,GIPCBASE=2,GIPCCLSA=2,GIPCCLSC=2,GIPCEXMP=2,GIPCGMOD=2,GIPCHEAD=2,GIPCMUX=2,GIPCNET=2,GIPCNULL=2,GIPCPKT=2,GIPCSMEM=2,GIPCHAUP=2,GIPCHALO=2,GIPCHTHR=2,GIPCHGEN=2,GIPCHLCK=2,GIPCHDEM=2,GIPCHWRK=2,GIPCTLS=2,GIPCHGNS=22020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Adding Environment Variables ORA_DAEMON_TRACE_FILE_OPTIONS=filesize=26214400,numsegments=102020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Adding Environment Variables _ORA_AGENT_ACTION=TRUE2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Adding Environment Variables __IS_HASD_AGENT=2020-06-09 16:34:19.107 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Utils:execCmd action = 3 flags = 6 ohome = (null) cmdname = acfsload. 2020-06-09 16:34:19.108 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.2.0/grid2020-06-09 16:34:19.108 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.2.0/grid2020-06-09 16:34:19.219 :CLSDYNAM:3425326848: [ora.drivers.acfs]{0:0:2} [check] execCmd ret = 02020-06-09 16:34:20.911 :CLSDYNAM:3427428096: [ora.driver.afd]{0:0:2} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = afddriverstate. 2020-06-09 16:34:20.911 :CLSDYNAM:3427428096: [ora.driver.afd]{0:0:2} [check] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.2.0/grid2020-06-09 16:34:20.911 :CLSDYNAM:3427428096: [ora.driver.afd]{0:0:2} [check] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.2.0/grid2020-06-09 16:34:21.121 :CLSDYNAM:3427428096: [ora.driver.afd]{0:0:2} [check] execCmd ret = 02020-06-09 16:34:22.763 : USRTHRD:2548033280: {0:5:3} Check: 0-12020-06-09 16:34:26.337 : AGFW:3435833088: {0:0:2} Agent received the message: AGENT_HB[Engine] ID 12293:20865052020-06-09 16:34:32.281 : CLSDMC:3425326848: command 0 failed with status 168432652020-06-09 16:34:32.282 :CLSDYNAM:3425326848: [ora.crsd]{0:1:52} [check] DaemonAgent::check returned 02020-06-09 16:34:32.282 :CLSDYNAM:3425326848: [ora.crsd]{0:1:52} [check] DaemonAgent::check checkErrorCode=16843265, pestate=512,perole=65536, pemode=12020-06-09 16:34:38.611 :CLSDYNAM:3427428096: [ora.storage]{0:1:251} [check] StorageAgent::check NODEROLE_HUB getOCRdetails2020-06-09 16:34:38.686 :CLSDYNAM:3427428096: [ora.storage]{0:1:251} [check] StorageAgent::parsekgforetcodes retcode = 0, kgfoCheckMount(OCR), flag 22020-06-09 16:34:38.686 :CLSDYNAM:3427428096: [ora.storage]{0:1:251} [check] StorageAgent::check kgfo returncode 02020-06-09 16:34:41.239 :CLSDYNAM:3444238080: [ ora.crf]{0:5:3} [check] Check return = 0, state detail = NULL2020-06-09 16:34:45.921 :CLSDYNAM:3444238080: [ora.ctssd]{0:5:3} [check] translateReturnCodes, return = 0, state detail = OBSERVERCheckcb data [0x7f23baa99360]: mode[0xee] offset[0 ms].2020-06-09 16:34:52.763 : USRTHRD:2548033280: {0:5:3} Check: 0-12020-06-09 16:34:56.341 : AGFW:3435833088: {0:0:2} Agent received the message: AGENT_HB[Engine] ID 12293:20865222020-06-09 16:35:02.285 : CLSDMC:2525263616: command 0 failed with status 168432652020-06-09 16:35:02.285 :CLSDYNAM:2525263616: [ora.crsd]{0:1:52} [check] DaemonAgent::check returned 0
三、故障分析
从告警日志中可知,CRS进程未能识别到OCR磁盘有挂载导致CRS服务启动失败。
尝试手工启动ASM实例,CRS启动成功:
sqlplus / as sysasmstartup
关闭CRS后再调用crsctl start crs,ASM实例一样未能自动拉起,只能通过startup的方式。
查看/u01/app/grid/diag/crs/db1/crs/trace/crsd.trc日志,发现如下信息:
2020-06-10 16:48:24.867*:kgfn.c@1370: kgfnFindLocalNode: found no members2020-06-10 16:48:24.867 : OCRRAW:633798784: kgfnFindLocalNode: not ok2020-06-10 16:48:24.867*:kgfn.c@1422: kgfnFindLocalNode: not ok2020-06-10 16:48:24.867 : OCRRAW:633798784: kgfnTgtInit: local node not found, free kgfnpds2020-06-10 16:48:24.867*:kgfn.c@2208: kgfnTgtInit: not found2020-06-10 16:48:24.867 : OCRRAW:633798784: kgfnGetBeqData failed init target; inst=(null) flags=0x60002020-06-10 16:48:24.867*:kgfn.c@5791: kgfnGetBeqData: kgfnTgtInit failed, inst=NULL flags=0x60002020-06-10 16:48:24.867 : CLSNS:633798784: clsns_SetTraceLevel:trace level set to 1.2020-06-10 16:48:24.900 : OCRRAW:633798784: 8154 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS2020-06-10 16:48:24.904 : OCRRAW:633798784: 8154 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS2020-06-10 16:48:24.940 : OCRRAW:633798784: 7872 Error 4 opening dom root in 0x5fb60d02020-06-10 16:48:24.958 : OCRRAW:633798784: kgfnConnect2: kgfnGetBeqData failed2020-06-10 16:48:24.958*:kgfn.c@5012: kgfnConnect2: kgfnGetBeqData failed2020-06-10 16:48:24.992 : OCRRAW:633798784: kgfnConnect2Int: cstr=(DESCRIPTION=(TCP_USER_TIMEOUT=1)(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.43)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))2020-06-10 16:48:24.992*:kgfn.c@6788: kgfnConnect2Int: cstr=(DESCRIPTION=(TCP_USER_TIMEOUT=1)(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.43)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))2020-06-10 16:48:24.993 : OCRRAW:633798784: kgfnConnect2Int: ServerAttach2020-06-10 16:48:24.993*:kgfn.c@6811: kgfnConnect2Int: OCIServerAttach failedFailed to connect to ASM 1 0 0 (nil) 02020-06-10 16:48:25.994 : OCRRAW:633798784: kgfnRecordErr 12547 OCI error:ORA-12547: TNS:lost contact2020-06-10 16:48:25.994*:kgfn.c@1740: kgfnRecordErrPriv: 12547 error=ORA-12547: TNS:lost contact2020-06-10 16:48:25.995 : OCRRAW:633798784: kgfnConnect2: failed to connect2020-06-10 16:48:25.995*:kgfn.c@5333: kgfnConnect2: failed to connect2020-06-10 16:48:25.995 : OCRRAW:633798784: kgfnConnect2Retry: failed to connect connect after 2 attempts, 331s elapsed2020-06-10 16:48:25.995 : OCRRAW:633798784: kgfo_kge2slos error stack at kgfoAl06: ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact的报错出现在该日志文件中,而该报错是要访问私网IP10.10.10.43 1526端口失败所致。
四、问题原因
1526端口是运行在ASMNET1LSNR_ASM监听上的。在12cRAC中,从FLEXASM开始,crsd.bin将使用ASMNET1LSNR_ASM监听进行远程连接。即会连接10.10.10.43的1526端口。
那么为何连接节点2的1526端口会失败呢。
查看节点2的sqlnet.ora配置,发现配置了白名单,但是没有增加节点1的私网IP。
五、解决办法
1) Modify the file sqlnet.oravi $GRID_HOME/network/admin/sqlnet.ora2) Add the IP which are used for Private interconnect.e.g.TCP.VALIDNODE_CHECKING = YESTCP.INVITED_NODES=(node1.localhost, node2.localhost,node1-priv.localhost,node2-priv.localhost, node1-vip.localhost, node2-vip.localhost, application server VIPS)3) Restart the CRS in issue node.
到此,关于"怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误"的学习就结束了,希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习,快去试试吧!若想继续学习更多相关知识,请继续关注网站,小编会继续努力为大家带来更多实用的文章!