通过案例学Oracle之--一次AIX rac误操作引起的“血案”

通过案例学Oracle之--一次AIX rac误操作引起的"血案"

系统环境：

操作系统： AIX 5300-09

集群软件： CRS 10.2.0.1

数据库： Oracle 10.2.0.1

本案例是用于基于VG Concurrent 的共享存储，通过HACMP 实现卷组的并发

案例分析：

一、错误现象：

1、Oracle 用户无法访问设备文件

2、CRS server启动失败

[oracle@aix211 ~]$ls -l /dev

/dev/__vg10: No permission/dev/audit: No permission/dev/cd0: No permission/dev/clone: No permission/dev/console: No permission/dev/error: No permission

查看设备文件属性，发现被改为oracle：dba

[oracle@aix211 ~]$ls -ld /dev

drw-rw---- 6 oracle dba 3584 Sep 16 11:38 /dev

重新更改设备文件属性

[root@aix211 /]#chown root.system /dev[root@aix211 /]#ls -ld /devdrw-rw----    6 root     system         3584 Sep 16 11:38 /dev[root@aix211 /]#chmod 775 /dev

Oracle用户可以正常访问设备文件

[root@aix211 /]#su - oracle[oracle@aix211 ~]$ls -l /devtotal 24crw-rw----    1 root     system       10,  0 Aug 29 2013  IPL_rootvgsrwxrwxrwx    1 root     system            0 Sep 16 10:22 SRCbrw-rw----    1 oracle   dba          88,  9 Sep 11 12:15 control1_1brw-rw----    1 oracle   dba          88, 10 Sep 11 12:15 control2_2brw-rw----    1 oracle   dba          88, 11 Sep 11 12:16 control3_3crw-rw----    1 root     system       88,  0 Sep 11 12:08 datavg

但是CRS server仍然不能正常启动！

二、重新配置CRS:

1、清理ocr和vote disk磁盘信息（两个节点）

[root@aix211 /]#dd if=/dev/zero of=/dev/rrac_ocr bs=8192 count=25602560+0 records in2560+0 records out[root@aix211 /]#dd if=/dev/zero of=/dev/rrac_vote bs=8192 count=25602560+0 records in2560+0 records out[root@aix211 /]#ls -l /dev|grep ocrbrw-rw----    1 oracle   dba          88,  1 Sep 11 12:15 rac_ocrcrw-r-----    1 root     oinstall     88,  1 Sep 16 11:05 rrac_ocr[root@aix211 /]#chown oracle:dba /dev/rrac_ocr

2、重新运行root.sh脚本，配置CRS（两个节点）

node1：

[root@aix211 install]#./rootdelete.sh

Shutting down Oracle Cluster Ready Services (CRS):Sep 16 11:48:57.011 | ERR | failed to connect to daemon, errno(2)Stopping resources.Error while stopping resources. Possible cause: CRSD is down.Stopping CSSD.Unable to communicate with the CSS daemon.Shutdown has begun. The daemons should exit soon.Checking to see if Oracle CRS stack is down...Oracle CRS stack is not running.Oracle CRS stack is down now.Removing script for Oracle Cluster Ready servicesUpdating ocr file for downgradeCleaning up SCR settings in '/etc/oracle/scls_scr'

[root@aix211 install]#/u01/crs_1/root.sh

WARNING: directory '/u01' is not owned by rootChecking to see if Oracle CRS stack is already configuredChecking to see if any 9i GSD is upSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successfullyWARNING: directory '/u01' is not owned by rootclscfg: EXISTING configuration version 3 detected.clscfg: version 3 is 10G Release 2.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node :   node 1: aix211 aix211-priv aix211node 2: aix212 aix212-priv aix212clscfg: Arguments check out successfully.NO KEYS WERE WRITTEN. Supply -force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.CSS is active on these nodes.        aix211CSS is inactive on these nodes.        aix212Local node checking complete.Run root.sh on remaining nodes to start CRS daemons.

node2：

[root@aix212 install]#./rootdelete.sh

Shutting down Oracle Cluster Ready Services (CRS):Sep 16 11:48:57.011 | ERR | failed to connect to daemon, errno(2)Stopping resources.Error while stopping resources. Possible cause: CRSD is down.Stopping CSSD.Unable to communicate with the CSS daemon.Shutdown has begun. The daemons should exit soon.Checking to see if Oracle CRS stack is down...Oracle CRS stack is not running.Oracle CRS stack is down now.Removing script for Oracle Cluster Ready servicesUpdating ocr file for downgradeCleaning up SCR settings in '/etc/oracle/scls_scr'

[root@aix212@ /]#/u01/crs_1/root.sh

WARNING: directory '/u01' is not owned by rootChecking to see if Oracle CRS stack is already configuredSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successfullyWARNING: directory '/u01' is not owned by rootclscfg: EXISTING configuration version 3 detected.clscfg: version 3 is 10G Release 2.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node :   node 1: aix211 aix211-priv aix211node 2: aix212 aix212-priv aix212clscfg: Arguments check out successfully.NO KEYS WERE WRITTEN. Supply -force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.CSS is active on these nodes.        aix211        aix212CSS is active on all nodes.Waiting for the Oracle CRSD and EVMD to startOracle CRS stack installed and running under init(1M)Running vipca(silent) for configuring nodeappsThe given interface(s), "en0" is not public. Public interfaces should be used to configure virtual IPs.

在node2上运行vipca，配置vip

@至此，CRS重新配置成功！

[root@aix212@ /]#crsctl check crsCSS appears healthyCRS appears healthyEVM appears healthy[root@aix212@ /]#crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212

三、重新注册Listener和Database

1、注册listener

通过netca工具，重新reconfigure就可以完成listener的注册！

[root@aix212@ /]#crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora....11.lsnr application    ONLINE    ONLINE    aix211      ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora....12.lsnr application    ONLINE    ONLINE    aix212      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212

2、注册Database和Instance

注册Database：

[root@aix212@ /]#srvctl add database -h

Usage: srvctl add database -d  -o  [-m ] [-p ] [-A /netmask] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [-s ] [-n ] [-y {AUTOMATIC | MANUAL}]    -d            Unique name for the database    -o     ORACLE_HOME for cluster database    -m          Domain for cluster database    -p          Server parameter file for cluster database    -A        Database cluster alias    -n         Database name (DB_NAME), if different from the unique name given by the -d option    -r            Role of the database (primary, physical_standby, logical_standby)    -s   Startup options for the database    -y        Management policy for the database (automatic, manual)    -h                  Print usage

[root@aix212@ /]#su - oracle

[oracle@aix212@ ~]$srvctl add database -d prod -o $ORACLE_HOME

[oracle@aix212@ ~]$crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora....11.lsnr application    ONLINE    ONLINE    aix211      ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora....12.lsnr application    ONLINE    ONLINE    aix212      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212      ora.prod.db    application    OFFLINE   OFFLINE

注册Instance：

[oracle@aix212@ ~]$srvctl add instance -hUsage: srvctl add instance -d  -i  -n     -d            Unique name for the database    -i            Instance name    -n            Node name    -h                  Print usage[oracle@aix212@ ~]$srvctl add instance -d prod -i prod1 -n aix211[oracle@aix212@ ~]$srvctl add instance -d prod -i prod2 -n aix212[oracle@aix212@ ~]$crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora....11.lsnr application    ONLINE    ONLINE    aix211      ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora....12.lsnr application    ONLINE    ONLINE    aix212      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212      ora.prod.db    application    OFFLINE   OFFLINE               ora....d1.inst application    OFFLINE   OFFLINE               ora....d2.inst application    OFFLINE   OFFLINE

通过crs工具启动Database：

[oracle@aix212@ ~]$srvctl start database -d prod

PRKP-1001 : Error starting instance prod1 on node aix211

CRS-0184: Cannot communicate with the CRS daemon.

PRKP-1001 : Error starting instance prod2 on node aix212

CRS-0184: Cannot communicate with the CRS daemon.

启动Instance失败，通过sqlplus手工启动：

[oracle@aix212@ ~]$sqlplus '/as sysdba'SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 16 12:08:10 2014Copyright (c) 1982, 2005, Oracle.  All rights reserved.Connected to an idle instance.SQL> startupORACLE instance started.Total System Global Area 1258291200 bytesFixed Size                  2020552 bytesVariable Size             352324408 bytesDatabase Buffers          889192448 bytesRedo Buffers               14753792 bytesDatabase mounted.Database opened.

[oracle@aix211 aix211]$sqlplus '/as sysdba'SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 16 12:09:37 2014Copyright (c) 1982, 2005, Oracle.  All rights reserved.Connected to an idle instance.SQL> startupORACLE instance started.Total System Global Area 1258291200 bytesFixed Size                  2020552 bytesVariable Size             335547192 bytesDatabase Buffers          905969664 bytesRedo Buffers               14753792 bytesDatabase mounted.Database opened.

查看crs启动resource信息：

[oracle@aix211 aix211]$crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora....11.lsnr application    ONLINE    ONLINE    aix211      ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora....12.lsnr application    ONLINE    ONLINE    aix212      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212      ora.prod.db    application    ONLINE    ONLINE    aix211      ora....d1.inst application    ONLINE    ONLINE    aix211      ora....d2.inst application    ONLINE    ONLINE    aix212

再通过crs工具重新启动Instance：

[oracle@aix211 aix211]$srvctl stop instance -d prod -i prod1

[oracle@aix211 aix211]$crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora....11.lsnr application    ONLINE    ONLINE    aix211      ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora....12.lsnr application    ONLINE    ONLINE    aix212      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212      ora.prod.db    application    ONLINE    ONLINE    aix211      ora....d1.inst application    OFFLINE   OFFLINE               ora....d2.inst application    ONLINE    ONLINE    aix212

[oracle@aix211 aix211]$srvctl start instance -d prod -i prod1

[oracle@aix211 aix211]$crs_stat -tName           Type           Target    State     Host        ------------------------------------------------------------ora....11.lsnr application    ONLINE    ONLINE    aix211      ora.aix211.gsd application    ONLINE    ONLINE    aix211      ora.aix211.ons application    ONLINE    ONLINE    aix211      ora.aix211.vip application    ONLINE    ONLINE    aix211      ora....12.lsnr application    ONLINE    ONLINE    aix212      ora.aix212.gsd application    ONLINE    ONLINE    aix212      ora.aix212.ons application    ONLINE    ONLINE    aix212      ora.aix212.vip application    ONLINE    ONLINE    aix212      ora.prod.db    application    ONLINE    ONLINE    aix211      ora....d1.inst application    ONLINE    ONLINE    aix211      ora....d2.inst application    ONLINE    ONLINE    aix212

@至此，通过crs工具可以正常启动和关闭Database，由于误操作而引起的血案，抢救成功！