rac has,crsd启动跟踪--linux平台RAC日志误删
删掉了$CRS_HOME/log/$HOSTNAME/下的所有日志文件和目录，于是has启不来了,单步执行：
sh -x /etc/init/ohasd
strace -p ohasd的pid
查看/var/log/messages,$ORACLE_HOME/log/$HOSTNAME/alert_$HOSTNAME.log,发现其步骤如下：
检查/opt/crs/product/bin/crsctl.bin文件：
进入目录：/var/tmp/.oracle 建立socket文件；
进入目录：/etc/oracle/
读取目录：
/opt/crs/product/log/mysql-1/ohasd
写入文件：ohasdOUT.log
读取：
/opt/crs/product/crs/mesg/crsus.msb
执行：
/opt/crs/product/crs/install/s_crsconfig_mysql-1_env.txt
进入/opt/crs/product/log/mysql-1/agent/
写日志，日志目录如下：
|-- client
|-- crflogd
|-- crsd
| |-- oraagent_grid
| | |-- oraagent_grid.log
| | |-- oraagent_gridOUT.log
| | `-- oraagent_grid.pid
| |-- oraagent_oracle
| | `-- oraagent_oracle.log
| |-- orarootagent_root
| | |-- orarootagent_root.log
| | |-- orarootagent_rootOUT.log
| | `-- orarootagent_root.pid
| `-- scriptagent_grid
| |-- scriptagent_grid.log
| `-- scriptagent_gridOUT.log
|-- cssd
`-- ohasd
|-- oraagent_grid
| |-- oraagent_grid.log
| |-- oraagent_gridOUT.log
| `-- oraagent_grid.pid
|-- oracssdagent_root
| `-- oracssdagent_root.log
|-- oracssdmonitor_root
| `-- oracssdmonitor_root.log
`-- orarootagent_root
|-- orarootagent_root.log
|-- orarootagent_rootOUT.log
`-- orarootagent_root.pid
进入/opt/crs/product/log/mysql-1，接着写目录：
|-- crflogd
| |-- crflogd.log
| `-- crflogdOUT.log
|-- crfmond
| |-- crfmond.log
| `-- crfmondOUT.log
|-- crsd
| |-- crsd.log
| `-- crsdOUT.log
|-- cssd
| |-- core.3307
| |-- cssdOUT.log
| `-- ocssd.log
|-- ctssd
| |-- ctssdOUT.log
| `-- octssd.log
|-- diskmon
|-- evmd
| |-- evmd.log
| `-- evmdOUT.log
|-- gipcd
| |-- gipcd.log
| `-- gipcdOUT.log
|-- gnsd
|-- gpnpd
| |-- gpnpd.log
| |-- gpnpdOUT.log
| `-- mysql-1.pid
|-- mdnsd
| |-- mdnsd.log
| `-- mdnsdOUT.log
|-- ohasd
| |-- ohasd.log
| `-- ohasdOUT.log
|-- racg
`-- srvm
笔记一下，防止再次发生；
另磁盘的检查超时也可能导致RAC启动不了，报：
2016-12-17 16:14:46.845: [ CSSD][1029]clssgmShutDown: Received abortive shutdown request from client.2016-12-17 16:14:46.845: [ CSSD][1029]clssnmRemoveNodeInTerm: node 2, terminated due to Normal Shutdown. Removing from member and connected bitmaps2016-12-17 16:14:46.845: [ CSSD][1029]###################################2016-12-17 16:14:46.845: [ CSSD][1029]clssscExit: CSSD aborting from thread GMClientListener2016-12-17 16:14:46.845: [ CSSD][1029]###################################2016-12-17 16:14:46.845: [ CSSD][1029](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally2016-12-17 16:14:47.276: [ SKGFD][1286]Handle 1119160d0 from lib :UFS:: for disk :/dev/rhdiskpower103:
建议临时解决办法：修改启动的超时时间：
crsctl modify resource "ora.cssd" -attr "START_TIMEOUT=1200" -init
