ORA-27300: OS system dependent operation:fork failed with status

问题现象：

Sun Sep 30 21:12:28 2018Errors in file /DBSoft2/oracle/diag/rdbms/fzdbcgbk/fzdbcgbk1/trace/fzdbcgbk1_psp0_221463.trc:ORA-27300: ͳϵͳز: fork ʧ ״̬Ϊ: 11ORA-27301: ͳϢ: Resource temporarily unavailableORA-27302: skgpspawn3Process J000 died, see its trace fileSun Sep 30 21:12:28 2018kkjcre1p: unable to spawn jobq slave process Sun Sep 30 21:12:28 2018Errors in file /DBSoft2/oracle/diag/rdbms/fzdbcgbk/fzdbcgbk1/trace/fzdbcgbk1_cjq0_222890.trc:Sun Sep 30 21:12:29 2018Process startup failed, error stack:Sun Sep 30 21:12:29 2018Errors in file /DBSoft2/oracle/diag/rdbms/fzdbcgbk/fzdbcgbk1/trace/fzdbcgbk1_psp0_221463.trc:ORA-27300: ͳϵͳز: fork ʧ ״̬Ϊ: 11ORA-27301: ͳϢ: Resource temporarily unavailableORA-27302: skgpspawn3

参考资料：

Mos ID:392006.1

SYMPTOMS：

The following message reported in alert log:

Mon Sep 18 18:10:34 2006

Errors in file /u01/oracle/admin/orcl/bdump/orcl1_psp0_954436.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn3

Mon Sep 18 18:10:35 2006

Process P073 died, see its trace file

CAUSE：

The error messages indicating that oracle has problem in forking more process, the maximum number of PROCESSES allowed per user could be too low.

This is configured by the following kernel settings:

AIX: maxuproc

HP and Solaris: maxuprc

Solaris 10/11: project.max-lwps

Linux: nproc (/etc/security/limits.conf)

检查配置：

[root@fzdb1 trace]# su - oracle2[oracle2@fzdb1 ~]$ ulimit -acore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 1031748max locked memory       (kbytes, -l) 64max memory size         (kbytes, -m) unlimitedopen files                      (-n) 1024pipe size            (512 bytes, -p) 8POSIX message queues     (bytes, -q) 819200real-time priority              (-r) 0stack size              (kbytes, -s) 10240cpu time               (seconds, -t) unlimitedmax user processes              (-u) 3047virtual memory          (kbytes, -v) unlimitedfile locks                      (-x) unlimited[oracle2@fzdb1 ~]$ ulimit -u 3047[root@fzdb1 ~]# grep proc /etc/security/limits.conf#        - nproc - max number of processes#        - priority - the priority to run user process with#@student        hard    nproc           20#@faculty        soft    nproc           20#@faculty        hard    nproc           50#ftp             hard    nproc           0oracle           soft    nproc           3047oracle           hard    nproc           16384grid             soft    nproc           3047grid             hard    nproc           16384

问题分析：

从上面检查我们可以发现oracle2的用户进程数量已经达到了1491，几乎接近oracle2用户的user processes数量，所以数据库日志里面频繁出现大量的ORA-27300,ORA-27301,ORA-27302，根据MOS:392006.1建议修改/etc/security/limits.conf文件里的oracle2 nproc值为更大的值，但是实际上我们并没有发现有关于oracle2用户的配置，所以我们手工添加如下配置后重启数据库服务器就可以了:

oracle2 soft nproc 16384

oracle2 hard nproc 16384

oracle2 soft nofile 65536

oracle2 hard nofile 65536

oracle2 soft memlock 3145728

oracle2 hard memlock 3145728

开始修改：

[root@fzdb1 ~]# cat  >> /etc/security/limits.conf << EOF> oracle2         soft    nproc           16384> oracle2         hard    nproc           16384> oracle2         soft    nofile          65536> oracle2         hard    nofile          65536> oracle2         soft    memlock         3145728> oracle2         hard    memlock         3145728> EOF

重启服务器：

[oracle2@fzdb1 ~]$ su - root[root@fzdb1 ~]$ reboot

检查刚才的参数修改：

[oracle2@fzdb1 ~]$ ulimit -u16384

检查数据库日志：

修改参数之后通过一天的观察未见ORA-27300,ORA-27301,ORA-27302等报错信息，此故障处理完毕。