ORA-19909(一次DataGuard Failover导致的故障
某医院客户，核心数据库有两个备库。因想将其中一个备库主机挪作测试使用，所以，想把数据库切换成读写模式。这样，就涉及到了做备库的failover动作。这个failover是在另外一个备库，主库都存活的情况下进行的。
Faiover进行得很顺利。Failover之后，修改了此数据库的DG的几个参数，当天以为事情就这么完了。
第二日，客户告知另一个备库的数据没有同步。于是登录过去，看到告警日志中有如下错误
MRP0: Incarnation has changed! Retry recovery...
Errors in file /oradata/u01/app/diag/rdbms/orcldg/orcl/trace/orcl_pr00_3272.trc:
ORA-19906: recovery target incarnation changed during recovery
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 19274002051
Tue Jun 09 16:37:39 2020
started logmerger process
Tue Jun 09 16:37:39 2020
Managed Standby Recovery starting Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 19274002051) is orphaned on incarnation#=3
MRP0: Detected orphaned datafiles!
Recovery will possibly be retried after flashback...
Errors in file /oradata/u01/app/diag/rdbms/orcldg/orcl/trace/orcl_pr00_27714.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/oradata/datafile/system01.dbf'
通过RMAN查看incarnation
connected to target database: ORCL (DBID=1385973181)
RMAN> list incarnation;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 ORCL 1385973181 PARENT 1 13-SEP-14
2 2 ORCL 1385973181 PARENT 4733207525 12-FEB-15
3 3 ORCL 1385973181 CURRENT 8619762801 08-JUL-17
4 4 ORCL 1385973181 ORPHAN 19273953428 09-JUN-20
其中incarnation 4正是昨天做failover产生的。
解决起来其实也相对较为简单：
reset database incarnation to 3;
然后在开启日志应用即可
Wed Jun 10 11:10:26 2020
RFS[5]: Assigned to RFS process 19310
RFS[5]: Allowing overwrite of partial archivelog for thread 2 sequence 40151
RFS[5]: Opened log for thread 2 sequence 40151 dbid 1385973181 branch 948759254
Completed: alter database recover managed standby database using current logfile disconnect
Archived Log entry 72765 added for thread 2 sequence 40151 rlc 948759254 ID 0x5d2e975e dest 2:
Media Recovery Log /oradata/arch/2_40151_948759254.dbf
Media Recovery Waiting for thread 1 sequence 1748
Archivelog record exists, but no file is found
RFS[5]: Opened log for thread 1 sequence 36160 dbid 1385973181 branch 948759254
Archived Log entry 72766 added for thread 1 sequence 36160 rlc 948759254 ID 0x5d2e975e dest 2:
MRP0: Log being waited on was inappropriate for recovery. Retry recovery...
Errors in file /oradata/u01/app/diag/rdbms/orcldg/orcl/trace/orcl_pr00_19224.trc:
ORA-16426: Recovery requested an incorrect log from which to apply redo data.
Managed Standby Recovery not using Real Time Apply
Wed Jun 10 11:10:34 2020
Recovery interrupted!
Wed Jun 10 11:10:54 2020
started logmerger process
Wed Jun 10 11:10:55 2020
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 16 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /oradata/arch/2_40151_948759254.dbf
Media Recovery Log /oradata/arch/1_36160_948759254.dbf
Media Recovery Log /oradata/arch/2_40152_948759254.dbf
Media Recovery Log /oradata/arch/1_36161_948759254.dbf
此次故障，应当在做failover之前修改主备库的DG相关参数。作为数据库工程师，应该考虑更为周全。
幸而影响范围较小。
