存储强制拉lun导致数据库异常恢复----惜分飞

标题：存储强制拉lun导致数据库异常恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户那边反馈有几个lun异常,无法正常online

---

通过存储工程师强制拉起来lun(清除掉了cache),但是数据库无法正常mount

 

Fri Sep 30 17:22:57 BEIST 2022

ALTER DATABASE   MOUNT

Fri Sep 30 17:22:57 BEIST 2022

This instance was first to mount

Fri Sep 30 17:22:58 BEIST 2022

Starting background process ASMB

ASMB started with pid=25, OS id=12976304

Starting background process RBAL

RBAL started with pid=26, OS id=12779520

Fri Sep 30 17:23:02 BEIST 2022

SUCCESS: diskgroup DATA was mounted

Fri Sep 30 17:23:06 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/udump/xifenfei2_ora_14549110.trc:

ORA-00600: internal error code, arguments: [kccpb_sanity_check_2], [423012], [422765], [0x000000000], [], [], [], []

Fri Sep 30 17:23:07 BEIST 2022

ORA-600 signalled during: ALTER DATABASE   MOUNT...

Fri Sep 30 17:23:07 BEIST 2022

Trace dumping is performing id=[cdmp_20220930172307]

Fri Sep 30 17:23:09 BEIST 2022

Shutting down instance (abort)

License high water mark = 1

Instance terminated by USER, pid = 9175148

可以要求保护第一现场,把asm中的数据恢复到文件系统中,然后进行恢复,由于客户是10g的环境,无法直接使用asmcmd中的cp实现此项操作,数据库也没有mount成功(无法使用rman的copy),考虑使用oracle的amdu实现此项操作需求.在拷贝过程中报AMDU-00204报错

root@xifenfei2:/recover/amduo#./amdu -diskstring '/dev/rhdiskpower*' -extract data.298 -noreport

amdu_2022_10_01_14_25_31/

AMDU-00204: file not found; arguments: [3] [DATA]

LEM-00031: Error encountered in lempgmh after calling lmserr.

通过dbv校验拷贝出来的数据文件

racle@xifenfei2:/recover#dbv file=/recover/amduo/amdu_2022_10_01_14_25_31/DATA_298.f

 

DBVERIFY: Release 10.2.0.5.0 - Production on Sat Oct 1 14:36:50 2022

 

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

 

DBVERIFY - Verification starting : FILE = /recover/amduo/amdu_2022_10_01_14_33_26/DATA_298.f

 

 

DBVERIFY - Verification complete

 

Total Pages Examined         : 262144

Total Pages Processed (Data) : 0

Total Pages Failing   (Data) : 0

Total Pages Processed (Index): 0

Total Pages Failing   (Index): 0

Total Pages Processed (Other): 262143

Total Pages Processed (Seg)  : 0

Total Pages Failing   (Seg)  : 0

Total Pages Empty            : 1

Total Pages Marked Corrupt   : 0

Total Pages Influx           : 0

Highest block SCN            : 121293100 (0.121293100)

确认此报错（AMDU-00204 LEM-00031）对于拷贝出来的数据文件无直接影响，可以忽略,拷贝出来所有文件进行重建ctl，报ORA-01159错误

SQL> CREATE CONTROLFILE REUSE DATABASE "xifenfei" NORESETLOGS  NOARCHIVELOG 

  2      MAXLOGFILES 50 

    MAXLOGMEMBERS 5 

    MAXDATAFILES 100 

    MAXINSTANCES 8 

    MAXLOGHISTORY 226 

LOGFILE 

group 5 '/recover/df/DATA_262.f'  size 200M ,

group 2 '/recover/df/DATA_266.f'  size 200M ,

group 1 '/recover/df/DATA_267.f'  size 200M ,

group 3 '/recover/df/DATA_281.f'  size 200M ,

group 4 '/recover/df/DATA_282.f'  size 200M ,

group 6 '/recover/df/DATA_283.f'  size 200M

DATAFILE 

'/recover/df/DATA_295.f',

'/recover/df/DATA_298.f',

'/recover/df/DATA_272.f',

'/recover/df/DATA_273.f',

'/recover/df/DATA_296.f',

'/recover/df/DATA_274.f',

'/recover/df/DATA_276.f',

'/recover/df/DATA_277.f',

'/recover/df/DATA_275.f',

'/recover/df/DATA_279.f',

'/recover/df/DATA_278.f',

'/recover/df/DATA_288.f',

'/recover/df/DATA_269.f',

'/recover/df/DATA_300.f',

'/recover/df/DATA_264.f',

'/recover/df/DATA_287.f',

'/recover/df/DATA_280.f',

'/recover/df/DATA_286.f',

'/recover/df/DATA_268.f',

'/recover/df/DATA_285.f',

'/recover/df/DATA_297.f'

CHARACTER SET UTF8 

; 

 37  CREATE CONTROLFILE REUSE DATABASE "xifenfei" NORESETLOGS  NOARCHIVELOG

*

ERROR at line 1:

ORA-01503: CREATE CONTROLFILE failed

ORA-01159: file is not from same database as previous files - wrong database id

ORA-01110: data file 3: '/recover/df/DATA_288.f'

由于部分文件不是该库的,通过进一步分析,除掉不是该库的文件,重建ctl文件成功.尝试recover数据库，报大量ORA-07445错误，由于cache丢失redo损坏导致,此类操作可能导致数据文件损坏【恢复需要谨慎,最好对数据文件做一次备份】

Sat Oct 01 16:23:16 BEIST 2022

ALTER DATABASE RECOVER  database 

Media Recovery Start

 parallel recovery started with 15 processes

Sat Oct 01 16:23:16 BEIST 2022

Recovery of Online Redo Log: Thread 1 Group 5 Seq 2202 Reading mem 0

  Mem# 0: /recover/df/DATA_262.f

Sat Oct 01 16:23:16 BEIST 2022

Recovery of Online Redo Log: Thread 2 Group 6 Seq 2394 Reading mem 0

  Mem# 0: /recover/df/DATA_283.f

Sat Oct 01 16:23:28 BEIST 2022

Recovery of Online Redo Log: Thread 2 Group 3 Seq 2395 Reading mem 0

  Mem# 0: /recover/df/DATA_281.f

Sat Oct 01 16:23:34 BEIST 2022

Recovery of Online Redo Log: Thread 1 Group 1 Seq 2203 Reading mem 0

  Mem# 0: /recover/df/DATA_267.f

Sat Oct 01 16:23:35 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p009_13107402.trc:

ORA-07445: exception encountered: core dump [kcbzfc+00dc] [SIGSEGV]

Sat Oct 01 16:23:35 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p014_7929944.trc:

ORA-07445: exception encountered: core dump [kcbzfc+00dc] [SIGSEGV]

Sat Oct 01 16:23:35 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p011_10092678.trc:

ORA-07445: exception encountered: core dump [kcbzfc+00dc] [SIGSEGV]

Sat Oct 01 16:23:35 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_dbw1_12189898.trc:

ORA-07445: exception encountered: core dump [kcbzdh+0324] [SIGSEGV]

Sat Oct 01 16:23:35 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p005_14549014.trc:

ORA-07445: exception encountered: core dump [kcbbufaddr2hdr+00d8] [SIGSEGV]

Sat Oct 01 16:23:35 BEIST 2022

Hex dump of (file 6, block 10) in trace file /home/oracle/admin/xifenfei/bdump/xifenfei2_dbw0_13565988.trc

Corrupt block relative dba: 0x0180000a (file 6, block 10)

Bad header found during buffer corrupt after write

Data in bad block:

 type: 2 format: 1 rdba: 0x00000180

 last change scn: 0xa0c3.000a6eeb seq: 0x0 flg: 0x00

 spare1: 0x2 spare2: 0xa2 spare3: 0x3486

 consistency value in tail: 0x0000a0c3

 check value in block header: 0x204

 block checksum disabled

Reread of rdba: 0x0180000a (file 6, block 10) found different data

Sat Oct 01 16:23:35 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_dbw0_13565988.trc:

ORA-07445: exception encountered: core dump [kcbbiop+01b8] [SIGSEGV] [Invalid permissions for mapped object]

Sat Oct 01 16:23:36 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p014_7929944.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbzfc+00dc] [SIGSEGV] [Address not mapped to object]

Sat Oct 01 16:23:36 BEIST 2022

Trace dumping is performing id=[cdmp_20221001162336]

Sat Oct 01 16:23:37 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p011_10092678.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbzfc+00dc] [SIGSEGV] [Address not mapped to object]

Sat Oct 01 16:23:37 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p009_13107402.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbzfc+00dc] [SIGSEGV] [Address not mapped to object]

Sat Oct 01 16:23:37 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_dbw1_12189898.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbzdh+0324] [SIGSEGV] [Address not mapped to object]

Sat Oct 01 16:23:37 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p005_14549014.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbbufaddr2hdr+00d8] [SIGSEGV] [Address not mapped to object]

Sat Oct 01 16:23:37 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_dbw0_13565988.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbbiop+01b8] [SIGSEGV] [Invalid permissions for mapped object]

Sat Oct 01 16:23:37 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p005_14549014.trc:

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbbufaddr2hdr+00d8] [SIGSEGV] [Address not mapped to object]

Sat Oct 01 16:23:38 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p010_9502918.trc:

ORA-00600: internal error code, arguments: [3020], [3], [31913], [2], [2395], [210418], [16], []

ORA-10567: Redo is inconsistent with data block (file# 3, block# 31913)

ORA-10564: tablespace SYSAUX

ORA-01110: data file 3: '/recover/df/DATA_278.f'

ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK'

Sat Oct 01 16:23:39 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p010_9502918.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-00600: internal error code, arguments: [3020], [3], [31913], [2], [2395], [210418], [16], []

ORA-10567: Redo is inconsistent with data block (file# 3, block# 31913)

ORA-10564: tablespace SYSAUX

ORA-01110: data file 3: '/recover/df/DATA_278.f'

ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK'

Sat Oct 01 16:23:39 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_pmon_14418166.trc:

ORA-00471: DBWR process terminated with error

Sat Oct 01 16:23:39 BEIST 2022

PMON: terminating instance due to error 471

Sat Oct 01 16:23:40 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/bdump/xifenfei2_p010_9502918.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state+027c] [SIGSEGV]

ORA-00600: internal error code, arguments: [3020], [3], [31913], [2], [2395], [210418], [16], []

ORA-10567: Redo is inconsistent with data block (file# 3, block# 31913)

ORA-10564: tablespace SYSAUX

ORA-01110: data file 3: '/recover/df/DATA_278.f'

ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK'

Sat Oct 01 16:23:46 BEIST 2022

Dump system state for local instance only

System State dumped to trace file /home/oracle/admin/xifenfei/bdump/xifenfei2_diag_15401212.trc

Sat Oct 01 16:23:46 BEIST 2022

Trace dumping is performing id=[cdmp_20221001162346]

Sat Oct 01 16:23:49 BEIST 2022

Instance terminated by PMON, pid = 14418166

绕过redo,直接强制启动库,报ORA-01092错误

QL> startup mount pfile='/tmp/pfile'

ORACLE instance started.

 

Total System Global Area      10737418240 bytes

Fixed Size                        2114208 bytes

Variable Size                  1560284512 bytes

Database Buffers               9160359936 bytes

Redo Buffers                     14659584 bytes

Database mounted.

SQL> alter database open resetlogs;

alter database open resetlogs

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

分析alert日志,确认是由于undo异常导致

Additional information: 3

Sat Oct 01 17:25:21 BEIST 2022

Setting recovery target incarnation to 2

Sat Oct 01 17:25:21 BEIST 2022

Assigning activation ID 1094862311 (0x414245e7)

Thread 1 opened at log sequence 1

  Current log# 1 seq# 1 mem# 0: /recover/df/DATA_267.f1

Successful open of redo thread 1

Sat Oct 01 17:25:21 BEIST 2022

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

Sat Oct 01 17:25:21 BEIST 2022

SMON: enabling cache recovery

Sat Oct 01 17:25:23 BEIST 2022

ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0000.6ee0bde5):

Sat Oct 01 17:25:23 BEIST 2022

select ctime, mtime, stime from obj$ where obj# = :1

Sat Oct 01 17:25:23 BEIST 2022

Errors in file /home/oracle/admin/xifenfei/udump/xifenfei2_ora_19726450.trc:

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 1

ORA-01555: snapshot too old: rollback segment number 8 with name "_SYSSMU8$" too small

Error 704 happened during db open, shutting down database

USER: terminating instance due to error 704

Instance terminated by USER, pid = 19726450

ORA-1092 signalled during: alter database open resetlogs...

类似此类错误的解决方案,以前写过参考:
在数据库open过程中常遇到ORA-01555汇总
数据库open过程遭遇ORA-1555对应sql语句补充
Oracle Recovery Tools恢复—ORA-00704 ORA-01555故障
使用_allow_resetlogs_corruption导致ORA-00704/ORA-01555故障
解决该问题,数据库启动正常,逻辑导出数据，导入数据完成此次恢复任务,实现绝大部分数据恢复

• 110T oracle故障恢复
• ORA-01172 ORA-01151 故障恢复
• spfile被覆盖导致ORA-600[kmgs_parameter_update_timeout_1]
• 又一例ORA-600 kcratr_nab_less_than_odr
• ORA-10562 故障恢复—allow 1 corruption
• O/S-Error: (OS 23) 数据错误(循环冗余检查) 数据库恢复
• 强制关机导致数据库无法正常启动恢复
• 记录一次ORA-00600[kdxlin:psno out of range]/ORA-00600[3020]/ORA-00600[4000]/ORA-00600[4193]的数据库恢复
• 10g数据库遭遇ORA-600 16703
• 存储重启,oracle无法启动故障处理
• ora-600 2037 ORA-7445 kcbs_dump_adv_state解决
• ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []