情况描述
客户sap系统运行在aix系统,ibm v7000存储上,数据存放在三个1T的lun组成的vg的多个lv中.异常断电之后,但是给数据库使用的三个lun丢失了2个,从而使得所有vg/lv异常,通过硬件恢复出来异常的2个lun和剩余的1个lun一起,把所有的数据文件恢复出来.但是由于某种原因,出现部分block被覆盖(其中还包括两个文件头损坏).
坏块检测
对于恢复的所有文件,为了快速做一遍坏块检查,直接在恢复的win机器上使用过obet做了一次dbv检查obet实现对数据文件坏块检测功能(使用obet的dbv检查有几个好处:1>可以在win上面检测aix的数据文件;2>可以检测数据文件头损坏的数据文件其他block;3>检测速度比原生dbv快[每个文件内部加了并行检测]),检测结果如下
--其中两个文件头损坏File #62: E:\sr3_54\sr3.data54 (1280000 blocks) - Started: 2026-07-28 23:57:53File #62: rfile=0 (0x00000000) header_block_num=0 (0x00000000) filesize_status:NOfile 62, block 0: block all zerofile 62, block 1: block all zeroFile #84: E:\sr3_76\sr3.data76 (4185601 blocks) - Started: 2026-07-29 00:55:32File #84: rfile=638937491 (0x26156993) header_block_num=77152556 (0x0499412C) filesize_status:NOfile 84, block 1: rdba error (expected 1, got 69760), bad blockfile 84, block 2: rdba error (expected 2, got 589836), bad block--坏块汇总DBV completed at: 2026-07-29 00:58:10===============================================DBV Summary:Total blocks checked: 196287237Total all zero blocks found: 337417Total all rdba error blocks found: 1062468Total all tailchk error blocks found: 1Total all soft corrupted blocks found: 0Total all checksum error blocks found: 0Total bad blocks found: 1399886Execution time: 10507.00 seconds=============================================== |
这个统计下来好的block在99.3%左右,证明硬件层面的会效果已经非常好.
碎片工具进一步恢复
恢复公司文件系统层面恢复有数据块遗漏的可能,通过碎片工具(OraScan(Oracle 碎片扫描工具) 使用说明)进一步扫描
通过确认62号文件还有少量block可以进一步恢复（也就是说碎片扫描到的62号文件的block多于硬件公司恢复出来的62号文件里面好的block数量）,通过obet的merge功能进行填补
基于上述操作,对于lun里面的数据文件实现了最大效果恢复.
数据库恢复操作
1. 上次恢复文件到aix,offline掉异常文件头数据文件打开数据库
sapprd2:oraprd 8> sqlplus / as sysdbaSQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:41:28 2026Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL> select status from v$instance;STATUS------------------------------------STARTEDSQL> alter database mount;alter database mount*ERROR at line 1:ORA-00214: control file '/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf' version9742943 inconsistent with file '/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf'version 9742931 |
解决ctl不一致问题之后继续mount库恢复
sapprd2:oraprd 11> sqlplus / as sysdbaSQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:42:53 2026Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL> alter database mount;Database altered.SQL> alter database datafile 62,84 offline;Database altered.SQL> recover database ;Media recovery complete.SQL> alter database open;Database altered. |
2.对于两个损坏的数据文件头进行修复
由于现在aix环境的客户比较少,obet没有在aix环境下进行编译,因此直接使用bbed工具进行文件头修复（列举了主要操作过程）
BBED> copy file 83 block 1 to file 84 block 1 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 0 to 31 Dba:0x15000001------------------------------------------------------------------------ 0ba20000 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d <32 bytes per line>BBED> set offset 368 OFFSET 368BBED> d File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 368 to 399 Dba:0x15000001------------------------------------------------------------------------ 00000053 00000000 00000000 495dc097 00000000 00000000 00000000 00000000 <32 bytes per line>BBED> m /x 00000054 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 368 to 399 Dba:0x15000001------------------------------------------------------------------------ 00000054 00000000 00000000 495dc097 00000000 00000000 00000000 00000000 <32 bytes per line>BBED> set offset 52 OFFSET 52BBED> d File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 52 to 83 Dba:0x15000001------------------------------------------------------------------------ 00530003 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <32 bytes per line>BBED> m /x 0054 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 52 to 83 Dba:0x15000001------------------------------------------------------------------------ 00540003 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <32 bytes per line>BBED> set offset 4 OFFSET 4BBED> d File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 4 to 35 Dba:0x15000001------------------------------------------------------------------------ 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400 <32 bytes per line>BBED> m /x 15000001 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 4 to 35 Dba:0x15000001------------------------------------------------------------------------ 15000001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400 <32 bytes per line>BBED> m /x E1B0Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 100 to 611 Dba:0x15000001------------------------------------------------------------------------ e1b0853f 00040000 4083fcfe 33606b63 02502452 0000ca28 49680fb3 93b9f6b9 <32 bytes per line>BBED> set offset +2 OFFSET 102BBED> m /x 8542 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84) Block: 1 Offsets: 102 to 613 Dba:0x15000001------------------------------------------------------------------------ 85420004 00004083 fcfe3360 6b630250 24520000 ca284968 0fb393b9 f6b90005 <32 bytes per line>BBED> sumCheck value for File 84, Block 1:current = 0x2225, required = 0x23e5BBED> sum applyCheck value for File 84, Block 1:current = 0x23e5, required = 0x23e5BBED> verifyDBVERIFY - Verification startingFILE = /oracle/PRD/sapdata3/sr3_76/sr3.data76BLOCK = 1DBVERIFY - Verification completeTotal Blocks Examined : 1Total Blocks Processed (Data) : 0Total Blocks Failing (Data) : 0Total Blocks Processed (Index): 0Total Blocks Failing (Index): 0Total Blocks Empty : 0Total Blocks Marked Corrupt : 0Total Blocks Influx : 0Message 531 not found; product=RDBMS; facility=BBED |
修改完成之后,还出现过几个错误
SQL> alter database open ;alter database open*ERROR at line 1:ORA-01122: database file 62 failed verification checkORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'ORA-01200: actual file size of 1279999 is smaller than correct size of 1280000 |
ORA-01200是由于数据文件比文件头记录信息小一个block,通过补上这个block解决
Read of datafile '/oracle/PRD/sapdata2/sr3_54/sr3.data54' (fno 62) header failed with ORA-01202Rereading datafile 62 header failed with ORA-01202Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_11272278.trc:ORA-01122: database file 62 failed verification checkORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'ORA-01202: wrong incarnation of this file - wrong creation time |
ORA-01202是由于create time没有修改正确导致,重新修改解决
Rereading datafile 84 header failed with ORA-01203Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105252.trc:ORA-01122: database file 84 failed verification checkORA-01110: data file 84: '/oracle/PRD/sapdata3/sr3_76/sr3.data76'ORA-01203: wrong incarnation of this file - wrong creation SCN |
ORA-01203是由于create scn没有修改正确导致,重新修改解决
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105278.trc:ORA-01122: database file 62 failed verification checkORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'ORA-01207: file is more recent than control file - old control fileORA-1122 signalled during: alter database open . |
ORA-01207是由于数据文件的ckp信息比控制文件的新,重建ctl解决,解决这些问题之后,顺利打开数据库
SQL> startup mount;ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instanceORACLE instance started.Total System Global Area 3.7548E+10 bytesFixed Size 2254136 bytesVariable Size 1.9193E+10 bytesDatabase Buffers 1.8254E+10 bytesRedo Buffers 98996224 bytesDatabase mounted.SQL> alter database open;Database altered.SQL> select status,count(1) from v$datafile_header group by status;STATUS COUNT(1)------- ----------ONLINE 84 |
然后跳过坏块,导出数据,对于无法导出的异常表进行特殊处理,完成本次恢复工作,最终恢复结果总结
浙公网安备 33010602011771号