asm dd 10M导致system文件部分坏块修复---惜分飞

有客户的数据库由于不当操作导致asm磁盘头损坏,进行的操作命令类似

dd if=/dev/zero of=/dev/dm-29 bs=1024K count=10

asm磁盘组无法mount,提示ORA-15042

SQL> ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {1:1712:2} */

2026-05-19T21:57:16.517284-04:00

NOTE: cache registered group DATA 5/0xAE103296

NOTE: cache began mount (first) of group DATA 5/0xAE103296

NOTE: Assigning number (5,10) to disk (/dev/dm-29)

…………

NOTE: Assigning number (5,18) to disk (/dev/dm-15)

NOTE: Assigning number (5,0) to disk (/dev/dm-7)

NOTE: Assigning number (5,15) to disk (/dev/dm-5)

2026-05-19T21:57:16.650481-04:00

cluster guid (b216d47e2bf86f6aff34a36119b0c161) generated for PST Hbeat for instance 1

2026-05-19T21:57:22.659262-04:00

NOTE: GMON heartbeating for grp 5 (DATA)

GMON querying group 5 at 28 for pid 42, osid 12996

2026-05-19T21:57:22.661447-04:00

NOTE: Assigning number (5,14) to disk ()

2026-05-19T21:57:22.662476-04:00

GMON querying group 5 at 29 for pid 42, osid 12996

2026-05-19T21:57:22.663144-04:00

NOTE: cache dismounting (clean) group 5/0xAE103296 (DATA)

NOTE: messaging CKPT to quiesce pins Unix process pid: 12996, image: oracle@dlycdb1 (TNS V1-V3)

NOTE: dbwr not being msg'd to dismount

NOTE: LGWR not being messaged to dismount

NOTE: cache dismounted group 5/0xAE103296 (DATA)

NOTE: cache ending mount (fail) of group DATA number=5 incarn=0xae103296

NOTE: cache deleting context for group DATA 5/0xae103296

2026-05-19T21:57:22.720673-04:00

GMON dismounting group 5 at 30 for pid 42, osid 12996

2026-05-19T21:57:22.721055-04:00

NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment

NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment

…………

NOTE: Disk DATA_0013 in mode 0x7f marked for de-assignment

NOTE: Disk DATA_0015 in mode 0x7f marked for de-assignment

NOTE: Disk DATA_0016 in mode 0x7f marked for de-assignment

NOTE: Disk DATA_0017 in mode 0x7f marked for de-assignment

NOTE: Disk DATA_0018 in mode 0x7f marked for de-assignment

ERROR: diskgroup DATA was not mounted

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "14" is missing from group number "5"

 

2026-05-19T21:57:22.745140-04:00

ERROR: ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {1:1712:2} */

由于客户这个是19c版本,直接使用备份au还原,然后mount磁盘组成功

SQL> alter diskgroup data mount

2026-05-19T22:40:59.137657+08:00

NOTE: cache registered group DATA 2/0xB126DA41

NOTE: cache began mount (first) of group DATA 2/0xB126DA41

NOTE: Assigning number (2,14) to disk (/dev/dm-29)

NOTE: Assigning number (2,8) to disk (/dev/dm-28)

…………

NOTE: Assigning number (2,2) to disk (/dev/dm-7)

NOTE: Assigning number (2,0) to disk (/dev/dm-6)

NOTE: Assigning number (2,1) to disk (/dev/dm-3)

2026-05-19T22:40:59.303908+08:00

cluster guid (b216d47e2bf86f6aff34a36119b0c161) generated for PST Hbeat for instance 2

2026-05-19T22:41:05.312792+08:00

NOTE: GMON heartbeating for grp 2 (DATA)

GMON querying group 2 at 80 for pid 35, osid 53051

2026-05-19T22:41:05.314925+08:00

NOTE: cache is mounting group DATA created on 2024/11/28 17:55:45

NOTE: cache opening disk 0 of grp 2: DATA_0000 path:/dev/dm-6

NOTE: 05/20/26 16:41:04 DATA.F1X0 found on disk 0 au 10 fcn 0.0 datfmt 1

NOTE: cache opening disk 1 of grp 2: DATA_0001 path:/dev/dm-3

…………

NOTE: cache opening disk 13 of grp 2: DATA_0013 path:/dev/dm-18

NOTE: cache opening disk 14 of grp 2: DATA_0014 path:/dev/dm-29

NOTE: cache opening disk 15 of grp 2: DATA_0015 path:/dev/dm-16

NOTE: cache opening disk 16 of grp 2: DATA_0016 path:/dev/dm-17

NOTE: cache opening disk 17 of grp 2: DATA_0017 path:/dev/dm-20

NOTE: cache opening disk 18 of grp 2: DATA_0018 path:/dev/dm-21

2026-05-19T22:41:05.317307+08:00

NOTE: cache mounting (first) external redundancy group 2/0xB126DA41 (DATA)

2026-05-19T22:41:05.522191+08:00

NOTE: attached to recovery domain 2

2026-05-19T22:41:05.558136+08:00

validate pdb 2, flags x4, valid 0, pdb flags x204

* validated domain 2, flags = 0x200

NOTE: cache recovered group 2 to fcn 0.46336611

NOTE: redo buffer size is 512 blocks (2105344 bytes)

2026-05-19T22:41:05.569479+08:00

NOTE: LGWR attempting to mount thread 1 for diskgroup 2 (DATA)

NOTE: LGWR found thread 1 closed at ABA 110.6507 lock domain=0 inc#=0 instnum=1

NOTE: LGWR mounted thread 1 for diskgroup 2 (DATA)

2026-05-19T22:41:05.574499+08:00

mntstmp=2026/05/20 16:41:05.572000

2026-05-19T22:41:05.574854+08:00

NOTE: cache mounting group 2/0xB126DA41 (DATA) succeeded

NOTE: cache ending mount (success) of group DATA number=2 incarn=0xb126da41

2026-05-19T22:41:05.616227+08:00

NOTE: Instance updated compatible.asm to 19.0.0.0.0 for grp 2 (DATA).

2026-05-19T22:41:05.616754+08:00

NOTE: Instance updated compatible.asm to 19.0.0.0.0 for grp 2 (DATA).

2026-05-19T22:41:05.617932+08:00

NOTE: Instance updated compatible.rdbms to 19.0.0.0.0 for grp 2 (DATA).

2026-05-19T22:41:05.618303+08:00

NOTE: Instance updated compatible.rdbms to 19.0.0.0.0 for grp 2 (DATA).

2026-05-19T22:41:05.643844+08:00

SUCCESS: diskgroup DATA was mounted

虽然磁盘组mount成功,但是asm依旧在报错

这个客户胆子也真够大的,这样mount起来的磁盘组,还往里面加入磁盘出发Rebalance操作

1

2026-05-20T09:18:34.292103-04:00

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc:

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1014] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1014] [0 != 1]

NOTE: cache repaired a corrupt block: group=2(DATA) dsk=14 blk=1014 on disk 14 from disk=14 (DATA_0014) incarn=4042690154 au=11 blk=1014 count=1

2026-05-20T09:18:36.721669-04:00

WARNING: cache read a corrupt block: group=2(DATA) dsk=14 blk=1015 disk=14 (DATA_0014) incarn=4042690154 au=0 blk=1015 count=1

2026-05-20T09:18:36.721982-04:00

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc:

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1015] [0 != 1]

NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc

WARNING: cache read (retry) a corrupt block: group=2(DATA) dsk=14 blk=1015 disk=14 (DATA_0014) incarn=4042690154 au=0 blk=1015 count=1

2026-05-20T09:18:36.724279-04:00

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc:

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1015] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1015] [0 != 1]

NOTE: cache repaired a corrupt block: group=2(DATA) dsk=14 blk=1015 on disk 14 from disk=14 (DATA_0014) incarn=4042690154 au=11 blk=1015 count=1

2026-05-20T09:44:20.469792-04:00

NOTE: Starting expel slave for group 2/0xcd67eb4 (DATA)

2026-05-20T09:44:20.473880-04:00

NOTE: GroupBlock outside rolling migration privileged region

NOTE: requesting all-instance membership refresh for group=2

2026-05-20T09:44:20.530101-04:00

NOTE: membership refresh pending for group 2/0xcd67eb4 (DATA)

2026-05-20T09:44:20.534097-04:00

GMON querying group 2 at 178 for pid 27, osid 70130

2026-05-20T09:44:20.557651-04:00

SUCCESS: refreshed membership for 2/0xcd67eb4 (DATA)

2026-05-20T09:44:23.489304-04:00

NOTE: Attempting voting file refresh on diskgroup DATA

NOTE: Refresh completed on diskgroup DATA. No voting file found.

2026-05-20T11:20:49.804413-04:00

NOTE: stopping process ARB0

NOTE: stopping process ARBA

2026-05-20T11:20:51.538777-04:00

SUCCESS: rebalance completed for group 2/0xcd67eb4 (DATA)

2026-05-20T23:20:51.541462+08:00

SUCCESS: ALTER DISKGROUP DATA ADD  DISK '/dev/dm-20' SIZE 4194304M

 REBALANCE WAIT

算幸运,由于ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] 错误导致rebalance没有真正运行起来,从而该磁盘组没有dismount(19c这个方面确实增强不少,如果以前版本大概率会直接dismount掉)

客户在这样mount的磁盘组上尝试启动库,报ORA-01578错误,无法启动成功

2026-05-20T17:46:07.060794+08:00

Error attempting to elevate LMHB's priority: no further priority changes will be attempted for this process

2026-05-20T17:46:07.647114+08:00

Undo initialization recovery: Parallel FPTR complete: start:26091908 end:26096229 diff:4321 ms (4.3 seconds)

Undo initialization recovery: err:0 start: 26091907 end: 26096229 diff: 4322 ms (4.3 seconds)

[50880] Successfully onlined Undo Tablespace 5.

Undo initialization online undo segments: err:0 start: 26096229 end: 26096576 diff: 347 ms (0.3 seconds)

Undo initialization finished serial:0 start:26091907 end:26096591 diff:4684 ms (4.7 seconds)

Database Characterset is AL32UTF8

No Resource Manager plan active

2026-05-20T17:46:08.734113+08:00

 

Corrupt block relative dba: 0x004030ee (file 1, block 12526)

Completely zero block found during buffer read

 

Reread (file 1, block 12526) found same corrupt data (no logical check)

2026-05-20T17:46:08.811455+08:00

Corrupt Block Found

         TIME STAMP (GMT) = 05/20/2026 17:46:07

         CONT = 0, TSN = 0, TSNAME = SYSTEM

         RFN = 1, BLK = 12526, RDBA = 4206830

         OBJN = 37, OBJD = 37, OBJECT = I_OBJ2, SUBOBJECT =

         SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc  (incident=800708):

ORA-01578: ORACLE data block corrupted (file # 1, block # 12526)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T05:46:09.047772-04:00

ALTER SYSTEM SET remote_listener=' xff-scan:11521' SCOPE=MEMORY SID='xff2';

2026-05-20T05:46:09.049615-04:00

ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='xff2';

2026-05-20T17:46:09.812271+08:00

*****************************************************************

An internal routine has requested a dump of selected redo.

This usually happens following a specific internal error, when

analysis of the redo logs will help Oracle Support with the

diagnosis.

It is recommended that you retain all the redo logs generated (by

all the instances) during the past 12 hours, in case additional

redo dumps are required to help with the diagnosis.

*****************************************************************

 

Corrupt block relative dba: 0x004030de (file 1, block 12510)

Completely zero block found during buffer read

 

Reread (file 1, block 12510) found same corrupt data (no logical check)

2026-05-20T17:46:10.350573+08:00

Corrupt Block Found

         TIME STAMP (GMT) = 05/20/2026 17:46:09

         CONT = 0, TSN = 0, TSNAME = SYSTEM

         RFN = 1, BLK = 12510, RDBA = 4206814

         OBJN = 83, OBJD = 83, OBJECT = DEPENDENCY$, SUBOBJECT =

         SEGMENT OWNER = SYS, SEGMENT TYPE = Table Segment

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc  (incident=800709):

ORA-01578: ORACLE data block corrupted (file # 1, block # 12510)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:10.694365+08:00

 

Corrupt block relative dba: 0x00403000 (file 1, block 12288)

 

Completely zero block found during validation

Reread of blocknum=12288, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12288, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12288, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12288, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12288, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

 

Corrupt block relative dba: 0x00403001 (file 1, block 12289)

 

Completely zero block found during validation

Reread of blocknum=12289, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12289, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12289, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12289, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12289, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

………………

Corrupt block relative dba: 0x004030ff (file 1, block 12543)

 

Completely zero block found during validation

Reread of blocknum=12543, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12543, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12543, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12543, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Reread of blocknum=12543, file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

 

Corrupt block relative dba: 0x0040301d (file 1, block 12317)

Completely zero block found during buffer read

 

Reread (file 1, block 12317) found same corrupt data (no logical check)

2026-05-20T17:46:12.183545+08:00

Corrupt Block Found

         TIME STAMP (GMT) = 05/20/2026 17:46:11

         CONT = 0, TSN = 0, TSNAME = SYSTEM

         RFN = 1, BLK = 12317, RDBA = 4206621

         OBJN = 37, OBJD = 37, OBJECT = I_OBJ2, SUBOBJECT =

         SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc  (incident=800710):

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

Incident details in: /u01/app/oracle/diag/rdbms/xff/xff2/incident/incdir_800710/xff2_ora_50880_i800710.trc

2026-05-20T05:46:12.371040-04:00

ALTER SYSTEM SET remote_listener=' xff-scan:11521' SCOPE=MEMORY SID='xff2';

2026-05-20T05:46:12.372924-04:00

ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='xff2';

2026-05-20T17:46:13.814346+08:00

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814407+08:00

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814442+08:00

Error 604 happened during db open, shutting down database

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc  (incident=800711):

ORA-00603: ORACLE server session terminated by fatal error

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

 

Corrupt block relative dba: 0x00403097 (file 1, block 12439)

Completely zero block found during buffer read

 

Reread (file 1, block 12439) found same corrupt data (no logical check)

2026-05-20T17:46:14.259044+08:00

Corrupt Block Found

         TIME STAMP (GMT) = 05/20/2026 17:46:13

         CONT = 0, TSN = 0, TSNAME = SYSTEM

         RFN = 1, BLK = 12439, RDBA = 4206743

         OBJN = 18, OBJD = 18, OBJECT = OBJ$, SUBOBJECT =

         SEGMENT OWNER = SYS, SEGMENT TYPE = Table Segment

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_gen0_48604.trc  (incident=798852):

ORA-01578: ORACLE data block corrupted (file # 1, block # 12439)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

 

2026-05-20T17:46:13.814346+08:00

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814407+08:00

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814442+08:00

Error 604 happened during db open, shutting down database

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc  (incident=800711):

ORA-00603: ORACLE server session terminated by fatal error

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file # 1, block # 12317)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

 

Corrupt block relative dba: 0x00403097 (file 1, block 12439)

Completely zero block found during buffer read

 

Reread (file 1, block 12439) found same corrupt data (no logical check)

2026-05-20T17:46:14.259044+08:00

Corrupt Block Found

         TIME STAMP (GMT) = 05/20/2026 17:46:13

         CONT = 0, TSN = 0, TSNAME = SYSTEM

         RFN = 1, BLK = 12439, RDBA = 4206743

         OBJN = 18, OBJD = 18, OBJECT = OBJ$, SUBOBJECT =

         SEGMENT OWNER = SYS, SEGMENT TYPE = Table Segment

Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_gen0_48604.trc  (incident=798852):

ORA-01578: ORACLE data block corrupted (file # 1, block # 12439)

ORA-01110: data file 1: '+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T05:46:14.436597-04:00

ALTER SYSTEM SET remote_listener=' xff-scan:11521' SCOPE=MEMORY SID='xff2';

2026-05-20T05:46:14.438492-04:00

ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='xff2';

2026-05-20T17:46:15.486758+08:00

opiodr aborting process unknown ospid (50880) as a result of ORA-603

2026-05-20T17:46:15.498707+08:00

ORA-603 : opitsk aborting process

License high water mark = 423

USER(prelim) (ospid: 50880): terminating the instance due to ORA error 604

2026-05-20T17:46:15.536740+08:00

opiodr aborting process unknown ospid (69547) as a result of ORA-1092

2026-05-20T05:46:16.321597-04:00

ORA-1092 : opitsk aborting process

这里基本上可以看出来是由于在数据库启动过程中递归调用一些sql,但是由于遭遇到坏块导致启动失败,通过dbv检查system数据文件发现256个坏块

256个连续的全0坏块,怀疑是2M的数据被dd全空覆盖,这样的情况,也就是怀疑是au=2的后面2M被覆盖(ausize为4M),分析system的数据分布情况

这里可以确认system的第24个au(从0开始)在14号盘au 2 上面,也就是数据块起始损坏为block：12288-12543（24M*4/8K[有block 0 需要考虑])，对于这种彻底损坏而且比较靠前的system中block,通过人工构造出来这些block的方式进行修复,在自研的Oracle Recovery Tools和obet工具都有该功能.运气不错,通过这个修复之后,直接expdp导出数据没有大问题,比较完美的恢复了这个故障.

 

• 又一例TRIM导致asm磁盘数据丢失的故障
• 删除asmlib磁盘导致磁盘组故障恢复
• 误删除asm disk导致磁盘组无法mount数据库恢复
• kfed修复ORA-15196
• ORA-15335: ASM metadata corruption detected in disk group ‘DATA’
• ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh]故障处理
• ORA-15335 ORA-15130 ORA-15066 ORA-15196
• ORA-600 kfrValAcd30 恢复
• ASM中磁盘组权限设置
• asm disk被加入到另外一个磁盘组故障恢复
• ORA-15096: lost disk write detected
• Exadata磁盘损坏导致磁盘组无法mount恢复（oracle一体机磁盘组异常恢复）