磁盘添加异常--属组属主
操作系统：hp unix
数据库版本：11.2.0.4
背景：客户再asm磁盘组空间不足，添加磁盘组过程中因磁盘组属性问题导致加入磁盘组磁盘状态不对
处理排查过程
1.查看磁盘组状态
#这里可以看到磁盘组的mount_status:CLOSED,但是磁盘头信息已经是member，HEADER_STATUS：MEMBER
SQL> select GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,STATE,NAME from v$asm_disk; GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 0 1 CLOSED MEMBER NORMAL /dev/rdisk/fabdat041 2 40 CACHED MEMBER NORMAL FABDAT_0040 /dev/rdisk/fabdat039 2 41 CACHED MEMBER NORMAL FABDAT_0041 /dev/rdisk/fabdat040 3 18 CACHED MEMBER NORMAL FABIDX_0018 /dev/rdisk/fabidx01 2 37 CACHED MEMBER NORMAL FABDAT_0037 /dev/rdisk/fabdat01 2 38 CACHED MEMBER NORMAL FABDAT_0038 /dev/rdisk/fabdat037 2 39 CACHED MEMBER NORMAL FABDAT_0039 /dev/rdisk/fabdat038 4 0 CACHED MEMBER NORMAL OCRVOTE_0000 /dev/rdisk/disk106 4 2 CACHED MEMBER NORMAL OCRVOTE_0002 /dev/rdisk/disk107 4 1 CACHED MEMBER NORMAL OCRVOTE_0001 /dev/rdisk/disk108 1 0 CACHED MEMBER NORMAL ARC_0000 /dev/rdisk/disk109 GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 1 1 CACHED MEMBER NORMAL ARC_0001 /dev/rdisk/disk110 1 2 CACHED MEMBER NORMAL ARC_0002 /dev/rdisk/disk111 1 3 CACHED MEMBER NORMAL ARC_0003 /dev/rdisk/disk112 3 0 CACHED MEMBER NORMAL FABIDX_0000 /dev/rdisk/disk113 3 1 CACHED MEMBER NORMAL FABIDX_0001 /dev/rdisk/disk114 3 2 CACHED MEMBER NORMAL FABIDX_0002 /dev/rdisk/disk115 3 3 CACHED MEMBER NORMAL FABIDX_0003 /dev/rdisk/disk116 3 4 CACHED MEMBER NORMAL FABIDX_0004 /dev/rdisk/disk117 3 5 CACHED MEMBER NORMAL FABIDX_0005 /dev/rdisk/disk118 2 0 CACHED MEMBER NORMAL FABDAT_0000 /dev/rdisk/disk247 2 1 CACHED MEMBER NORMAL FABDAT_0001 /dev/rdisk/disk248 GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 2 2 CACHED MEMBER NORMAL FABDAT_0002 /dev/rdisk/disk249 2 3 CACHED MEMBER NORMAL FABDAT_0003 /dev/rdisk/disk250 2 4 CACHED MEMBER NORMAL FABDAT_0004 /dev/rdisk/disk251 2 5 CACHED MEMBER NORMAL FABDAT_0005 /dev/rdisk/disk252 2 6 CACHED MEMBER NORMAL FABDAT_0006 /dev/rdisk/disk253 2 7 CACHED MEMBER NORMAL FABDAT_0007 /dev/rdisk/disk254 2 8 CACHED MEMBER NORMAL FABDAT_0008 /dev/rdisk/disk255 2 10 CACHED MEMBER NORMAL FABDAT_0010 /dev/rdisk/disk257 2 9 CACHED MEMBER NORMAL FABDAT_0009 /dev/rdisk/disk256 2 11 CACHED MEMBER NORMAL FABDAT_0011 /dev/rdisk/disk258 2 12 CACHED MEMBER NORMAL FABDAT_0012 /dev/rdisk/disk259 GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 2 13 CACHED MEMBER NORMAL FABDAT_0013 /dev/rdisk/disk260 2 14 CACHED MEMBER NORMAL FABDAT_0014 /dev/rdisk/disk261 2 15 CACHED MEMBER NORMAL FABDAT_0015 /dev/rdisk/disk262 2 16 CACHED MEMBER NORMAL FABDAT_0016 /dev/rdisk/disk701 2 17 CACHED MEMBER NORMAL FABDAT_0017 /dev/rdisk/disk702 2 18 CACHED MEMBER NORMAL FABDAT_0018 /dev/rdisk/disk703 2 19 CACHED MEMBER NORMAL FABDAT_0019 /dev/rdisk/disk704 2 20 CACHED MEMBER NORMAL FABDAT_0020 /dev/rdisk/disk705 2 21 CACHED MEMBER NORMAL FABDAT_0021 /dev/rdisk/disk706 2 22 CACHED MEMBER NORMAL FABDAT_0022 /dev/rdisk/disk707 2 23 CACHED MEMBER NORMAL FABDAT_0023 /dev/rdisk/disk708 GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 2 24 CACHED MEMBER NORMAL FABDAT_0024 /dev/rdisk/disk709 2 25 CACHED MEMBER NORMAL FABDAT_0025 /dev/rdisk/disk710 2 26 CACHED MEMBER NORMAL FABDAT_0026 /dev/rdisk/disk711 3 6 CACHED MEMBER NORMAL FABIDX_0006 /dev/rdisk/disk712 3 7 CACHED MEMBER NORMAL FABIDX_0007 /dev/rdisk/disk713 3 8 CACHED MEMBER NORMAL FABIDX_0008 /dev/rdisk/disk714 3 9 CACHED MEMBER NORMAL FABIDX_0009 /dev/rdisk/disk715 2 27 CACHED MEMBER NORMAL FABDAT_0027 /dev/rdisk/disk716 2 28 CACHED MEMBER NORMAL FABDAT_0028 /dev/rdisk/disk717 2 29 CACHED MEMBER NORMAL FABDAT_0029 /dev/rdisk/disk718 2 30 CACHED MEMBER NORMAL FABDAT_0030 /dev/rdisk/disk719 GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 2 31 CACHED MEMBER NORMAL FABDAT_0031 /dev/rdisk/disk720 2 32 CACHED MEMBER NORMAL FABDAT_0032 /dev/rdisk/disk721 2 33 CACHED MEMBER NORMAL FABDAT_0033 /dev/rdisk/disk722 3 10 CACHED MEMBER NORMAL FABIDX_0010 /dev/rdisk/disk723 3 11 CACHED MEMBER NORMAL FABIDX_0011 /dev/rdisk/disk724 3 12 CACHED MEMBER NORMAL FABIDX_0012 /dev/rdisk/disk725 3 13 CACHED MEMBER NORMAL FABIDX_0013 /dev/rdisk/disk726 3 15 CACHED MEMBER NORMAL FABIDX_0015 /dev/rdisk/disk728 3 14 CACHED MEMBER NORMAL FABIDX_0014 /dev/rdisk/disk727 3 17 CACHED MEMBER NORMAL FABIDX_0017 /dev/rdisk/disk730 3 16 CACHED MEMBER NORMAL FABIDX_0016 /dev/rdisk/disk729 GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE NAME PATH ------------ ----------- ------- ------------ -------- ------------------------------ ------------------------------------------------------------ 2 34 CACHED MEMBER NORMAL FABDAT_0034 /dev/rdisk/disk771 2 35 CACHED MEMBER NORMAL FABDAT_0035 /dev/rdisk/disk772 2 36 CACHED MEMBER NORMAL FABDAT_0036 /dev/rdisk/disk773 1 4 CACHED MEMBER NORMAL ARC_0004 /dev/rdisk/disk781 1 5 CACHED MEMBER NORMAL ARC_0005 /dev/rdisk/disk782
2.尝试重新添加磁盘组，如下报错
SQL> alter diskgroup FABDAT add disk '/dev/rdisk/fabdat041';
alter diskgroup FABDAT add disk '/dev/rdisk/fabdat041'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/rdisk/fabdat041' belongs to diskgroup "FABDAT"
SQL> alter diskgroup FABDAT drop disk '/dev/rdisk/fabdat041';
alter diskgroup FABDAT drop disk '/dev/rdisk/fabdat041'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15054: disk "/DEV/RDISK/FABDAT041" does not exist in diskgroup "FABDAT"
3.查看磁盘头状态
fabdb1[/home/grid]$kfed read /dev/rdisk/fabdat041 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483690 ; 0x008: disk=42 kfbh.check: 1009641813 ; 0x00c: 0x3c2de955 kfbh.fcn.base: 21938159 ; 0x010: 0x014ebfef kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 186646528 ; 0x020: 0x0b200000 kfdhdb.dsknum: 42 ; 0x024: 0x002a kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: FABDAT_0042 ; 0x028: length=11 ---->>这里已经可以看到磁盘组已经被命名，但是在v$asm_disk信息中，未能发现 kfdhdb.grpname: FABDAT ; 0x048: length=6 kfdhdb.fgname: FABDAT_0042 ; 0x068: length=11 kfdhdb.capname: ; 0x088: length=0 kfdhdb.crestmp.hi: 33152145 ; 0x0a8: HOUR=0x11 DAYS=0x4 MNTH=0x7 YEAR=0x7e7 kfdhdb.crestmp.lo: 1210455040 ; 0x0ac: USEC=0x0 MSEC=0x185 SECS=0x2 MINS=0x12 kfdhdb.mntstmp.hi: 33152145 ; 0x0b0: HOUR=0x11 DAYS=0x4 MNTH=0x7 YEAR=0x7e7 kfdhdb.mntstmp.lo: 1210459136 ; 0x0b4: USEC=0x0 MSEC=0x189 SECS=0x2 MINS=0x12 kfdhdb.secsize: 1024 ; 0x0b8: 0x0400 kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000 kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80 kfdhdb.dsksize: 204800 ; 0x0c4: 0x00032000 kfdhdb.pmcnt: 3 ; 0x0c8: 0x00000003 kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001 kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000 kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000 kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000 kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000 kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000 kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000 kfdhdb.grpstmp.hi: 33009745 ; 0x0e4: HOUR=0x11 DAYS=0x2 MNTH=0xc YEAR=0x7de kfdhdb.grpstmp.lo: 3037726720 ; 0x0e8: USEC=0x0 MSEC=0x2 SECS=0x11 MINS=0x2d kfdhdb.vfstart: 0 ; 0x0ec: 0x00000000 kfdhdb.vfend: 0 ; 0x0f0: 0x00000000 kfdhdb.spfile: 0 ; 0x0f4: 0x00000000 kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000 kfdhdb.ub4spare[0]: 0 ; 0x0fc: 0x00000000 kfdhdb.ub4spare[1]: 0 ; 0x100: 0x00000000 kfdhdb.ub4spare[2]: 0 ; 0x104: 0x00000000 kfdhdb.ub4spare[3]: 0 ; 0x108: 0x00000000 kfdhdb.ub4spare[4]: 0 ; 0x10c: 0x00000000 kfdhdb.ub4spare[5]: 0 ; 0x110: 0x00000000 kfdhdb.ub4spare[6]: 0 ; 0x114: 0x00000000 kfdhdb.ub4spare[7]: 0 ; 0x118: 0x00000000 kfdhdb.ub4spare[8]: 0 ; 0x11c: 0x00000000 kfdhdb.ub4spare[9]: 0 ; 0x120: 0x00000000 kfdhdb.ub4spare[10]: 0 ; 0x124: 0x00000000 kfdhdb.ub4spare[11]: 0 ; 0x128: 0x00000000 kfdhdb.ub4spare[12]: 0 ; 0x12c: 0x00000000 kfdhdb.ub4spare[13]: 0 ; 0x130: 0x00000000 kfdhdb.ub4spare[14]: 0 ; 0x134: 0x00000000 kfdhdb.ub4spare[15]: 0 ; 0x138: 0x00000000 kfdhdb.ub4spare[16]: 0 ; 0x13c: 0x00000000 kfdhdb.ub4spare[17]: 0 ; 0x140: 0x00000000 kfdhdb.ub4spare[18]: 0 ; 0x144: 0x00000000 kfdhdb.ub4spare[19]: 0 ; 0x148: 0x00000000 kfdhdb.ub4spare[20]: 0 ; 0x14c: 0x00000000 kfdhdb.ub4spare[21]: 0 ; 0x150: 0x00000000 kfdhdb.ub4spare[22]: 0 ; 0x154: 0x00000000 kfdhdb.ub4spare[23]: 0 ; 0x158: 0x00000000 kfdhdb.ub4spare[24]: 0 ; 0x15c: 0x00000000 kfdhdb.ub4spare[25]: 0 ; 0x160: 0x00000000 kfdhdb.ub4spare[26]: 0 ; 0x164: 0x00000000 kfdhdb.ub4spare[27]: 0 ; 0x168: 0x00000000 kfdhdb.ub4spare[28]: 0 ; 0x16c: 0x00000000 kfdhdb.ub4spare[29]: 0 ; 0x170: 0x00000000 kfdhdb.ub4spare[30]: 0 ; 0x174: 0x00000000 kfdhdb.ub4spare[31]: 0 ; 0x178: 0x00000000 kfdhdb.ub4spare[32]: 0 ; 0x17c: 0x00000000 kfdhdb.ub4spare[33]: 0 ; 0x180: 0x00000000 kfdhdb.ub4spare[34]: 0 ; 0x184: 0x00000000 kfdhdb.ub4spare[35]: 0 ; 0x188: 0x00000000 kfdhdb.ub4spare[36]: 0 ; 0x18c: 0x00000000 kfdhdb.ub4spare[37]: 0 ; 0x190: 0x00000000 kfdhdb.ub4spare[38]: 0 ; 0x194: 0x00000000 kfdhdb.ub4spare[39]: 0 ; 0x198: 0x00000000 kfdhdb.ub4spare[40]: 0 ; 0x19c: 0x00000000 kfdhdb.ub4spare[41]: 0 ; 0x1a0: 0x00000000 kfdhdb.ub4spare[42]: 0 ; 0x1a4: 0x00000000 kfdhdb.ub4spare[43]: 0 ; 0x1a8: 0x00000000 kfdhdb.ub4spare[44]: 0 ; 0x1ac: 0x00000000 kfdhdb.ub4spare[45]: 0 ; 0x1b0: 0x00000000 kfdhdb.ub4spare[46]: 0 ; 0x1b4: 0x00000000 kfdhdb.ub4spare[47]: 0 ; 0x1b8: 0x00000000 kfdhdb.ub4spare[48]: 0 ; 0x1bc: 0x00000000 kfdhdb.ub4spare[49]: 0 ; 0x1c0: 0x00000000 kfdhdb.ub4spare[50]: 0 ; 0x1c4: 0x00000000 kfdhdb.ub4spare[51]: 0 ; 0x1c8: 0x00000000 kfdhdb.ub4spare[52]: 0 ; 0x1cc: 0x00000000 kfdhdb.ub4spare[53]: 0 ; 0x1d0: 0x00000000 kfdhdb.acdb.aba.seq: 0 ; 0x1d4: 0x00000000 kfdhdb.acdb.aba.blk: 0 ; 0x1d8: 0x00000000 kfdhdb.acdb.ents: 0 ; 0x1dc: 0x0000 kfdhdb.acdb.ub2spare: 0 ; 0x1de: 0x0000
4.根据mos提示，重新添加磁盘，依然报错
SQL> alter diskgroup FABDAT add disk '/dev/rdisk/fabdat041' name FABDAT_0042;
alter diskgroup FABDAT add disk '/dev/rdisk/fabdat041' name FABDAT_0042
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/rdisk/fabdat041' belongs to diskgroup "FABDAT"
5.再次查阅相关mos，对磁盘头进行dd操作
fabdb1#[/dev/rdisk]dd if=/dev/zero of=/dev/rdisk/fabdat041 bs=1024k count=100 100+0 records in 100+0 records out fabdb3#[/dev/rdisk]dd if=/dev/zero of=/dev/rdisk/fabdat041 bs=1024k count=100 100+0 records in 100+0 records out
6.重新添加磁盘，正常添加进磁盘组，磁盘rebalance
fabdb2[/dev/rdisk]$sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 5 10:29:25 2023 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter diskgroup FABDAT add disk '/dev/rdisk/fabdat041'; Diskgroup altered. SQL> SQL> select * from v$asm_operation; GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE ------------ ----- ---- ---------- ---------- ---------- ---------- ---------- EST_MINUTES ERROR_CODE ----------- -------------------------------------------- 2 REBAL WAIT 1 1 0 0 0 0
7.经验教训
#在添加磁盘组的时候，一定要注意磁盘组属性，避免不必要的麻烦
可能会出现的问题：
（1）添加属性不对的磁盘组，数据库状态可能会异常（生产库添加redo磁盘组，导致数据库宕机--重新赋权后解决）
（2）添加属性不对的磁盘组，磁盘属性异常，并无法加入到磁盘组中（如上案例）
