Apply Bug10010310 On Oracle RAC 10.2.0.5

9月24日数据库上频繁出现例如以下错误
Errors in file /u04/admin/njord/udump/njord_ora_25895.trc:
ORA-27300: OS system dependent operation:invalid_process_id failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgpalive1
 
该错误将影响client的连接,经过查阅资料证明是Oracle 10.2.0.5的bug,下面是mos上面的说明

 

ORA-27300 ORA-27301 ORA-27302: failure occurred at: skgpalive1 (文档 ID 1252265.1) 转究竟部  

In this Document

Symptoms
  Changes
  Cause
  Solution
  References

 

APPLIES TO:

Oracle Database – Enterprise Edition – Version 10.2.0.5 and later
Information in this document applies to any platform.

SYMPTOMS

Database alert_<ORACLE_SID>.log shows following error and keeps running:

Errors in file /u04/admin/njord/udump/njord_ora_25895.trc:
ORA-27300: OS system dependent operation:invalid_process_id failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgpalive1
 

CHANGES

Applied 10.2.0.5 patch set.

CAUSE

This is caused by bug 10010310

SOLUTION

bug 10010310 is fixed in 11.2.0.3, 12.1 and above. Please check patch 10010310 for existing interim patches.

REFERENCES

BUG:10207340 - ORA-27302 SKGPALIVE1 AFTER UPGRADE TO 10.2.0.5.0
BUG:10010310 - ORA-27300: INVALID_PROCESS_ID FAILED WITH STATUS: 0; ORA-27302: SKGPALIVE1
BUG:10154137 - ORA-27300: OS SYSTEM DEPENDENT OPERATION:INVALID_PROCESS_ID FAILED WITH STATUS 0

 

 
生产环境:
OS:IBM AIX power 7
Oracle Version:10.2.0.5
Storage:Raw device
Opatch Version:10.2.0.4.9
 
測试环境:
OS:redhat 5.8 x86-64
Oracle Version:10.2.0.5
Storage:Raw device
Opatch Version:10.2.0.4.9
 
操作步骤:
1、下载补丁、上传补丁、解压补丁
2、配置环境变量
3、关闭oracle数据库
4、停止crs
5、在一节点应用补丁
6、在二节点应用补丁
7、start crs
8、开启数据库
 
 
在My Support Oracle 上下载Bug 10010310
[oracle@rac1 ~]$ unzip p10010310_10205_Linux-x86-64.zip
 
配置opatch环境变量:
export OPatch=/u01/app/oracle/product/10.2.0/db_1/OPatch
export PATH=$HOME/bin:$ORACLE_HOME/bin:$OPatch:/sbin:$PATH
 
 
opatch 版本号
[oracle@rac1 10010310]$ opatch version
Invoking OPatch 10.2.0.4.9
 
OPatch Version: 10.2.0.4.9
 
OPatch succeeded.
 
 
停止数据库:
srvctl stop instance -d zhdb -i "zhdb1,zhdb2" -o immediate
 
停止 crs:
crsctl stop crs
 
 
 
一节点:
[oracle@rac1 10010310]$ opatch apply
Invoking OPatch 10.2.0.4.9
 
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /u01/app/oracle/product/10.2.0/db_1
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.9
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db_1/oui
Log file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2014-09-29_05-05-15AM.log
 
Patch history file: /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
 
ApplySession applying interim patch '10010310' to OH '/u01/app/oracle/product/10.2.0/db_1'
 
Running prerequisite checks…
 
OPatch detected the node list and the local node from the inventory.  OPatch will patch the local system then propagate the patch to the remote nodes.
 
 
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'rac2' 
Local node: 'rac1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
 
 
Is the local system ready for patching?

[y|n]

y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '10010310' for restore. This might take a while…
Backing up files affected by the patch '10010310' for rollback. This might take a while…
 
Patching component oracle.rdbms, 10.2.0.5.0…
Updating archive file "/u01/app/oracle/product/10.2.0/db_1/lib/libserver10.a"  with "lib/libserver10.a/ksu.o"
Running make for target ioracle
ApplySession adding interim patch '10010310' to inventory
 
Verifying the update…
Inventory check OK: Patch ID 10010310 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 10010310 are present in Oracle Home.
 
The local system has been patched.  You can restart Oracle instances on it.
 
 
Patching in rolling mode.
 
 
The node 'rac2' will be patched next.
 
 
Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac2'.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
 
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'rac2' 
   Apply-related files are:
     FP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_files.txt"
     DP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_dirs.txt"
     MP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/make_cmds.txt"
     RC = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/remote_cmds.txt"
 
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_files.txt" with actual path.
Propagating files to remote nodes…
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes…
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/make_cmds.txt" with actual path.
Running command on remote node 'rac2': 
cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1 || echo REMOTE_MAKE_FAILED::>&2 
 
 
The node 'rac2' has been patched.  You can restart Oracle instances on it.
 
There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'rac2' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
'
 
 
OPatch succeeded.
[oracle@rac1 10010310]$ opatch lsinventory
Invoking OPatch 10.2.0.4.9
 
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /u01/app/oracle/product/10.2.0/db_1
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.9
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db_1/oui
Log file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2014-09-29_05-08-40AM.log
 
Patch history file: /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2014-09-29_05-08-40AM.txt
 
——————————————————————————–
Installed Top-level Products (2): 
 
Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 4                            10.2.0.5.0
There are 2 products installed in this Oracle Home.
 
 
Interim patches (1) :
 
Patch  10010310     : applied on Mon Sep 29 05:06:10 EDT 2014
Unique Patch ID:  13106370
   Created on 3 Nov 2010, 02:11:57 hrs PST8PDT
   Bugs fixed:
     10010310
 
 
 
Rac system comprising of multiple nodes
  Local node = rac1
  Remote node = rac2
 
——————————————————————————–
 
OPatch succeeded.
[oracle@rac1 10010310]$ 
 
 
 
节点二:
[oracle@rac2 10010310]$ opatch  apply
Invoking OPatch 10.2.0.4.9
 
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /u01/app/oracle/product/10.2.0/db_1
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.9
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db_1/oui
Log file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2014-09-29_05-10-16AM.log
 
Patch history file: /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
 
ApplySession applying interim patch '10010310' to OH '/u01/app/oracle/product/10.2.0/db_1'
Interim patch 10010310 is a superset of the patch(es) [  10010310 ] in the Oracle Home
OPatch will rollback the subset patches and apply the given patch.
 
Running prerequisite checks…
 
OPatch detected the node list and the local node from the inventory.  OPatch will patch the local system then propagate the patch to the remote nodes.
 
 
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'rac1' 
Local node: 'rac2'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
 
 
Is the local system ready for patching?

[y|n]

y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '10010310' for restore. This might take a while…
Backing up files affected by the patch '10010310' for restore. This might take a while…
ApplySession rolling back interim patch '10010310' from OH '/u01/app/oracle/product/10.2.0/db_1'
 
Patching component oracle.rdbms, 10.2.0.5.0…
Updating archive file "/u01/app/oracle/product/10.2.0/db_1/lib/libserver10.a"  with "lib/libserver10.a/ksu.o"
RollbackSession removing interim patch '10010310' from inventory
 
 
OPatch back to application of the patch '10010310' after auto-rollback.
 
Backing up files affected by the patch '10010310' for rollback. This might take a while…
 
Patching component oracle.rdbms, 10.2.0.5.0…
Updating archive file "/u01/app/oracle/product/10.2.0/db_1/lib/libserver10.a"  with "lib/libserver10.a/ksu.o"
Running make for target ioracle
ApplySession adding interim patch '10010310' to inventory
 
Verifying the update…
Inventory check OK: Patch ID 10010310 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 10010310 are present in Oracle Home.
 
The local system has been patched.  You can restart Oracle instances on it.
 
 
Patching in rolling mode.
 
 
The node 'rac1' will be patched next.
 
 
Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac1'.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
 
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'rac1' 
   Apply-related files are:
     FP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_files.txt"
     DP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_dirs.txt"
     MP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/make_cmds.txt"
     RC = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/remote_cmds.txt"
 
   Auto-Rollback-related files are:
     FR = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/remove_files.txt"
     DR = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/remove_dirs.txt"
 
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/remove_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/remove_dirs.txt" with actual path.
Removing directories on remote nodes…
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_files.txt" with actual path.
Propagating files to remote nodes…
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes…
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/10010310_Nov_3_2010_02_11_57/rac/make_cmds.txt" with actual path.
Running command on remote node 'rac1': 
cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1 || echo REMOTE_MAKE_FAILED::>&2 
 
 
The node 'rac1' has been patched.  You can restart Oracle instances on it.
 
There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'rac1' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
'
 
——————————————————————————–
The following warnings have occurred during OPatch execution:
1) OUI-67620:Interim patch 10010310 is a superset of the patch(es) [  10010310 ] in the Oracle Home
——————————————————————————–
OPatch Session completed with warnings.
 
OPatch completed with warnings.
[oracle@rac2 10010310]$ 
[oracle@rac2 10010310]$ 
[oracle@rac2 10010310]$ opatch lsinventory
Invoking OPatch 10.2.0.4.9
 
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /u01/app/oracle/product/10.2.0/db_1
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.9
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db_1/oui
Log file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2014-09-29_05-13-18AM.log
 
Patch history file: /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
 
Lsinventory Output file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2014-09-29_05-13-18AM.txt
 
——————————————————————————–
Installed Top-level Products (2): 
 
Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 4                            10.2.0.5.0
There are 2 products installed in this Oracle Home.
 
 
Interim patches (1) :
 
Patch  10010310     : applied on Mon Sep 29 05:11:58 EDT 2014
Unique Patch ID:  13106370
   Created on 3 Nov 2010, 02:11:57 hrs PST8PDT
   Bugs fixed:
     10010310
 
 
 
Rac system comprising of multiple nodes
  Local node = rac2
  Remote node = rac1
 
——————————————————————————–
 
OPatch succeeded.
[oracle@rac2 10010310]$ 
 
 
 
[root@rac1 oracle]# crsctl start crs
Attempting to start CRS stack 
The CRS stack will be started shortly
[root@rac1 oracle]# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
 
[root@rac1 oracle]# crs_stat -t
Name           Type           Target    State     Host        
————————————————————
ora….C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    ONLINE    ONLINE    rac1        
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   application    ONLINE    ONLINE    rac1        
ora….C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    ONLINE    ONLINE    rac2        
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   application    ONLINE    ONLINE    rac2        
ora.zhdb.db    application    ONLINE    ONLINE    rac1        
ora….b1.inst application    ONLINE    ONLINE    rac1        
ora….b2.inst application    OFFLINE   OFFLINE               
 
 
 
測试环境须要宕机时间大概为10分钟。生产环境保守预计30分钟

posted @ 2017-04-20 16:37  llguanli  阅读(318)  评论(0编辑  收藏  举报