@<Note to Author: DO NOT DELETE the following Disclaimer>
@<Note to Publisher: DELETE the following Disclaimer when you set article to PUBLISHED>
***
This article is being delivered in Draft form and may contain
errors. Please use the MetaLink "Feedback" button to advise
Oracle of any issues related to this article.
***
1) What is OPATCH and where can I download it from ?
2) Requirements for OPatch
3) How to apply a patch to an installation and confirm if this was applied correctly ?
4) How does the OPatch versioning system work
5) I get an "unknown error" when trying to apply a patch
6) My inventory is in a different location as compared to the normal location and my oraInst.loc is present in $ORACLE_HOME. How do I install the patch under such cases?
7) Unable to get inventory lock exception encountered when applying a patch using OPatch how should I proceed here ?
8) If OPatch aborts during application then how does it proceed
9) Applied a patch using the invPtrLoc option .Since rollback does not have this option?
10) Where is the storage location for OPatch where the current patch history can been located ?
11) Where are the apply logs located and how to check if a patch has been applied successfully ?
12) When running OPatch encounter an error saying fuser is not available why should be done?
13) A patch is bundled as a client and server patch and applying this patch on the client in the absence of the server fails ?
14) When applying patches , patch is not for this operating system ?
15) When the patches are applied in RAC can this be applied in a rolling fashion, if yes then how can this be applied ?
16) How are OPatch versions obsoleted and how should the OPatch executables be stored
17) Internals of OPatch - perl - java and what else ?
18) Which JDK versions is used with OPatch and how does it normally function without having to install anything ?
19) What are the options available for OPatch during the process of "apply"
20) Options available for OPatch when rolling back the patches
21) What is attachhome and when should it be used ?
22) How do I list the inventory contents -
23) OPatch directory structure -
24) When applying patches there is a disparity between the size of the oracle executables on all the boxes
25) Application of a patch on a node hangs when propagating the inventory in the post inventory tasks
26) Java classes which are used in OPatch and which classes perform which functions:
27) OPatch encounters java.lang.OutOfMemoryException when trying to apply patches , possibly comps.xml ?
28) Different kinds of inventories and when are they synchronized - central inventory , local inventory and the important files which are accessed ? Where is this oracle home information picked up from ?
29) Installing OUI/OSP from the 2.2.0.18.0 patchset , should I install the OSP or only the OUI
30) Found a file called patch_free in $ORACLE_HOME/.opatch_storage is there anything which needs to be done. Also what about patch_locked ?
31) When applying a patch on the cluster faced the "PRKC-1021 : Problem in the clusterware" or a problem in the clusterware during the "getLocalNode call.
32) What is the OPATCH_DEBUG=TRUE flag and when should this be set. Also
what information should be provided when filing a bug for an OPatch issue
33) Can patches be rolled back using the rolling RAC or minimize_downtime options ?
34) Received Exception in "thread "main" java.lang.UnsatisfiedLinkError: and no oraInstaller in java.library.path" when applying patches, what next ?
35) When applying a patch what does it mean when I get the message "Not a valid patch area." ?
36) Got exception " Exception in thread "main" java.lang.NullPointerException at GetSingleInventory.main(GetSingleInventory.java:124" when trying to run the lsinventory command?
37) Which Oracle homes are registered with my central inventory ?
38) How can I to minimize the downtime when applying a patch to my RAC?
39) Why does the patch installation abort with the message that the "local node could not be determined."?
40) Questions on rolling RAC -
41) What happens if the rolling RAC patch application process is interrupted
This document is currently being edited and is still not a final version.
1) What is OPATCH and where can I download it from ?
Opatch is a set of perl scripts and java classes which allow the application and rolling back of interim (one-off) patches currently for the Oracle RDBMS product. The program requires that the perl language interpreter version 5.005_03 or greater to run but version 5.6 or greater is recommended. The main shell of OPatch is called from opatch, which calls other Perl modules and subsequently java classes as well. Under a normal Oracle home installation the JRE is used from the Oracle home and the Perl installation as well which is installed with the Apache software in $ORACLE_HOME/Apache/perl .
The opatch tool for all products can be downloaded from Metalink placeholder bug (Patch 6880880). Note: OPatch is platform specific, so you must select the appropiate Platform. Also, there are 3 versions. Select the appropiate version:
-
For all releases of 11.1, select the OPatch version 11.1.0.0.0
-
For all releases of 10.2, select the OPatch version 10.2.0.0.0
-
For all releases prior to 10.2 (9.0.4, 9.2, 10.1), select the OPatch version 10.1.0.0.0
For more detail, please see NOTE 224346.1, "Opatch - Where Can I Find the Latest Version of Opatch?"
2) Requirements for OPatch
The primary dependencies for an OPatch installation are
- Perl 5.6 or greater (is recommended) or the one which comes with the Oracle installation in $ORACLE_HOME/Apache/perl directory works too . If installed using the typical option with the Apache HTTP server software which in-turn installs perl , OPatch runs out of the box and does not require an external perl interpreter to be installed.
- The inventory which is the central inventory for storing product information (pointed to by /var/opt/oraInst.loc) , the local inventory for reading the components installed in this specific home ($ORACLE_HOME/inventory) . In extreme cases its possible to skip the inventory in case of corruptions but its not recommended as the DBA would need to keep track of the patches that were applied manually
- The patch stage area which is the location where the patch has been downloaded and unzipped. Note that OPatch is not bundled along with the patch itself (we used to do this with 9.2.0.1 patches only) and its highly recommended to download the latest OPatch version from placeholder bug (Patch 6880880) as subsequent fixes to the product are merged and uploaded to the latest version under this placeholder. See 1) What is OPATCH and where can I download it from ?
- When patching a RAC installation the paths of $ORACLE_HOME/lib and $ORACLE_HOME/srvm/lib must be in the user's environment for library searches. i.e.: Under Solaris these should be in LD_LIBRARY_PATH so the extra libraries can be read. If not set then required libraries will not be found and patch will not install.
3) How to apply a patch to an installation and confirm if this was applied correctly ?
The README file that is bundled with eaach patch contains exact patch installation instructions. Usually these instructions are to run "opatch apply" as the Oracle account, from the "un-compressed" patch subdirectory.
The opatch tool applies patches to the Oracle home which is specified and set as $ORACLE_HOME. Logs are placed as follows:
For opatch version 1.0.0.0.xx, logs are written into the patch storage area under $ORACLE_HOME/.patch_storage/bug number and the log in this case starts with
"<bugnum>_Apply_<date>.log"
For the newer opatch versions for products installed with OUI release 10.2 and after, logs are written into the cfgtoollogs/opatch area under $ORACLE_HOME/ and the log in this case starts with
"opatch<date>.log"
In each case, the logs contain the complete sequence of steps followed by opatch which is echoed on the screen with the relevant commands being executed to complete the patch application. In case of a patch application failure the logs would be the first place to look.
4) How does the OPatch versioning system work
As explained in NOTE 224346.1, "Opatch - Where Can I Find the Latest Version of Opatch?", different versions of patch are required for products installed with OUI version 10.2 or after, and for products installed with OUI versions before 10.2
For products installed with OUI versions 10.2 or after, OPatch versioning will match the associated OUI version, until the 5th digit. for example 10.2.0.3.1 The last digit is incremented for minor changes or fixes which have been added to the opatch tool. When checking the ARU placeholder for OPatch you will find for example "Fixed In Ver: 10.2.0.3.1" at the end , .
For products installed with OUI versions before 10.2, OPatch versioning is still at version 1 with 5 digits separated by dots. for example 1.0.0.0.57 The last digit is incremented for minor changes or fixes which have been added to the opatch tool. When checking the ARU placeholder for OPatch you will find for example "Fixed In Ver: 1.0.0.0.57" at the end , .
5) I get an "unknown error" when trying to apply a patch
An unknown error is basically an unhandled exception for OPatch. This could be caused in several different areas of the OPatch code and could be primarily treated as a bug or an existing issue if you are not using the latest version
Some of the common solutions observed for correcting "Unknown error" type errors are as follows
- Check if the patch stage area has appropriate permissions namely "chmod 755 <dir where unzipped>" and retry
- Check the APPLY log and see which stage has caused the exception , normally if this is encountered say in the apply phase of the patch application then check if the "opatch lsinventory" command works or fails in the same manner. Usually from past experiences this has been associated with getting an exception from the inventory for which there is no stack trace printed as this is unhandled . Enhancement request has been logged for this - 3020660.
- Check if skipping the inventory brings you to the first stage namely "Ready to apply the patch?" , if yes then the problem lies with interfacing with the inventory - Search Webiv for known issues and test with the latest version of OPatch.
- If there is no additional information in the apply logs please collect the truss output for the opatch command For example on solaris â
- "truss -aefdD -o test.out opatch apply" . Note to follow forks must be included in the options. Also check Note:732697.1
6) My inventory is in a different location as compared to the normal location and my oraInst.loc is present in $ORACLE_HOME. How do I install the patch under such cases?
OPatch expects the oraInst.loc file in the standard location namely /var/opt/oracle/oraInst.loc however there are certain systems where this file is present at other locations. This is overridden using the inventory API flag invPtrLoc (note the case) Typically used for Oracle installations installed with
./runInstaller -invPtrLoc $ORACLE_HOME/oraInst.loc .
For such oracle homes OPatch would error out saying "Unable to access inventory" or
errors with "Couldn't find required file liboraInstallerâ which can be misleading. Under such cases when applying a patch you need to specify the same flag for OPatch application , for example
opatch apply -invPtrLoc $ORACLE_HOME/oraInst.loc .
From a supportability point of view this flag would be available as long as the Installer supports and this would also imply the customer has a non-standard installation hence its not encouraged.
7) Unable to get inventory lock exception encountered when applying a patch using OPatch how should I proceed here ?
This has been difficult to diagnose ,
- Collect the information required to determine the root cause of the problem - see Note:732697.1.
- We are unable to get an inventory lock (which is a lock for the inventory control area for this oracle home) which is required when trying to store this information in the inventory. This is also taken in an attempt to synchronize the binary
-inventory with the non-binary version if they are out of synch and might hint that the customer might have tried to modify the XML files in the inventory manually but its not necessarily the case.
- From previous issues this does not appear to be a problem with a stale lock file in oraInventory/locks as then we would receive an exception suggesting an existing log file and not an unknown exception.
- ![endif]>In certain cases you might get a java function like GetSingleInventory and an exception with the unable to lock inventory problem . In that case check for matches for this issue and check if $ORACLE_HOME has been set correctly and the information in the location of oraInst.loc , ContentsXML/inventory should contain the Oracle Home in concern.
8) If OPatch aborts during application then how does it proceed
Opatch has been written so the commands are idempotent. Running them once is the same as running them many times. This makes them restartable in the case of interruption. Currently a partially installed interim patch cannot be removed. The patch process must finish. Once the patch process has finished the patch can be removed using rollback If this aborts due to a problem say a library file which was not located and one of the object files was not applied then one could restart the patch application after the library file has been made available , only the last remaining object file would be applied. Other exceptions for patch applications being aborted is make failures due to bad patches, no space left on the device when making the new executables with "ld" failures. All these are restartable.
9) Applied a patch using the invPtrLoc option .Since rollback does not have this option?
Yes rollback also works with the invPtrLoc option however again this option is tied to the supportability of the installer for the same option. This also removes the information for the patch from the inventory when the process is complete.
10) Where is the storage location for OPatch where the current patch history can been located ?
Although the opatch tool can be placed anywhere, it is usually located in the $ORACLE_HOME/OPatch directory.
Current patch history is automatically contained in the Oracle Central Inventory, which can be viewed with the command "opatch lsinventory -detail".
11) Where are the apply logs located and how to check if a patch has been applied successfully ?
Opatch logs are placed as follows:
For opatch version 1.0.0.0.xx, logs are written into the patch storage area under $ORACLE_HOME/.patch_storage/bug number and the log in this case starts with
"<bugnum>_Apply_<date>.log"
For the newer opatch versions for products installed with OUI release 10.2 and after, logs are written into the cfgtoollogs/opatch area under $ORACLE_HOME/ and the log in this case starts with
"opatch<date>.log"
In each case, the logs contain the complete sequence of steps followed by OPatch which is echoed on the screen with the relevant commands being executed to complete the patch application. In case of a patch application failure the logs would be the first place to look.
Also the library (if an archive) can be scanned for the object file and if the object file has a new timestamp as compared to the older object files then this has been patched. For example a patch was applied to libserver9.a under kko.o then
cd $ORACLE_HOME/lib
ar tv libserver9.a | grep kko.o
ar tv libserver9.a | grep kke.o
Compare say the timestamps ..
rw-rw-r-- 94110/ 42424 589088 Jun 30 18:49 2003 kko.o
rw-rw-r-- 94110/ 42424 12988 Feb 27 18:43 2003 kkocri.o
rw-rw-r-- 94110/ 42424 4824 Feb 27 18:43 2003 kkop.o
From the above it can be seen that the object file has been plugged in recently. Also the libserver9.a timestamp changes. The patch storage directory is created irrespective of the inventory updates.
12) When running OPatch encounter an error saying fuser is not available why should be done?
When running OPatch it checks if any oracle processes are running. If yes then it warns the user before the patch application begins. This check is done using the "fuser" command and on most operating systems this is located in /sbin or /usr/sbin and this is not present in the PATH variable for the user. The solution to this is to include
/sbin:/usr/sbin to the PATH variable.
13) A patch is bundled as a client and server patch and applying this patch on the client in the absence of the server fails ?
Usually patches are bundled which are appropriate to the client or the server and if client and server then at OPatch's level this cannot be distinguished hence needs to be repackaged. This is more like an ongoing decision at which level this needs to be fixed but OPatch has flags in etc/config/inventory for example :
|
<component internal_name="oracle.cartridges.context" version="9.2.0.3.0" opt_req="R"/>
the component is oracle.cartridges.context for version 9.2.0.3.0 and which is (R)equired else this would not install. If the client and server components are bundled together an the opt_req flag is marked as "R" then both will be installed , for Windows PSE's curiously this is always optional probably due to the concept of mini patchsets with which these are bundled. Normally this can be rebuilt and the PSE can be repackaged after facing such a problem
|
- For the rdbms product it looks for "oracle.rdbms" - enhancement filed - 2727907 which indicates that the patch should be packaged according to the fix.
14) When applying patches , patch is not for this operating system ?
The check is done for the patch in the etc/config/inventory file in the patch directory and this check is compared against the platform where this is being executed. When requesting the PSE the platform ID is used to check this and the methods for fetching this information from the O.S vary depending on the platform. This is consistent with the
operating system ID filed in the bug database. Double check if the patch is for the same platform , for example
Patches for AIX 5L and AIX 4.3.3 are different hence have different ID's ..
Determined from the inventory file bundled in etc/config/inventory with the patch.
<os_platforms>
<platform name="Sun SPARC Solaris" id="453"/>
</os_platforms>
15) When the patches are applied in RAC can this be applied in a rolling fashion, if yes then how can this be applied ?
- There are two kinds of patch applications for RAC .One is if the patch is marked as rolling upgradable namely in the etc/config/inventory - <online_rac_installable>false</online_rac_installable> which is marked as true. This patch could be applied to the boxes in a cascaded fashion and the different instances they could be running with different versions of the code (the patch causes the code diff) and this is specific for certain patches which can be applied and would not affect the SGA structures and heaps. Also this would help us to check a patch on one of the nodes before propagating this to all the other nodes and the patch , if this causes some unexpected behaviour on part of the application can be rolled back from this node. Only a certain set of patches will be marked as rolling upgradable and currently this feature will be provided with subsequent releases of Opatch and patches which support this concept. This feature is currently being tested and would be available in a future release for patches specifically marked as ârollableâ.
- The second kind here is to apply the patches using minimize_downtime , check the section in the FAQ for the minimize_downtime option to apply patches in a cascading fashion for RAC.
16) How are OPatch versions obsoleted and how should the OPatch executables be stored
It is recommended to use the latest versions for OPatch and this should be obtained by the Metalink placeholder bug. Older versions for OPatch are automatically obsoleted however backward compatibility is maintained for the patch. Therefore, a patch released when OPatch version 10.2.0.2.0 was available can also be applied using OPatch version 10.2.0.2.1.
17) Internals of OPatch - perl - java and what else ?
The main scripts for opatch are opatch , opatch.pl , the class files in the jlib directory and the opatch_modules which are again in perl. The code flow usually starts with opatch and then the modules in opatch_modules which might call other module functions and then calls the classes. The source for opatch is accessible through ADE (the OUI labels).
The perl scripts can be examined directly. The functionality of calling the OUI/SRVM API is present in the java classes and this is visible when the functions in the classes are called. This is present in the logs and is a vital element for debugging OPatch issues if it can be narrowed down to a specific OPatch class.
18) Which JDK versions is used with OPatch and how does it normally function without having to install anything ?
OPatch uses the JDK installed with the Oracle version for example: $ORACLE_BASE/jre/1.3.1/bin/java .This is displayed in the following manner when the OPatch installation is commenced.
OPatch Version 1.0.0.0.39 <== Opatch version currently being used
Perl Version 5.00503 <== Perl version which was found
Performing pre-patch installation checks.
Using oraInst.loc to look up oui libs... <== Scanning oraInst.loc
inventory_location = /oracle/app/oracle/oraInventory <== Location of central inventory
liboraInstaller_lib= /oracle/app/oracle/oui/bin/hpunix/liboraInstaller.sl
path_to_java = /oracle/app/oracle/jre/1.3.1/bin/java <== path to JRE
path_to_oI_loc = /var/opt/oracle/oraInst.loc <== Path to oraInst.loc (default) or as
specified for -invPtrLoc
required_jar_file = lib/OraInstaller.jar <== Required for OUI calls
oui_component_loc = /oracle/app/oracle/oui <== gets the liboraInstaller.sl/so from here
Checking if this is a RAC system... <== Checks for RAC system and further checks for the cluster filesystem
19) What are the options available for OPatch during the process of "apply"
The apply command is used to install a interim patch.
Help for the command could be found by using :
OPatch/opatch.pl apply -help
SYNOPSIS:
apply [ -f[orce] ] [ -i[nvPtrLoc] <Path to oraInst.loc> ]
[ -m[inimize_downtime] ] [ -n[o_inventory] ]
[ -oh <ORACLE_HOME> ] [ <patch_location> ]
|
Option |
Details |
|
-force |
If a conflict exist which prevents the patch from being applied, the -force flag can be used to apply the patch. *** This removes the conflicting patches fromthe system. |
|
-invPtrLoc |
Used to locate the oraInst.loc file. Needed when the installation used the unsupported invPtrLoc flag. This should be the path to oraInst.loc file. |
|
-minimize_downtime |
Only applies to RAC instances. User supplies the order of nodes to be patched. The nodes are going to be patched in this list order and it would be the responsibility of the user applying the patch to shutdown the instances before specifying the next node as OPatch would not shutdown the instance. This option is applicable for applying patches to a group of instances and keeping one instance up. After the patch has been applied to all the instances then the last instance is shutdown and all the other instances are brought up , this reduced the downtime to this window. Then the last node can be patched , useful when patching several nodes in a cluster (>2 nodes) |
|
-no_inventory |
Bypass the inventory for reading and updates. This option only works if the inventory is not available. *** This puts the installation into an unsupported state. *** |
|
-oh |
The directory to use instead of the default of $ORACLE_HOME |
|
patch_location |
Where to install the interim patch from.This should be a directory with the same name as the interim patch , default would be the currently directory from where this has been invoked. |
The "-force" argument will remove any conflicting patches before
installing the current patch. This argument is needed when reinstalling the current patch.
20) Options available for OPatch when rolling back the patches
Used to rollback interim patches which have been applied
rollback -id patch id [ -i[nvPtrLoc] <Path to oraInst.loc> ]
[ -oh <ORACLE_HOME> ] -ph patch dir
|
-id |
Use 'lsinventory' option to display all patch id's. Each one-off patch is indicated by it's id. To rollback a patch the id for that patch must be supplied |
|
-invPtrLoc |
Used to locate the oraInst.loc file. Needed when the installation used the unsupported invPtrLoc flag. This should be the path to oraInst.loc file |
|
-oh |
The directory to use instead of the default of $ORACLE_HOME. |
|
-ph |
The directory that is a valid patch area. Rollback will use the command types found in the patch directory to identify what commands are to be used for the current operating system. |
21) What is attachhome and when should it be used ?
When the Oracle rdbms is installed and transferred across to another system without the inventory information which is a non standard installation the central inventory would not be synchronized with the local inventory. Also if this is the part of a RAC setup then the nodelist information which is read from the inventory file in
oraInventory/ContentsXML/inventory file under the
<NODE_LIST>
<NODE NAME="TESTSERVER" />
</NODE_LIST>
is picked up for deciding which nodes are part of the RAC setup would not have this information present. The attachhome command serves to attach an installed database to the central inventory and also can be used to add RAC node names to the inventory since they were not already present.
attach [ -i[nvPtrLoc] <Path to oraInst.loc> ]
[ -na[me] <ORACLE_HOME_NAME> ] [ -no[des] <node1 node2.. > ]
[ -oh <ORACLE_HOME> ]
Attach an installed database to a central inventory.
|
Option |
Details |
|
-invPtrLoc |
Used to locate the oraInst.loc file. Needed when the installation used theinvPtrLoc flag. This should be the path to oraInst.loc file |
|
-name |
The name to use when attaching the product back to the default central inventory |
|
-nodes |
The nodes to register in a RAC environment when attaching a product back to the default central inventory |
|
-oh |
The directory to use instead of the default of $ORACLE_HOME |
22) How do I list the inventory contents -
The lsinventory command lists the contents of the inventory as specified for the current Oracle home and if wanting this for all the installed oracle homes then use the -all option.
lsinventory [ -all ] [ -i[nvPtrLoc] <Path to oraInst.loc> ]
[ -oh <ORACLE_HOME> ]
|
|
|
|
-all |
Report the name and installation directory for each $ORACLE_HOME found. |
|
-invPtrLoc |
Used to locate the oraInst.loc file. Needed when the installation used the unsupported invPtrLoc flag. This should be the path to oraInst.loc file. |
|
-oh |
The directory to use instead of the default of $ORACLE_HOME found. |
Sample Output :
PRODUCT NAME VERSION
============ =======
Advanced Queueing (AQ) API 9.2.0.1.0
Advanced Replication 9.2.0.1.0
.
.
.
XSQL Servlet 9.2.0.1.0
23) OPatch directory structure -
The OPatch directory structure is as follows
|
|
Opatch - contains the opatch.pl , opatch scripts which are the main entry point for opatch |
|
|
Documentation - some useful documentation
|
Jlib - class files used for OPatch |
opatch_modules - perl modules
|
24) When applying patches there is a disparity between the size of the oracle executables on all the boxes
This used to be a problem on older versions but not anymore. In a RAC installation when the patch is applied on the first node then propagated to the second node (the associated libraries and then relinked). The size of the Oracle executables might be different sometimes due to the affect of other bugs - like 2727837 where libknlopt.a does not
propagate in a RAC setup (fixed in 9.2.0.3) which causes oracle executables to be of a different size. One should always check the OPatch logs to ensure that the patch applied successfully on all the nodes .
25) Application of a patch on a node hangs when propagating the inventory in the post inventory tasks
There were a couple of issues which have been observed in very old releases where this was happening but as of release 1.0.0.0.39, these have been fixed. An issue which might of particular interest is bug 2999294 where a potential race conditions in SRVM could hang the propagation. One of the symptoms here could be hanging cpio
processes when trying to transfer inventory/libraries across the node during OPatch post inventory checks.
26) Java classes which are used in OPatch and which classes perform which functions:
Internals and might not be much applicable for customers :
Some of the classes and their functions have been listed below
|
Class |
Description |
|
AttachHome.java |
Performs the attachhome function , checking if homes are clonable converting between the binary and XML OUI information and applying the install configuration to an install home |
|
CheckRACDetails.java |
Return details for a current Oracle home and if the current installation is RAC. Most of the functionality in this class is also performed in InitialDatabaseCheck. Of particular interest is the LocalNode call in this class which gets the local nodename for this instance.
|
|
GetAllInstallations.java |
expected output is the name and location of the oracle homes read from the inventory |
|
GetPatchDetails.java |
Performs a dual function - namely gets information about RAC nodes as well as the list of affected files by the patch whose ID is passed as inputs to this |
|
GetSingleInventory.java |
Returns the details associated with a particular home namely products installed and the one-off patches applied |
|
InitialDatabaseCheck |
One of the main classes which gets information about patches applied , nodes present if RAC which base bugs and files which are to be patched and if we have conflicts. Note that most of the checks in "CheckRACDetails.java" have been integrated into this class as initially this class did not have the RAC API calls |
|
PropagatePatchToCluster.java |
propagates the patch to the other nodes in the cluster |
|
PropagatePatchToNode.java |
propagates patch to a single node instead of all nodes or a set of nodes in the cluster |
|
RaiseOrLowerRACNode.java |
Details of the current RAC installation |
|
RemovePatchDetails.java |
Deregister the patch and the inventory information from multiple nodes if RAC |
|
UpdateInventory.java |
Update the inventory with patch information |
27) OPatch encounters java.lang.OutOfMemoryException when trying to apply patches , possibly comps.xml ?
Previous issues for OutOfMemoryExceptions pointed to the large size of comps.xml and the corruption of the inventory which was caused when using an OUI release lesser than 2.2.0.16.0 and OPatch for oneoff patch application. To check if the inventory was corrupted . Perform the following steps if the inventory has been corrupted
2878462 and the instructions for the same are as follows
- Install the new OUI version into the Oracle home with the affected inventory.
- During the process of installation of the product select "OUI only"
- Proceed with the installation of the product. The installation would cleanup the comps.xml file if it was corrupted in the first place .
- The location for the comps.xml would be in $ORACLE_HOME/inventory/ContentsXML/comps.xml and should be a few kb ( around 200 -300 ). If this is corrupted then this might be a few megs with spurious lines like ( this is an example )
--------------------------------------------------------------------------
<FILE PARENT_PATH="/u03/app/oracle/product/9.2/rdbms/mesg/"/>
<FILE PARENT_PATH="/u03/app/oracle/product/9.2/lib/"
CHILD_PATH="libserver9.a"/>
<FILE PARENT_PATH="/u03/app/oracle/product/9.2/lib/"
CHILD_PATH="libserver9.a"/>
<FILE PARENT_PATH="/u03/app/oracle/product/9.2/lib/"
CHILD_PATH="libserver9.a"/>
.... repeated over and over
-------------------------------------------------------------------------
- After the OUI new version is installed then these additional lines should not be present. Subsequent product installations would automatically be done using this version of the OUI. To verify the installer has been upgraded then in the list of "installed products" please check the Installer version and this should be 2.2.0.18.0. and the size of the comps.xml should be small after this application.
28) Different kinds of inventories and when are they synchronized - central inventory , local inventory and the important files which are accessed ? Where is this oracle home information picked up from ?
Most of this information is maintained in $ORACLE_HOME/inventory which is the local inventory. The central inventory is pointed to by the oraInst.loc file and these are usually in synchronization. The binary the XML version of the information stored in the inventory is in the Contents and ContentsXML folders in the local inventory and the
nodename and Oracle home information is picked from the ContentsXML/inventory file in the central inventory.
29) Installing OUI/OSP from the 2.2.0.18.0 patchset , should I install the OSP or only the OUI
Only the OUI installation option should be selected when applying the 2.2.0.18.0 version should be fine
30) Found a file called patch_free in $ORACLE_HOME/.opatch_storage is there anything which needs to be done. Also what about patch_locked ?
When the patch is being applied a patch_locked file is created with the information of the patch being applied , this should be removed once the patch application is complete. This file is required for the internal working of OPatch and the customer should not bother about it.
31) When applying a patch on the cluster faced the "PRKC-1021 : Problem in the clusterware" or a problem in the clusterware during the "getLocalNode call.
When trying to access the localnodename the previous API used "uname -a" to query
this information however in the newer versions of OPatch we make use of SRVM calls to get the local nodename. The nodename here is not necessarily the same as the hostname and the name of what the cluster knows this node as. The system calls for checking the cluster, checking if this is a clustered filesystem and subsequently getting the nodelist and local nodename is done using OUI API'S . Most of OPatch's interfaces are to the OUI API namely for the "PRKC-1021 : Problem in the clusterware" exception is usually encountered in "getLocalNode()". Most of these issues identified have been fixed by 1.0.0.0.39
32) What is the OPATCH_DEBUG=TRUE flag and when should this be set. Also
what information should be provided when filing a bug for an OPatch issue
- oraInst.loc file
- Local inventory namely $ORACLE_HOME/inventory/ContentsXML/comps.xml
- All files under the $ORACLE_HOME/.patch_storage directory tarred
- Central inventory pointed to by oraInst.loc - in the central inventory the ContentsXML/inventory file
- If failure with OPatch apply or lsinventory provide outputs with OPATCH_DEBUG=TRUE
- Details about which patch is being applied - bug number
- If the problem is not reproducible then the tarred copies of the inventories namely $ORACLE_HOME/inventory and the central inventory pointed to by oraInst.loc
- Debug output looks like :
Result is:
main:: set up Inventory...
setEnvironment()
setupInventory()
instantiate a OiicStandardInventorySession obj.
initSession()
getInstallAreaControl()
getInstallInventory()
getOracleHomeInfo()
found oracleHome /amer/ambde/v92
main:: probe for RAC...
OiipgDetectCluster.isCluster()
OiipgDetectCluster.checkCluster()
Cluster.isShared()
getNodeList()
getLocalNode()
output to OPatch:
HOME_INDEX=1
IS_CLUSTER=0
CHECK_CLUSTER=0
IS_CFS=0
NODE_LIST=NULL
NODE_COUNT=0
LOCAL_NODE=NULL
RAC_CODE=0
33) Can patches be rolled back using the rolling RAC or minimize_downtime options ?
For RAC installations there is currently no ability to uninstall a patch
using the equivalent of "rolling RAC" or "-minimum_downtime" that may have been used during the install process.
34) Received Exception in "thread "main" java.lang.UnsatisfiedLinkError: and no oraInstaller in java.library.path" when applying patches, what next ?
This is a unix specific issue. The most frequent reason for this is the liboraInstaller shared object file (liboraInstaller.so/sl) cannot be located at run time. The extension is ".so" for most systems, ".sl" for most other systems(usually HP) . It means LD_LIBRARY_PATH (or SHLIB_PATH, LIBPATH, DYLD_LIBRARY_PATH or it's equivalent)
doesn't include a path to liboraInstaller file. The file is located under the "oui" hierarchy (normally under $ORACLE_BASE/oui/bin/<plaform name>, like oui/bin/solaris). The "oui" directory should be in the same directory as the oraInventory directory that is pointed to by the oraInst.loc file.
35) When applying a patch what does it mean when I get the message "Not a valid patch area." ?
The directory OPatch is using to find the patch doesn't match the template for what it's checking for. Normally your running OPatch on directory out from where you think you are.
When starting OPatch the directory needs to have:
a. a directory .../etc that has the files that drive OPatch,
b. a directory .../files that has patches for the files, and
c. the directory should have the same name as patch id.
The third point can be confirmed by the file:
.../<patch_id>/etc/config/inventory
mso-bidi-font-size:10.0pt'>which is an XML file. The second line of the file has the XML tag "reference_id number" and this number is the patch_id. If you didn't start OPatch from the directory <patch_id> then you can tell OPatch where to look for this directory by giving the directory as the last argument to "apply", i.e. opatch apply /tmp/<patch_id>
36) Got exception " Exception in thread "main" java.lang.NullPointerException at
GetSingleInventory.main(GetSingleInventory.java:124" when trying to run the lsinventory command?
The inventory stores the location of each ORACLE_HOME as a string. The ORACLE_HOME it's using (either from the environment or from the "-oh" option) doesn't match any entry in the oracle homes picked up from the inventory. Check if the ORACLE_HOME environment variable has been set correctly and that the central inventory's file ContentsXML/inventory contains the relevant Oraclehome or if this oracle home was cloned from another machine and then use "attachhome" first and then apply the patch.
37) Which Oracle homes are registered with my central inventory ?
Use the command "lsinventory -all" to get a list of all recorded ORACLE_HOME entries.
38) How can I to minimize the downtime when applying a patch to my RAC?
The "-m" flag to the "apply" command and follow the prompts for the same. It would ask for a list of nodes. Ensure that the local instance is down first before applying the patch using minimize_downtime.
- The local node is always patched first.
- This is used a base to patch the other nodes.
- The user is prompted for the list of nodes to patch first out of the remaining nodes.
- For each node in this first list the user is asked to stop the instance and then the patch is propagated to that node before continuing to the next node.- When the initial group of nodes has been patched the user is asked to bring down the remaining nodes.
- When this is done the patch is propagated to this last group and the inventory is updated.The last instances are stopped on the remote nodes the memory used has been freed. You can then bring up the patched nodes (the first group) before patching the remaining nodes. This is not noted in the prompts at this time.
39) Why does the patch installation abort with the message that the "local node could not be determined."?
The patch tool needs to use some libraries that are expected to be in the search path. The libraries needed depeneds on the type of RAC installation: 32 or 64 bit namely the srvm libs. For 32 bit systems the libraries $ORACLE_HOME/lib32 and $ORACLE_HOME/srvm/lib either need to be included or moved to the front of LD_LIBRARY_PATH (or SHLIB_PATH for HP-UX). For 64 bit systems the libraries $ORACLE_HOME/lib and $ORACLE_HOME/srvm/lib either need to be included or moved to the front of LD_LIBRARY_PATH.
40) Questions on rolling RAC -
Rolling RAC patching allows the interoperation of a patched node and an unpatched node simultaneously. This means only one node is out of commission while it is patched. Note that currently this feature is being tested and only certain patches will be marked as rolling RAC enabled. This cannot be enabled from the command line. The user is
prompted to stop the instances on the node to be patched. First the local node is patched then the user is asked for the next node to patch from a list. As each node is patched the user is prompted when it is safe to restart the patched node. The cycle of prompt for a node, stop the instances on the node, patch the node and restarting the instances continues until stopped by the user or all nodes are patched.
41) What happens if the rolling RAC patch application process is interrupted
There is a prompt that allows the user to stop applying the patch. This means another patch cannot be applied until the process is restarted and all nodes are patched or the partially applied patch is rolled back. In the same manner the rolling back of a patch can also be stopped after any node in the same way applying a patch can be stopped. However the rollback process needs to finish before applying another patch or reapplying the same patch.
References
----------
Opatch documentation . source and specifications , thanks to David Evans for his help in understanding Opatch at its infancy and the OUI/Opatch team for making this FAQ.
References
@ BUG:2999294 - POTENTIAL RACE CONDITION IN SRVM CLIENT SIDE CODE CAN CAUSE CLIENTS TO HANG.
NOTE:224346.1 - OPatch - Where Can I Find the Latest Version of OPatch? [Video]
NOTE:293369.1 - Master Note For OPatch
NOTE:732697.1 - What Information Oracle Support Need To Work On OPatch Service Requests?
浙公网安备 33010602011771号