How To Apply A Minimum Downtime OPatch
OVERVIEW
An Oracle OPatch for a RAC configuration can take take one of two forms, either as a Rolling Opatch or as a Minimum Downtime OPatch. The instructions for applying the former can be found here. Instructions for performing the latter can be found below.
In addition to being informed by Oracle Support, the type of OPatch can be determined by checking the <Patch No>/etc/config/inventory file. If the the variable online_rac_installable is set to true then the patch is a rolling patch.
A WORKED EXAMPLE
1. Ensure all of the instances that make up the RAC database can be shutdown.
2. On each node, shutdown the instance, using the following commands:
. oraenv
<SID name>
sqlplus '/ as sysdba'
shutdown transactional
exit
3. On the primary node in the cluster follow the Opatch instructions as prescribed by the README.txt file.
- Here is a real example:
cd $HOME/DBA/patches
unzip p6239052_10203_SOLARIS64.zip
cd 6239052
export OBJECT_MODE=32_64
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch apply
Invoking OPatch 10.2.0.3.0
Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..
Oracle Home : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.3.0
OUI version : 10.2.0.3.0
OUI location : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-17-48AM.log
ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
Invoking fuser to check for active processes.
Invoking fuser on "/u01/app/oracle/product/db10.2.0/bin/oracle"
OPatch detected the node list and the local node from the inventory.
OPatch will patch the local system then propagate the patch to the remote nodes.
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'node2' 'node3'
Local node: 'node1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')
Is the local system ready for patching?
Do you want to proceed? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '6239052' for restore. This might take a while...
Backing up files affected by the patch '6239052' for rollback. This might take a while...
Patching component oracle.rdbms, 10.2.0.3.0...
Updating archive file "/u01/app/oracle/product/db10.2.0/lib/libserver10.a"
with "lib/libserver10.a/kjbm.o"
ApplySession adding interim patch '6239052' to inventory
Verifying the update...
Inventory check OK: Patch ID 6239052 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 6239052 are present in Oracle Home.
Running make for target ioracle
The local system has been patched. You can restart Oracle instances on it.
Patching in rolling mode.
Remaining nodes to be patched:
'node2' 'node3'
What is the next node to be patched?
node2
You have selected 'node2' from 'node2' 'node3'
The node 'node2' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'node2'.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')
Is the node ready for patching?
Do you want to proceed? [y|n]
y
User Responded with: Y
Updating nodes 'node2'
Apply-related files are:
FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"
Instantiating the
file "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated"
by replacing $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
with actual path.
Propagating files to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt" with actual path.
Running command on remote node 'node2': cd /u01/app/oracle/product/db10.2.0/rdbms/lib;
/usr/ccs/bin/make -f ins_rdbms.mk
ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2
The node 'node2' has been patched. You can restart Oracle instances on it.
The node 'node3' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'node3'.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')
Is the node ready for patching?
Do you want to proceed? [y|n]
y
User Responded with: Y
Updating nodes 'node3'
Apply-related files are:
FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/..../rac/make_cmds.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
Running command on remote node 'node3': cd /u01/app/oracle/product/db10.2.0/rdbms/lib;
/usr/ccs/bin/make -f ins_rdbms.mk
ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2
The node 'node3' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes.
Remember to check the binary size and timestamp on the nodes 'node2' 'node3' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/db10.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk
ioracle $ORACLE_HOME=/u01/app/oracle/product/db10.2.0'
OPatch succeeded.
4. Ensure the new patch appears in the inventory by running the following command:
opatch lsinventory
Invoking OPatch 10.2.0.3.0
Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..
Oracle Home : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.3.0
OUI version : 10.2.0.3.0
OUI location : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/....log
Lsinventory Output file location : /u01/app/oracle/product/db10.2.0/....txt
--------------------------------------------------------------------------------
Installed Top-level Products (2):
Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 2 10.2.0.3.0
There are 2 products installed in this Oracle Home.
Interim patches (5) :
Patch 6239052 : applied on Tue Nov 20 02:18:58 GMT 2007
Created on 31 Oct 2007, 08:01:04 hrs PST8PDT
Bugs fixed:
6239052
Patch 4655998 : applied on Fri May 04 10:23:04 BST 2007
Created on 13 Apr 2007, 00:31:22 hrs PST8PDT
Bugs fixed:
4655998
Patch 5749953 : applied on Fri Apr 20 16:13:34 BST 2007
Created on 15 Feb 2007, 02:45:16 hrs US/Pacific
Bugs fixed:
5749953
Patch 5556081 : applied on Mon Apr 16 10:18:22 BST 2007
Created on 9 Nov 2006, 22:20:50 hrs PST8PDT
Bugs fixed:
5556081
Patch 5557962 : applied on Mon Apr 16 10:17:33 BST 2007
Created on 9 Nov 2006, 23:23:06 hrs PST8PDT
Bugs fixed:
4269423, 5557962, 5528974
Rac system comprising of multiple nodes
Local node = node1
Remote node = node2
Remote node = node3
--------------------------------------------------------------------------------
OPatch succeeded
5. On each node, start the instance, using the following commands:
. oraenv
<SID name>
sqlplus '/ as sysdba'
startup
exit
6. On the primary node, ensure the RAC cluster has returned to normal with the following commands:
- Note: It takes a few minutes to achieve a completely ONLINE state.
cd $HOME
. ./10gcrs.env
crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....L01.srv application ONLINE ONLINE node1
ora....L02.srv application ONLINE ONLINE node2
ora....L03.srv application ONLINE ONLINE node3
...
ora....b03.gsd application ONLINE ONLINE node3
ora....b03.ons application ONLINE ONLINE node3
ora....b03.vip application ONLINE ONLINE node3
TROUBLESHOOTING
- If the OPatch is NOT applied to the primary node in the cluster then the following error output will be produced:
- ACTION: Restart the patch process on the primary node.
Invoking OPatch 10.2.0.3.0
Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..
Oracle Home : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.3.0
OUI version : 10.2.0.3.0
OUI location : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-16-36AM.log
ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/db10.2.0. Possible #
causes are:
No read or write permission to ORACLE_HOME/.patch_storage
Central Inventory is locked by another OUI instance
No read permission to Central Inventory
The lock file exists in ORACLE_HOME/.patch_storage
The Oracle Home does not exist in Central Inventory
ApplySession failed: ApplySession failed to prepare the system. No detail available.
System intact, OPatch will not attempt to restore the system
OPatch failed with error code 73
- If the crs_stat -t output doesn't show all items in an ONLINE state after a suitable amount of time, e.g. 15 mins, the cluster may require restarting completely.
- ACTION: On each node, ONE NODE AT A TIME, use the following commands:
su -
<root password>
sync;sync;reboot
page revision: 6, last edited: 04 Oct 2010 13:30
