How To Apply A Minimum Downtime OPatch

OVERVIEW

An Oracle OPatch for a RAC configuration can take take one of two forms, either as a Rolling Opatch or as a Minimum Downtime OPatch. The instructions for applying the former can be found here. Instructions for performing the latter can be found below.

In addition to being informed by Oracle Support, the type of OPatch can be determined by checking the <Patch No>/etc/config/inventory file. If the the variable online_rac_installable is set to true then the patch is a rolling patch.

A WORKED EXAMPLE

1. Ensure all of the instances that make up the RAC database can be shutdown.

2. On each node, shutdown the instance, using the following commands:

         . oraenv
         <SID name>
         sqlplus '/ as sysdba'
         shutdown transactional
         exit

3. On the primary node in the cluster follow the Opatch instructions as prescribed by the README.txt file.

  • Here is a real example:
         cd $HOME/DBA/patches
         unzip p6239052_10203_SOLARIS64.zip
         cd 6239052
         export OBJECT_MODE=32_64
         export PATH=$ORACLE_HOME/OPatch:$PATH
         opatch apply

         Invoking OPatch 10.2.0.3.0

         Oracle interim Patch Installer version 10.2.0.3.0
         Copyright (c) 2005, Oracle Corporation.  All rights reserved..

         Oracle Home       : /u01/app/oracle/product/db10.2.0
         Central Inventory : /u01/app/oracle/oraInventory
            from           : /var/opt/oracle/oraInst.loc
         OPatch version    : 10.2.0.3.0
         OUI version       : 10.2.0.3.0
         OUI location      : /u01/app/oracle/product/db10.2.0/oui
         Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-17-48AM.log

         ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
         Invoking fuser to check for active processes.
         Invoking fuser on "/u01/app/oracle/product/db10.2.0/bin/oracle"

         OPatch detected the node list and the local node from the inventory.  
         OPatch will patch the local system then propagate the patch to the remote nodes.

         This node is part of an Oracle Real Application Cluster.
         Remote nodes: 'node2' 'node3'
         Local node: 'node1'
         Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
         (Oracle Home = '/u01/app/oracle/product/db10.2.0')

         Is the local system ready for patching?

         Do you want to proceed? [y|n]
         y
         User Responded with: Y
         Backing up files and inventory (not for auto-rollback) for the Oracle Home
         Backing up files affected by the patch '6239052' for restore. This might take a while...
         Backing up files affected by the patch '6239052' for rollback. This might take a while...

         Patching component oracle.rdbms, 10.2.0.3.0...
         Updating archive file "/u01/app/oracle/product/db10.2.0/lib/libserver10.a"  
         with "lib/libserver10.a/kjbm.o"
         ApplySession adding interim patch '6239052' to inventory

         Verifying the update...
         Inventory check OK: Patch ID 6239052 is registered in Oracle Home inventory with proper meta-data.
         Files check OK: Files from Patch ID 6239052 are present in Oracle Home.
         Running make for target ioracle

         The local system has been patched.  You can restart Oracle instances on it.

         Patching in rolling mode.

         Remaining nodes to be patched:
         'node2' 'node3'
         What is the next node to be patched?
         node2
         You have selected 'node2' from 'node2' 'node3'

         The node 'node2' will be patched next.

         Please shutdown Oracle instances running out of this ORACLE_HOME on 'node2'.
         (Oracle Home = '/u01/app/oracle/product/db10.2.0')
         Is the node ready for patching?

         Do you want to proceed? [y|n]
         y
         User Responded with: Y
         Updating nodes 'node2'
            Apply-related files are:
              FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
              DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
              MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"

         Instantiating the  
         file "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated" 
         by replacing $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt" 
         with actual path.
         Propagating files to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt" with actual path.
         Propagating directories to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt" with actual path.
         Running command on remote node 'node2': cd /u01/app/oracle/product/db10.2.0/rdbms/lib; 
         /usr/ccs/bin/make -f ins_rdbms.mk
         ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2

         The node 'node2' has been patched.  You can restart Oracle instances on it.

         The node 'node3' will be patched next.

         Please shutdown Oracle instances running out of this ORACLE_HOME on 'node3'.
         (Oracle Home = '/u01/app/oracle/product/db10.2.0')
         Is the node ready for patching?

         Do you want to proceed? [y|n]
         y
         User Responded with: Y
         Updating nodes 'node3'
            Apply-related files are:
              FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
              DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
              MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"

         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt" with actual path.
         Propagating files to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
         Propagating directories to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/..../rac/make_cmds.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
         Running command on remote node 'node3': cd /u01/app/oracle/product/db10.2.0/rdbms/lib; 
         /usr/ccs/bin/make -f ins_rdbms.mk
         ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2

         The node 'node3' has been patched.  You can restart Oracle instances on it.

         There were relinks on remote nodes.  
         Remember to check the binary size and timestamp on the nodes 'node2' 'node3' .
         The following make commands were invoked on remote nodes:
         'cd /u01/app/oracle/product/db10.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk 
          ioracle $ORACLE_HOME=/u01/app/oracle/product/db10.2.0'

         OPatch succeeded.

4. Ensure the new patch appears in the inventory by running the following command:

         opatch lsinventory

         Invoking OPatch 10.2.0.3.0

         Oracle interim Patch Installer version 10.2.0.3.0
         Copyright (c) 2005, Oracle Corporation.  All rights reserved..

         Oracle Home       : /u01/app/oracle/product/db10.2.0
         Central Inventory : /u01/app/oracle/oraInventory
            from           : /var/opt/oracle/oraInst.loc
         OPatch version    : 10.2.0.3.0
         OUI version       : 10.2.0.3.0
         OUI location      : /u01/app/oracle/product/db10.2.0/oui
         Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/....log

         Lsinventory Output file location : /u01/app/oracle/product/db10.2.0/....txt

         --------------------------------------------------------------------------------
         Installed Top-level Products (2):

         Oracle Database 10g                                                  10.2.0.1.0
         Oracle Database 10g Release 2 Patch Set 2                            10.2.0.3.0
         There are 2 products installed in this Oracle Home.

         Interim patches (5) :

         Patch  6239052      : applied on Tue Nov 20 02:18:58 GMT 2007
            Created on 31 Oct 2007, 08:01:04 hrs PST8PDT
            Bugs fixed:
              6239052

         Patch  4655998      : applied on Fri May 04 10:23:04 BST 2007
            Created on 13 Apr 2007, 00:31:22 hrs PST8PDT
            Bugs fixed:
              4655998

         Patch  5749953      : applied on Fri Apr 20 16:13:34 BST 2007
            Created on 15 Feb 2007, 02:45:16 hrs US/Pacific
            Bugs fixed:
              5749953

         Patch  5556081      : applied on Mon Apr 16 10:18:22 BST 2007
            Created on 9 Nov 2006, 22:20:50 hrs PST8PDT
            Bugs fixed:
              5556081

         Patch  5557962      : applied on Mon Apr 16 10:17:33 BST 2007
            Created on 9 Nov 2006, 23:23:06 hrs PST8PDT
            Bugs fixed:
              4269423, 5557962, 5528974

         Rac system comprising of multiple nodes
           Local node = node1
           Remote node = node2
           Remote node = node3

         --------------------------------------------------------------------------------

         OPatch succeeded

5. On each node, start the instance, using the following commands:

         . oraenv
         <SID name>
         sqlplus '/ as sysdba'
         startup
         exit

6. On the primary node, ensure the RAC cluster has returned to normal with the following commands:

  • Note: It takes a few minutes to achieve a completely ONLINE state.
         cd $HOME
         . ./10gcrs.env
         crs_stat -t

         Name           Type           Target    State     Host
         ------------------------------------------------------------
         ora....L01.srv application    ONLINE    ONLINE    node1
         ora....L02.srv application    ONLINE    ONLINE    node2
         ora....L03.srv application    ONLINE    ONLINE    node3
         ...
         ora....b03.gsd application    ONLINE    ONLINE    node3
         ora....b03.ons application    ONLINE    ONLINE    node3
         ora....b03.vip application    ONLINE    ONLINE    node3

TROUBLESHOOTING

  • If the OPatch is NOT applied to the primary node in the cluster then the following error output will be produced:
    • ACTION: Restart the patch process on the primary node.
         Invoking OPatch 10.2.0.3.0

         Oracle interim Patch Installer version 10.2.0.3.0
         Copyright (c) 2005, Oracle Corporation.  All rights reserved..

         Oracle Home       : /u01/app/oracle/product/db10.2.0
         Central Inventory : /u01/app/oracle/oraInventory
            from           : /var/opt/oracle/oraInst.loc
         OPatch version    : 10.2.0.3.0
         OUI version       : 10.2.0.3.0
         OUI location      : /u01/app/oracle/product/db10.2.0/oui
         Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-16-36AM.log

         ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
         OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/db10.2.0. Possible #
         causes are:
            No read or write permission to ORACLE_HOME/.patch_storage
            Central Inventory is locked by another OUI instance
            No read permission to Central Inventory
            The lock file exists in ORACLE_HOME/.patch_storage
            The Oracle Home does not exist in Central Inventory

         ApplySession failed: ApplySession failed to prepare the system.  No detail available.
         System intact, OPatch will not attempt to restore the system

         OPatch failed with error code 73
  • If the crs_stat -t output doesn't show all items in an ONLINE state after a suitable amount of time, e.g. 15 mins, the cluster may require restarting completely.
    • ACTION: On each node, ONE NODE AT A TIME, use the following commands:
         su -
         <root password>
         sync;sync;reboot
© copyright 2001-2014 ABCdba.com | all rights reserved