Silent Out-of-Place Patching For Oracle 19c GI

I am going through a process of patching an Oracle Grid Infrastructure (GI) from 19.6 to 19.8 (Patch 31305339) and I wanted to try out-of-place patching (OOP). Since I didn't have access to the GUI (graphical user interface) environment, I proceeded to use the silent option. Below is how it went:

I started with verifying that the GI services were up and running using Fred Dennis's script and the current patches applied to the GI. This was a sanity check before I began work on the patch.

[grid@node1 ~]$ ./rac_status.sh -a
 
 Cluster rene-ace-c
 
  Type | Name | node1 | node2 |
  ---------------------------------------------------------------------
  asm | asm | Online | Online |
  asmnetwork | asmnet1 | Online | Online |
  chad | chad | Online | Online |
  cvu | cvu | Online | - |
  dg | DATA | Online | Online |
  dg | RECO | Online | Online |
  network | net1 | Online | Online |
  ons | ons | Online | Online |
  qosmserver | qosmserver | Online | - |
  vip | node1 | Online | - |
  vip | node2 | - | Online |
  vip | scan1 | - | Online |
  vip | scan2 | Online | - |
  vip | scan3 | Online | - |
  ---------------------------------------------------------------------
  x : Resource is disabled
  : Has been restarted less than 24 hours ago
  
  Listener | Port | node1 | node2 | Type |
  ------------------------------------------------------------------------------------
  ASMNET1LSNR_ASM| TCP:1525 | Online | Online | Listener |
  LISTENER | TCP:1521 | Online | Online | Listener |
  LISTENER_SCAN1 | TCP:1521 | - | Online | SCAN |
  LISTENER_SCAN2 | TCP:1521 | Online | - | SCAN |
  LISTENER_SCAN3 | TCP:1521 | Online | - | SCAN |
  ------------------------------------------------------------------------------------
  
  
  DB | Version | node1 | node2 | DB Type |
  ---------------------------------------------------------------------------------------
  renedev | 19.6.0.0 (1) | - | Open | SINGLE (P) |
  reneqa | 19.6.0.0 (2) | Open | Open | RAC (P) |
 
 [grid@node1 ~]$ echo $ORACLE_HOME
 /u01/app/19.3.0.0/grid
 [grid@node1 ~]$ cd $ORACLE_HOME/OPatch
 [grid@node1 OPatch]$ ./opatch lspatches
 30655595;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30655595)
 30557433;Database Release Update : 19.6.0.0.200114 (30557433)
 30489632;ACFS RELEASE UPDATE 19.6.0.0.0 (30489632)
 30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)
 
 OPatch succeeded.
 
 
  

With that done, I made sure the patches for 19.8 were applicable to the 19.6 GI and wouldn't lead to any conflicts or issues.

## Note.- I trimmed the log for readability
 [root@node1 ~]$ . oraenv
 ORACLE_SID = [root] ? +ASM1
 The Oracle base has been set to /u01/app/oracle
 [root@node1 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
 [root@node1 ~]$ opatchauto apply /u01/patches/30501910 -analyze
 ...
 Host:node1
 CRS Home:/u01/app/19.6.0.0/grid
 Version:19.0.0.0.0
 
 ==Following patches were SUCCESSFULLY analyzed to be applied:
 
 Patch: /u01/patches/31305339/31281355
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log
 
 Patch: /u01/patches/31305339/31304218
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log
 
 Patch: /u01/patches/31305339/31305087
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log
 
 Patch: /u01/patches/31305339/31335188
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log
 
 OPatchauto session completed at Wed Oct 7 11:42:46 2020
 
 Time taken to complete the session 2 minutes, 41 seconds

Once I was sure the 19.8 patch wouldn't cause an issue, I proceeded with the following methodology:

Step one was to unzip the 19.3 GI software to the ORACLE_HOME that will be the 19.8 patched GI. You can download the software from either OTN or edelivery. In my case I used edelivery.

I also updated the OPatch version to the one the 19.8 (31305339) patch needs, which is OPatch version 12.2.0.1.21 for DB 19.x releases (April 2020).

[grid@node1 grid]$ unzip -q /u01/patches/v982068-01.zip -d /u01/app/19.8.0.0/grid
 [grid@node1 grid]$ mv /u01/app/19.8.0.0/grid/OPatch /u01/app/19.8.0.0/grid/OPatch.old
 [grid@node1 grid]$ unzip -q /u01/patches/p6880880_190000_Linux-x86-64.zip -d /u01/app/19.8.0.0/grid
 

In these cases I normally unset my Oracle variables, so that whatever I do, I don't hurt anything currently running. After that I set the ORACLE_HOME to the 19.8 environment.

# a. Unset all Oracle related shell variables
 [grid@node1 ~]$ unset ORACLE_BASE
 [grid@node1 ~]$ unset ORACLE_HOME
 [grid@node1 ~]$ unset ORACLE_SID
 [grid@node1 ~]$ unset ORA_CRS_HOME
 [grid@node1 ~]$ unset ORA_NLS10
 [grid@node1 ~]$ unset TNS_ADMIN
  
 #b. The following must return nothing
 [grid@node1 ~]$ env | egrep "ORA|TNS" | wc -l
  0
 
 #c. Set the 19.8 ORACLE_HOME
 [grid@node1 ~]$ export ORACLE_HOME=/u01/app/19.8.0.0/grid

Now I was ready to run the gridSetup.sh in silent mode. I applied the 19.8 patch passing the response file parameters via command line.

When I did this, I faced the the patch error ZOP-51. I tried looking for a special character in what I was passing, but saw nothing. Even though I verified that the patch binaries where owned by grid:oinstall, and that everything on that location look good to me, I kept running into this error.

[grid@node1 ~]$ cd $ORACLE_HOME
 [grid@node1 ~]$ pwd
 /u01/app/19.8.0.0/grid
 [grid@node1 grid]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
 > -ApplyRU /u01/patches/31305339 \
 > -responseFile /u01/app/19.8.0.0/grid/install/response/gridsetup.rsp \
 > INVENTORY_LOCATION=/u01/app/oraInventory \
 > SELECTED_LANGUAGES=en,en_GB \
 > oracle.install.option=CRS_SWONLY \
 > ORACLE_BASE=/u01/app/grid \
 > oracle.install.asm.OSDBA=asmdba \
 > oracle.install.asm.OSOPER=asmoper \
 > oracle.install.asm.OSASM=asmadmin \
 > oracle.install.crs.config.ClusterConfiguration=STANDALONE \
 > oracle.install.crs.config.configureAsExtendedCluster=false \
 > oracle.install.crs.config.clusterName=rene-ace-c \
 > oracle.install.crs.config.gpnp.configureGNS=false \
 > oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
 > oracle.install.crs.config.clusterNodes=node1.localdomain,node2.localdomain
 Launching Oracle Grid Infrastructure Setup Wizard...
 
 [WARNING] [INS-13014] Target environment does not meet some optional requirements.
  CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log
  ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
 The response file for this session can be found at:
  /u01/app/19.8.0.0/grid/install/response/grid_2020-10-07_11-51-50AM.rsp
 
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log
 [FATAL] [INS-32500] OPatch tool failed.
  ACTION: Review /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log for more details.
 *ADDITIONAL INFORMATION:*
  - Oracle Interim Patch Installer version 12.2.0.1.21
  - Copyright (c) 2020, Oracle Corporation. All rights reserved.
  - 
  - 
  - ZOP-51: The patch location is not valid for apply, because it doesn't have correct metadata, or it points to a patch directory.
  - Please check the arguments and try again.
  - 
  - OPatch failed with error code 135
  - Argument(s) Error... Patch location is not valid for apply
  - 
 

To address this, I built a response file containing the exact values that I was passing in the command line. Once I did this, I was able to install the CRS without an issue.

[grid@node1 ~]$ cd $ORACLE_HOME
 [grid@node1 ~]$ pwd
 /u01/app/19.8.0.0/grid
 [grid@node1 grid]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent -responseFile /home/grid/grid_19.rsp -applyRU /u01/patches/31305339
 
 Preparing the home to patch...
 Applying the patch /u01/patches/31305339...
 Successfully applied the patch.
 The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/installerPatchActions_2020-10-07_12-32-13PM.log
 Launching Oracle Grid Infrastructure Setup Wizard...
 
 [WARNING] [INS-13014] Target environment does not meet some optional requirements.
  CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/gridSetupActions2020-10-07_12-32-13PM.log
  ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/gridSetupActions2020-10-07_12-32-13PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
 The response file for this session can be found at:
  /u01/app/19.8.0.0/grid/install/response/grid_2020-10-07_12-32-13PM.rsp
 
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/gridSetupActions2020-10-07_12-32-13PM.log
 
 As a root user, execute the following script(s):
 1. /u01/app/19.8.0.0/grid/root.sh
 
 Execute /u01/app/19.8.0.0/grid/root.sh on the following nodes: 
 [node1, node2]
 
 Successfully Setup Software with warning(s).
 ...
 [root@node1 patches]# /u01/app/19.8.0.0/grid/root.sh
 Check /u01/app/19.8.0.0/grid/install/root_node1_2020-10-07_13-01-25-549937823.log for the output of root script
 [root@node1 patches]# ssh node2
 Last login: Wed Oct 7 12:57:07 2020
 
 [root@node2 ~]# /u01/app/19.8.0.0/grid/root.sh
 Check /u01/app/19.8.0.0/grid/install/root_node2_2020-10-07_13-01-32-411313252.log for the output of root script
 

I next proceeded to switch the GI to the installed 19.8. Keep in mind that running the root.sh will take a while as this is essentially where the GI moves from the 19.6 to 19.8. It's also important never to run the root.sh concurrently; it has to run on node1 first and then on node2.

[grid@node1 ~]$ cd /u01/app/19.8.0.0/grid
 [grid@node1 grid]$ ./gridSetup.sh -switchGridHome -silent
 Launching Oracle Grid Infrastructure Setup Wizard...
 
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/cloneActions2020-10-07_01-19-16PM.log
 
 As a root user, execute the following script(s):
 1. /u01/app/19.8.0.0/grid/root.sh
 
 Execute /u01/app/19.8.0.0/grid/root.sh on the following nodes: 
 [node1, node2]
 
 Run the scripts on the local node first. After successful completion, run the scripts in sequence on all other nodes.
 
 Successfully Setup Software.
 ...
 [root@node1 patches]# /u01/app/19.8.0.0/grid/root.sh
 Check /u01/app/19.8.0.0/grid/install/root_node1_2020-10-07_13-32-59-377526581.log for the output of root script
 [root@node1 patches]# ssh node2
 Last login: Wed Oct 7 13:33:03 2020
 
 [root@node2 ~]# /u01/app/19.8.0.0/grid/root.sh
 Check /u01/app/19.8.0.0/grid/install/root_node2_2020-10-07_13-38-09-323573349.log for the output of root script
 
 

After running the root.sh in each node, I proceeded to update the /etc/oratab. I verified that the patches were correct. I also checked that the services that were up prior to the patching exercise remained up after it.

[grid@node1 ~]$ cat /etc/oratab |grep "+ASM"
 +ASM1:/u01/app/19.8.0.0/grid:N
 [grid@node1 ~]$ . oraenv
 ORACLE_SID = [+ASM1] ? 
 The Oracle base has been set to /u01/app/grid
 
 [grid@node1 OPatch]$ $ORACLE_HOME/OPatch/opatch lspatches
 31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
 31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
 31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
 31281355;Database Release Update : 19.8.0.0.200714 (31281355)
 
 OPatch succeeded.
 
 [grid@node1 ~]$ ./rac_status.sh -a
 
 Cluster rene-ace-c
 
  Type | Name | node1 | node2 |
  ---------------------------------------------------------------------
  asm | asm | Online | Online |
  asmnetwork | asmnet1 | Online | Online |
  chad | chad | Online | Online |
  cvu | cvu | Online | - |
  dg | DATA | Online | Online |
  dg | RECO | Online | Online |
  network | net1 | Online | Online |
  ons | ons | Online | Online |
  qosmserver | qosmserver | Online | - |
  vip | node1 | Online | - |
  vip | node2 | - | Online |
  vip | scan1 | - | Online |
  vip | scan2 | Online | - |
  vip | scan3 | Online | - |
  ---------------------------------------------------------------------
  x : Resource is disabled
  : Has been restarted less than 24 hours ago
  
  Listener | Port | node1 | node2 | Type |
  ------------------------------------------------------------------------------------
  ASMNET1LSNR_ASM| TCP:1525 | Online | Online | Listener |
  LISTENER | TCP:1521 | Online | Online | Listener |
  LISTENER_SCAN1 | TCP:1521 | - | Online | SCAN |
  LISTENER_SCAN2 | TCP:1521 | Online | - | SCAN |
  LISTENER_SCAN3 | TCP:1521 | Online | - | SCAN |
  ------------------------------------------------------------------------------------
  
  
  DB | Version | node1 | node2 | DB Type |
  ---------------------------------------------------------------------------------------
  renedev | 19.6.0.0 (1) | - | Open | SINGLE (P) |
  reneqa | 19.6.0.0 (2) | Open | Open | RAC (P) |
 

Last but not least, it's important to remove the 19.6 GI Home from the inventory and delete it. I recommend taking at least a few days to delete it as if you need to rollback this is where you will rollback to.

[grid@node1 ~]$ /u01/app/19.8.0.0/grid/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME="/u01/app/19.6.0.0/grid"
 Starting Oracle Universal Installer...
 
 Checking swap space: must be greater than 500 MB. Actual 4076 MB Passed
 The inventory pointer is located at /etc/oraInst.loc
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/DetachHome2020-10-07_02-26-40PM.log
 'DetachHome' was successful.
 
 # As root 
 [root@node1 ~]$ cd /u01/app/
 [root@node1 ~]$ rm -rf 19.6.0.0
 
 [grid@node2 ~]$ /u01/app/19.8.0.0/grid/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME="/u01/app/19.6.0.0/grid"
 Starting Oracle Universal Installer...
 
 Checking swap space: must be greater than 500 MB. Actual 4073 MB Passed
 The inventory pointer is located at /etc/oraInst.loc
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/DetachHome2020-10-07_02-26-54PM.log
 'DetachHome' was successful.
 
 # As root 
 [root@node2 ~]$ cd /u01/app/
 [root@node2 ~]$ rm -rf 19.6.0.0

I hope this blog helps if you're going to do a silent OOP Patching for your GI. I'm also including the grid rsp file I used so you can use it as reference.

Note: This was originally published on rene-ace.com.