OPatchAuto Bug on 19.3 GI/RDBMS When Patching to 19.6

The other day I was doing a fresh install of Oracle 19.3. I was using the binaries from edelivery, doing some testing in that version, then patching to 19.6. The installation and the tests went fine. Next, I proceeded to analyze patch 30501910 and saw that everything was OK.

## Note.- I trimmed the log for readability
 [root@node2 ~]$ . oraenv
 ORACLE_SID = [root] ? DB193H12
 The Oracle base has been set to /u01/app/oracle
 [root@node2 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
 [root@node2 ~]$ opatchauto apply /u01/patches/30501910 -analyze
 OPatchauto session is initiated at Tue Oct 6 13:23:11 2020
 ...
 Host:node2
 CRS Home:/u01/app/19.3.0.0/grid
 Version:19.0.0.0.0
 
 ==Following patches were SUCCESSFULLY analyzed to be applied:
 
 Patch: /u01/patches/30501910/30489227
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log
 
 Patch: /u01/patches/30501910/30489632
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log
 
 Patch: /u01/patches/30501910/30655595
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log
 
 Patch: /u01/patches/30501910/30557433
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log
 
 OPatchauto session completed at Tue Oct 6 11:42:46 2020
 
 Time taken to complete the session 2 minutes, 41 seconds
 
 
 

I proceeded with the patch with the command below:

opatchauto apply /u01/patches/30501910

Everything went well in node 1, with the patching completed successfully in GI and RDBMS, but the patch on node 2 failed on the RDBMS with the error below:

## Note.- I trimmed the log for readability
 [root@node2 ~]$ opatchauto apply /u01/patches/30501910
 ...
 ---------------------------Patching Failed---------------------------------
 Command execution failed during patching in home: /u01/app/oracle/product/19.3.0.0/dbhome_1, host: node2.
 Command failed: /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/opatchauto apply /u01/patches/30501910 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_node2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_node2_rac_2.ser
 Command failure output: 
 ==Following patches FAILED in apply:
 
 Patch: /u01/patches/30501910/30489227
 Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-05_17-25-22PM_1.log
 Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 
 
 After fixing the cause of failure Run opatchauto resume
 ...

After looking for information, I found MOS (My Oracle Support) note 2582139.1, which directed me to copy the oui-patch.xml from node 1 to node 2.

[grid@node1 ~]$ cd /u01/app/oraInventory/ContentsXML/
 [grid@node1 ContentsXML]$ scp oui-patch.xml node2:/u01/app/oraInventory/ContentsXML/
 oui-patch.xml 
  
 [root@node2 ContentsXML]# ls -ltr
 total 20
 -rw-rw----. 1 grid oinstall 565 Oct 6 11:13 inventory.xml
 -rw-r--r--. 1 oracle oinstall 174 Oct 6 12:43 oui-patch.xml.back
 -rw-rw----. 1 grid oinstall 300 Oct 6 12:43 comps.xml
 -rw-rw----. 1 grid oinstall 292 Oct 6 12:43 libs.xml
 -rw-rw----. 1 grid oinstall 174 Oct 6 14:02 oui-patch.xml 

After doing this, I reran the OPatchAuto. This time I faced the error "Unable to create patchObject."

## Note.- I trimmed the log for readability
 [root@node2 ~]$ opatchauto apply /u01/patches/30501910
 ...
 ---------------------------Patching Failed---------------------------------
 Command execution failed during patching in home: /u01/app/oracle/product/19.3.0.0/dbhome_1, host: node2.
 Command failed: /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/opatchauto apply /u01/patches/30501910 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_node2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_node2_rac_2.ser
 Command failure output: 
 ==Following patches FAILED in apply:
 
 Patch: /u01/patches/30501910/30489227
 Log: 
 Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
 Possible causes are:
  ORACLE_HOME/inventory/oneoffs/30489227 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.3.0.0/dbhome_1/inventory/oneoffs/30489227/etc/config/actions" or "/u01/app/oracle/product/19.3.0.0/dbhome_1/inventory/oneoffs/30489227/etc/config/inventory" does not exist. 
 
 After fixing the cause of failure Run opatchauto resume
 ...
 

To address this error, I found another note — 2286025.1 — which advises taking any of the following actions:

1. OPatch lsinventory
2. OPatch - analyze
3. Restore database home backup for Hostname2.

After trying options 1 and 2, I kept getting the same error, so I proceeded to patch the GI on node 2.

## Note.- I trimmed the log for readability
 [root@node2 ~]$ . oraenv
 ORACLE_SID = [root] ? +ASM2
 The Oracle base has been set to /u01/app/grid
 [root@node2 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
 [root@node2 ~]$ opatchauto apply /u01/patches/30501910 -oh $ORACLE_HOME
 OPatchauto session is initiated at Tue Oct 6 12:28:25 2020
 
 ...
 
 --------------------------------Summary--------------------------------
 
 Patching is completed successfully. Please find the summary as follows:
 
 Host:node2
 CRS Home:/u01/app/19.3.0.0/grid
 Version:19.0.0.0.0
 Summary:
 
 ==Following patches were SUCCESSFULLY applied:
 
 Patch: /u01/patches/30501910/30489227
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log
 
 Patch: /u01/patches/30501910/30489632
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log
 
 Patch: /u01/patches/30501910/30557433
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log
 
 Patch: /u01/patches/30501910/30655595
 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log
 
 OPatchauto session completed at Tue Oct 6 12:49:02 2020
 Time taken to complete the session 20 minutes, 38 seconds
 

For the RDBMS I followed my own blog post (How to Clone an Oracle Home in 19c) and cloned the Oracle home from node 1, as this one was already patched correctly with 30501910.

[oracle@node1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
 [oracle@node1 ~]$ $ORACLE_HOME/runInstaller -silent -createGoldImage -destinationLocation /u01/patches
 Launching Oracle Database Setup Wizard...
 
 Successfully Setup Software.
 Gold Image location: /u01/patches/db_home_2020-10-06_02-11-59PM.zip
 
 [oracle@node1 ~]$ unset ORACLE_HOME
 [oracle@node1 ~]$ unset ORACLE_SID
 [oracle@node1 ~]$ unset ORACLE_RSID
 [oracle@node1 ~]$ unset ORACLE_UNQNAME
 [oracle@node1 ~]$ unset ORACLE_BASE
 
 [oracle@node1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_2
 [oracle@node1 ~]$ export ORACLE_HOSTNAME=node1
 [oracle@node1 ~]$ export ORA_INVENTORY=/u01/app/oraInventory
 [oracle@node1 ~]$ export NODE1_HOSTNAME=node1
 [oracle@node1 ~]$ export NODE2_HOSTNAME=node2
 [oracle@node1 ~]$ export ORACLE_BASE=/u01/app/oracle
 
 [oracle@node1 ~]$ unzip -q /u01/patches/db_home_2020-10-06_02-11-59PM.zip -d $ORACLE_HOME
  
 [oracle@node1 ~]$ env | egrep "ORA|NODE"
 ORACLE_BASE=/u01/app/oracle
 NODE2_HOSTNAME=node2
 ORACLE_HOSTNAME=node1
 NODE1_HOSTNAME=node1
 ORA_INVENTORY=/u01/app/oraInventory
 ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_2
 
 [oracle@node1 ~]$ cd $ORACLE_HOME
 [oracle@node1 dbhome_2]$ ${ORACLE_HOME}/runInstaller -ignorePrereq -waitforcompletion -silent \
 > -responseFile ${ORACLE_HOME}/install/response/db_install.rsp \
 > oracle.install.option=INSTALL_DB_SWONLY \
 > ORACLE_HOSTNAME=${ORACLE_HOSTNAME} \
 > UNIX_GROUP_NAME=oinstall \
 > INVENTORY_LOCATION=${ORA_INVENTORY} \
 > SELECTED_LANGUAGES=en \
 > ORACLE_HOME=${ORACLE_HOME} \
 > ORACLE_BASE=${ORACLE_BASE} \
 > oracle.install.db.InstallEdition=EE \
 > oracle.install.db.OSDBA_GROUP=dba \
 > oracle.install.db.OSOPER_GROUP=dba \
 > oracle.install.db.OSBACKUPDBA_GROUP=dba \
 > oracle.install.db.OSDGDBA_GROUP=dba \
 > oracle.install.db.OSKMDBA_GROUP=dba \
 > oracle.install.db.OSRACDBA_GROUP=dba \
 > oracle.install.db.CLUSTER_NODES=${NODE1_HOSTNAME},${NODE2_HOSTNAME} \
 > oracle.install.db.isRACOneInstall=false \
 > oracle.install.db.rac.serverpoolCardinality=0 \
 > oracle.install.db.config.starterdb.type=GENERAL_PURPOSE \
 > oracle.install.db.ConfigureAsContainerDB=false \
 > SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
 > DECLINE_SECURITY_UPDATES=true
 Launching Oracle Database Setup Wizard...
 ...
 The response file for this session can be found at:
  /u01/app/oracle/product/19.3.0.0/dbhome_2/install/response/db_2020-10-06_03-06-10PM.rsp
 
 You can find the log of this install session at:
  /u01/app/oraInventory/logs/InstallActions2020-10-06_03-06-10PM/installActions2020-10-06_03-06-10PM.log
 
 As a root user, execute the following script(s):
 1. /u01/app/oracle/product/19.3.0.0/dbhome_2/root.sh
 
 Execute /u01/app/oracle/product/19.3.0.0/dbhome_2/root.sh on the following nodes: 
 [node1, node2]
 
 

After doing this, the only task remaining was to verify that the Oracle home was patched correctly, and switch the database to the new Oracle home.

[oracle@node1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
 30557433;Database Release Update : 19.6.0.0.200114 (30557433)
 30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)
 
 OPatch succeeded.
 
 [oracle@node2 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
 30557433;Database Release Update : 19.6.0.0.200114 (30557433)
 30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)
 
 OPatch succeeded.
 
 [oracle@node1 ~]$ srvctl modify database -db test -oraclehome /u01/app/oracle/product/19.3.0.0/dbhome_2
 [oracle@node1 ~]$ srvctl config database -db test -a
 Database unique name: test
 Database name: test
 Oracle home: /u01/app/oracle/product/19.3.0.0/dbhome_2
 Oracle user: oracle
 Spfile: +DATA/test/PARAMETERFILE/spfile.270.1053084685
 Password file: +DATA/test/PASSWORD/pwdtest.258.1053083711
 Domain: 
 Start options: open
 Stop options: immediate
 Database role: PRIMARY
 Management policy: AUTOMATIC
 Server pools: 
 Disk Groups: RECO,DATA
 Mount point paths: 
 Services: 
 Type: RAC
 Start concurrency: 
 Stop concurrency: 
 OSDBA group: dba
 OSOPER group: oper
 Database instances: test1,test2
 Configured nodes: node1,node2
 CSS critical: no
 CPU count: 0
 Memory target: 0
 Maximum memory: 0
 Default network number for database services: 
 Database is administrator managed
 

I plan to re-try this exercise by copying the oui-patch.xml before patching and see if that fixes this error. In the meantime, I hope this blog post helps you in case you face the above-mentioned bugs.

Ready to optimize your Oracle Database for the future?