How To Downgrade Grid 19c Clusterware to 12.1

There may be many reasons for downgrading from Grid 19c to 12.1; however, the reason I am downgrading is to rerun a test requested by Oracle support from the 12.1 clusterware. Here are the steps performed.
### Check cluster:
oracle@ol7-121-rac2 ~]$ crsctl check cluster -all ************************************************************** ol7-121-rac1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** ol7-121-rac2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [oracle@ol7-121-rac2 ~]$
### Downgrade Node2:
(Can run the downgrade script in parallel on all cluster nodes, but one.)[root@ol7-121-rac2 ~]# env|egrep -i 'ORA|HOME|TNS' HOME=/root [root@ol7-121-rac2 ~]# ls /u01/app/19.3.0.0/grid/crs/install/rootcrs.sh /u01/app/19.3.0.0/grid/crs/install/rootcrs.sh [root@ol7-121-rac2 ~]# /u01/app/19.3.0.0/grid/crs/install/rootcrs.sh -downgrade Using configuration parameter file: /u01/app/19.3.0.0/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/oracle/crsdata/ol7-121-rac2/crsconfig/crsdowngrade_ol7-121-rac2_2019-12-09_10-58-49PM.log 2019/12/09 23:04:14 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector. 2019/12/09 23:05:04 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector. 2019/12/09 23:05:04 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node [root@ol7-121-rac2 ~]#
### Downgrade Node1:
(Downgrade the last node after downgrade all other nodes)[root@ol7-121-rac1 ~]# env|egrep -i 'ORA|HOME|TNS' HOME=/root [root@ol7-121-rac1 ~]# ls /u01/app/19.3.0.0/grid/crs/install/rootcrs.sh /u01/app/19.3.0.0/grid/crs/install/rootcrs.sh [root@ol7-121-rac1 ~]# /u01/app/19.3.0.0/grid/crs/install/rootcrs.sh -downgrade Using configuration parameter file: /u01/app/19.3.0.0/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/oracle/crsdata/ol7-121-rac1/crsconfig/crsdowngrade_ol7-121-rac1_2019-12-09_11-06-04PM.log CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.evmd' on 'ol7-121-rac1' CRS-2672: Attempting to start 'ora.mdnsd' on 'ol7-121-rac1' CRS-2676: Start of 'ora.evmd' on 'ol7-121-rac1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'ol7-121-rac1' CRS-2676: Start of 'ora.gpnpd' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ol7-121-rac1' CRS-2672: Attempting to start 'ora.gipcd' on 'ol7-121-rac1' CRS-2676: Start of 'ora.cssdmonitor' on 'ol7-121-rac1' succeeded CRS-2676: Start of 'ora.gipcd' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'ol7-121-rac1' CRS-2672: Attempting to start 'ora.diskmon' on 'ol7-121-rac1' CRS-2676: Start of 'ora.diskmon' on 'ol7-121-rac1' succeeded CRS-2676: Start of 'ora.cssd' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.crf' on 'ol7-121-rac1' CRS-2672: Attempting to start 'ora.ctssd' on 'ol7-121-rac1' CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ol7-121-rac1' CRS-2676: Start of 'ora.crf' on 'ol7-121-rac1' succeeded CRS-2676: Start of 'ora.ctssd' on 'ol7-121-rac1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'ol7-121-rac1' CRS-2676: Start of 'ora.asm' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.storage' on 'ol7-121-rac1' CRS-2676: Start of 'ora.storage' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'ol7-121-rac1' CRS-2676: Start of 'ora.crsd' on 'ol7-121-rac1' succeeded 2019/12/09 23:18:00 CLSRSC-338: Successfully downgraded OCR to version 12.1.0.2.0 CRS-2672: Attempting to start 'ora.crf' on 'ol7-121-rac1' CRS-2672: Attempting to start 'ora.storage' on 'ol7-121-rac1' CRS-2676: Start of 'ora.storage' on 'ol7-121-rac1' succeeded CRS-2676: Start of 'ora.crf' on 'ol7-121-rac1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'ol7-121-rac1' CRS-2676: Start of 'ora.crsd' on 'ol7-121-rac1' succeeded 2019/12/09 23:18:23 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector. 2019/12/09 23:18:49 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector. 2019/12/09 23:18:50 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node 2019/12/09 23:18:51 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/u01/app/12.1.0.2/grid'. 2019/12/09 23:18:52 CLSRSC-592: Run 'crsctl start crs' from home /u01/app/12.1.0.2/grid on each node to complete downgrade. 2019/12/09 23:19:13 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. [root@ol7-121-rac1 ~]#
### Remove Grid 19c from Clusterware:
[oracle@ol7-121-rac1 ~]$ /u01/app/19.3.0.0/grid/oui/bin/runInstaller -nowait -waitforcompletion -ignoreSysPrereqs \ > -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/19.3.0.0/grid \ > "CLUSTER_NODES=ol7-121-rac1, ol7-121-rac2" -doNotUpdateNodeList Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 18431 MB Passed The inventory pointer is located at /etc/oraInst.loc You can find the log of this install session at: /u01/app/oraInventory/logs/UpdateNodeList2019-12-09_11-57-44PM.log 'UpdateNodeList' was successful. [oracle@ol7-121-rac1 ~]$
### Configure Grid 12c with Clusterware:
[oracle@ol7-121-rac1 ~]$ /u01/app/12.1.0.2/grid/oui/bin/runInstaller -nowait -waitforcompletion -ignoreSysPrereqs \ > -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/12.1.0.2/grid \ > "CLUSTER_NODES=ol7-121-rac1, ol7-121-rac2" Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 18431 MB Passed The inventory pointer is located at /etc/oraInst.loc 'UpdateNodeList' was successful. [oracle@ol7-121-rac1 ~]$### Start crs for Node1
[root@ol7-121-rac1 ~]# . oraenv <<< +ASM1 ORACLE_SID = [root] ? The Oracle base has been set to /u01/app/oracle [root@ol7-121-rac1 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@ol7-121-rac1 ~]#
### Start crs for Node2:
[root@ol7-121-rac2 ~]# . oraenv <<< +ASM2 ORACLE_SID = [root] ? The Oracle base has been set to /u01/app/oracle [root@ol7-121-rac2 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@ol7-121-rac2 ~]#
### Validate Cluster:
[oracle@ol7-121-rac2 ~]$ olsnodes ol7-121-rac1 ol7-121-rac2 [oracle@ol7-121-rac2 ~]$ crsctl check cluster -all ************************************************************** ol7-121-rac1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** ol7-121-rac2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [oracle@ol7-121-rac2 ~]$ crsctl query crs softwareversion Oracle Clusterware version on node [ol7-121-rac2] is [12.1.0.2.0] [oracle@ol7-121-rac2 ~]$ crsctl query crs softwarepatch Oracle Clusterware patch level on node ol7-121-rac2 is [365281826]. [oracle@ol7-121-rac2 ~]$ crsctl query crs releaseversion Oracle High Availability Services release version on the local node is [12.1.0.2.0] [oracle@ol7-121-rac2 ~]$ crsctl query crs releasepatch Oracle Clusterware release patch level is [365281826] and the complete list of patches [28553832 ] have been applied on the local node. [oracle@ol7-121-rac2 ~]$ crsctl query crs activeversion -f Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [365281826]. [oracle@ol7-121-rac2 ~]$
### Validate inventory:
[oracle@ol7-121-rac2 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -all|head -50 Oracle Interim Patch Installer version 12.2.0.1.18 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/12.1.0.2/grid Central Inventory : /u01/app/oraInventory from : /u01/app/12.1.0.2/grid/oraInst.loc OPatch version : 12.2.0.1.18 OUI version : 12.1.0.2.0 Log file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/opatch2019-12-10_00-22-52AM_1.log Lsinventory Output file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2019-12-10_00-22-52AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: ol7-121-rac2.localdomain ARU platform id: 226 ARU platform description:: Linux x86-64 List of Oracle Homes: Name Location OraGI12Home1 /u01/app/12.1.0.2/grid OraDB12Home1 /u01/app/oracle/product/12.1.0.2/dbhome_1 OraGI19Home1 /u01/app/19.3.0.0/grid Installed Top-level Products (1): Oracle Grid Infrastructure 12c 12.1.0.2.0 There are 1 products installed in this Oracle Home. Interim patches (1) : Patch 28553832 : applied on Fri Dec 06 16:38:26 UTC 2019 Unique Patch ID: 22732107 Patch description: "OCW Interim patch for 28553832" Created on 11 Feb 2019, 02:30:42 hrs PST8PDT Bugs fixed: 21339083, 24831217, 18589889, 20768643, 19061429, 26512962, 26147987 19133945, 19341538, 19521169, 20011424, 19361757, 19187207, 21123932 19302350, 19130141, 23340259, 19530755, 21441387, 19028836, 18701017 19699720, 19620918, 19168690, 18899171, 19244316, 17005215, 19653795 19471722, 18634372, 19536337, 19184188, 26973611, 19131709, 20160490 20995001, 21131037, 24509984, 19380733, 20985167, 18943696, 20552947 19591984, 18135723, 18461246, 20014326, 19524857, 25250824, 22999793 18890943, 19154753, 25055154, 24808260, 19499021, 25206628, 20923930 19046190, 25074091, 20028666, 23170674, 24451580, 19051385, 19682695 19050688, 19831611, 22891868, 21607972, 26755613, 18922918, 21087799 25956724, 18956780, 19683886, 23058136, 23757020, 19026993, 20110156 21269977, 18261648, 18362971, 19558324, 22651583, 20185476, 18882642 19478119, 17447588, 19414274, 19262534, 20752167, 19147513, 22475834 [oracle@ol7-121-rac2 ~]$
### Deinstall Grid 19c Clusterware for Node1:
[oracle@ol7-121-rac1 ~]$ /u01/app/19.3.0.0/grid/oui/bin/runInstaller -detachHome -silent ORACLE_HOME=/u01/app/19.3.0.0/grid Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 18431 MB Passed The inventory pointer is located at /etc/oraInst.loc You can find the log of this install session at: /u01/app/oraInventory/logs/DetachHome2019-12-10_12-24-09AM.log 'DetachHome' was successful. [oracle@ol7-121-rac1 ~]$
### Deinstall Grid 19c Clusterware for Node2:
[oracle@ol7-121-rac2 ~]$ /u01/app/19.3.0.0/grid/oui/bin/runInstaller -detachHome -silent ORACLE_HOME=/u01/app/19.3.0.0/grid Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 18431 MB Passed The inventory pointer is located at /etc/oraInst.loc You can find the log of this install session at: /u01/app/oraInventory/logs/DetachHome2019-12-10_12-24-39AM.log 'DetachHome' was successful. [oracle@ol7-121-rac2 ~]$
### Check inventory for Node1:
[oracle@ol7-121-rac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -all|head -50 Oracle Interim Patch Installer version 12.2.0.1.18 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0.2/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0.2/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.18 OUI version : 12.1.0.2.0 Log file location : /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatch/opatch2019-12-10_00-34-03AM_1.log Lsinventory Output file location : /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2019-12-10_00-34-03AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: ol7-121-rac1.localdomain ARU platform id: 226 ARU platform description:: Linux x86-64 List of Oracle Homes: Name Location OraGI12Home1 /u01/app/12.1.0.2/grid OraDB12Home1 /u01/app/oracle/product/12.1.0.2/dbhome_1 Installed Top-level Products (1): Oracle Database 12c 12.1.0.2.0 There are 1 products installed in this Oracle Home. Interim patches (1) : Patch 28553832 : applied on Fri Dec 06 16:54:30 UTC 2019 Unique Patch ID: 22732107 Patch description: "OCW Interim patch for 28553832" Created on 11 Feb 2019, 02:30:42 hrs PST8PDT Bugs fixed: 21339083, 24831217, 18589889, 20768643, 19061429, 26512962, 26147987 19133945, 19341538, 19521169, 20011424, 19361757, 19187207, 21123932 19302350, 19130141, 23340259, 19530755, 21441387, 19028836, 18701017 19699720, 19620918, 19168690, 18899171, 19244316, 17005215, 19653795 19471722, 18634372, 19536337, 19184188, 26973611, 19131709, 20160490 20995001, 21131037, 24509984, 19380733, 20985167, 18943696, 20552947 19591984, 18135723, 18461246, 20014326, 19524857, 25250824, 22999793 18890943, 19154753, 25055154, 24808260, 19499021, 25206628, 20923930 19046190, 25074091, 20028666, 23170674, 24451580, 19051385, 19682695 19050688, 19831611, 22891868, 21607972, 26755613, 18922918, 21087799 25956724, 18956780, 19683886, 23058136, 23757020, 19026993, 20110156 21269977, 18261648, 18362971, 19558324, 22651583, 20185476, 18882642 19478119, 17447588, 19414274, 19262534, 20752167, 19147513, 22475834 18893463, 21242118, 19514085, 25496699, 20758997, 27738131, 27006180 [oracle@ol7-121-rac1 ~]$
### Check inventory for Node2:
[oracle@ol7-121-rac2 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -all|head -50 Oracle Interim Patch Installer version 12.2.0.1.18 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/12.1.0.2/grid Central Inventory : /u01/app/oraInventory from : /u01/app/12.1.0.2/grid/oraInst.loc OPatch version : 12.2.0.1.18 OUI version : 12.1.0.2.0 Log file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/opatch2019-12-10_00-24-54AM_1.log Lsinventory Output file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2019-12-10_00-24-54AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: ol7-121-rac2.localdomain ARU platform id: 226 ARU platform description:: Linux x86-64 List of Oracle Homes: Name Location OraGI12Home1 /u01/app/12.1.0.2/grid OraDB12Home1 /u01/app/oracle/product/12.1.0.2/dbhome_1 Installed Top-level Products (1): Oracle Grid Infrastructure 12c 12.1.0.2.0 There are 1 products installed in this Oracle Home. Interim patches (1) : Patch 28553832 : applied on Fri Dec 06 16:38:26 UTC 2019 Unique Patch ID: 22732107 Patch description: "OCW Interim patch for 28553832" Created on 11 Feb 2019, 02:30:42 hrs PST8PDT Bugs fixed: 21339083, 24831217, 18589889, 20768643, 19061429, 26512962, 26147987 19133945, 19341538, 19521169, 20011424, 19361757, 19187207, 21123932 19302350, 19130141, 23340259, 19530755, 21441387, 19028836, 18701017 19699720, 19620918, 19168690, 18899171, 19244316, 17005215, 19653795 19471722, 18634372, 19536337, 19184188, 26973611, 19131709, 20160490 20995001, 21131037, 24509984, 19380733, 20985167, 18943696, 20552947 19591984, 18135723, 18461246, 20014326, 19524857, 25250824, 22999793 18890943, 19154753, 25055154, 24808260, 19499021, 25206628, 20923930 19046190, 25074091, 20028666, 23170674, 24451580, 19051385, 19682695 19050688, 19831611, 22891868, 21607972, 26755613, 18922918, 21087799 25956724, 18956780, 19683886, 23058136, 23757020, 19026993, 20110156 21269977, 18261648, 18362971, 19558324, 22651583, 20185476, 18882642 19478119, 17447588, 19414274, 19262534, 20752167, 19147513, 22475834 18893463, 21242118, 19514085, 25496699, 20758997, 27738131, 27006180 [oracle@ol7-121-rac2 ~]$
### Remove Grid 19c directory for Node1:
[root@ol7-121-rac1 ~]# ls -ld /u01/app/19.3.0.0 drwxr-xr-x. 3 root oinstall 18 Dec 6 17:30 /u01/app/19.3.0.0 [root@ol7-121-rac1 ~]# rm -rf /u01/app/19.3.0.0 [root@ol7-121-rac1 ~]# ls -ld /u01/app/19.3.0.0 ls: cannot access /u01/app/19.3.0.0: No such file or directory [root@ol7-121-rac1 ~]# mkdir -p /u01/app/19.3.0.0 [root@ol7-121-rac1 ~]# chmod -R 775 /u01/app/19.3.0.0 [root@ol7-121-rac1 ~]# chown -R oracle:oinstall /u01/app/19.3.0.0 [root@ol7-121-rac1 ~]# ls -ld /u01/app/19.3.0.0 drwxrwxr-x. 2 oracle oinstall 6 Dec 10 00:26 /u01/app/19.3.0.0 [root@ol7-121-rac1 ~]#
### Remove Grid 19c directory for Node2:
[root@ol7-121-rac2 ~]# ls -ld /u01/app/19.3.0.0 drwxr-xr-x. 3 root oinstall 18 Dec 9 15:47 /u01/app/19.3.0.0 [root@ol7-121-rac2 ~]# rm -rf /u01/app/19.3.0.0 [root@ol7-121-rac2 ~]# ls -ld /u01/app/19.3.0.0 ls: cannot access /u01/app/19.3.0.0: No such file or directory [root@ol7-121-rac2 ~]# mkdir -p /u01/app/19.3.0.0 [root@ol7-121-rac2 ~]# chmod -R 775 /u01/app/19.3.0.0 [root@ol7-121-rac2 ~]# chown -R oracle:oinstall /u01/app/19.3.0.0 [root@ol7-121-rac2 ~]# ls -ld /u01/app/19.3.0.0 drwxrwxr-x. 2 oracle oinstall 6 Dec 10 00:29 /u01/app/19.3.0.0 [root@ol7-121-rac2 ~]#Hopefully the steps provided will be useful in the event that a downgrade is necessary.