Upgrading grid infrastructure from 12.1 to 12.2 in Exadata

1 Pre-requisites / preparation
1.1 Install GI 12.2 from exadatadb01.
- 12.2 Grid Infrastructure and Database Upgrade steps for Exadata Database Machine running 11.2.0.3 and later on Oracle Linux (Doc ID 2111010.1)
- Patches to apply before upgrading Oracle GI and DB to 12.2.0.1 (Doc ID 2180188.1)
sudo su - dcli -g /tmp/dbs_group -l root "ls -ltr /u01/app/12.2.0.1/grid" dcli -g /tmp/dbs_group -l root mkdir -p /u01/app/12.2.0.1/grid dcli -g /tmp/dbs_group -l root chown oracle:oinstall /u01/app/12.2.0.1/grid dcli -g /tmp/dbs_group -l root "ls -altr /u01/app/12.2.0.1/grid" # Install GI using this gold image : /software/V840012-01.zip sudo su - oracle unzip -q /software/V840012-01.zip -d /u01/app/12.2.0.1/grid
1.2 Upgrade OPatch if OPatch is not already at the latest version:
[oracle@exadatadb01 ~]$ unzip -q /software/27475857/Database/OPatch/12.2/12.2.0.1.13/p6880880_12*_Linux-x86-64.zip -d /u01/app/12.2.0.1/grid
1.3 Check that Oracle has a 10240 limits for the "soft stack" (if not, set it, log off and log on).
1.4 Verify that 1500 huge pages are available.[root@exadatadb01~]$ dcli -g ~/dbs_group -l root grep stack /etc/security/limits.conf | grep soft exadatadb01: * soft stack 10240 exadatadb02: * soft stack 10240 exadatadb03: * soft stack 10240 exadatadb04: * soft stack 10240
[oracle@exadatadb01~]$ dcli -g ~/dbs_group -l root grep -i huge /proc/meminfo .... AnonHugePages: 0 kB HugePages_Total: 200000 HugePages_Free: 132171 HugePages_Rsvd: 38338 HugePages_Surp: 0 Hugepagesize: 2048 kB ....
1.5 Fix the ntp failures in prerequisites check.
-- Verifying NTP daemon is synchronized with at least one external time source ... FAILED.
It was resolved by setting entry to /etc/ntp.conf (on all db nodes): tos maxdist 16and restarting ntpd service ntpd stop service ntpd start
1.6 Run the prerequisites.
2 Upgrade GI to 12.2sudo su - oracle cd /u01/app/12.2.0.1/grid ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/12.2.0.1/grid -dest_version 12.2.0.1 -fixup -verbose
2.1 Set the ASM memory prerequisites.
2.2 Reset mis-count.sqlplus / as sysasm alter system set sga_max_size = 3G scope=spfile sid='*'; alter system set sga_target = 3G scope=spfile sid='*'; alter system set memory_target=0 sid='*' scope=spfile; alter system set memory_max_target=0 sid='*' scope=spfile /* required workaround */; alter system reset memory_max_target sid='*' scope=spfile; alter system set use_large_pages=true sid='*' scope=spfile /* 11.2.0.2 and later(Linux only) */;
2.3 Check for any rebalance operation (there should be none).sudo su - oracle . oraenv <<< +ASM1 crsctl unset css misscount
2.4 Gridsetup - verify the response file.sudo su - oracle . oraenv <<< +ASM1 sqlplus / as sysasm select count(*) from gv$asm_operation;
2.5 Gridsetup - run from exadatadb01.[oracle@exadatadb01 ~]$ egrep -v "^#|^$" /tmp/giresponse.rsp | head -10 oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v12.2.0 INVENTORY_LOCATION= oracle.install.option=UPGRADE ORACLE_BASE=/u01/app/oracle oracle.install.asm.OSDBA=dba oracle.install.asm.OSOPER=dba oracle.install.asm.OSASM=dba oracle.install.crs.config.gpnp.scanName= oracle.install.crs.config.gpnp.scanPort= oracle.install.crs.config.ClusterConfiguration= [oracle@exadatadb01 ~]$
2.6 rootupgrade.sh on exadatadb01/2/3:sudo su - oracle cd /u01/app/12.2.0.1/grid ./gridSetup.sh -silent -responseFile /tmp/giresponse.rsp -J-Doracle.install.mgmtDB=false -J-Doracle.install.crs.enableRemoteGIMR=false -applyPSU /software/27475857/Database/12.2.0.1.0/12.2.0.1.180417GIRU/27850694
2.7 rootupgrade.sh on exadatadb04, it will be the last one./u01/app/12.2.0.1/grid/rootupgrade.sh
2.8 Check if GI is relinked with RDS:/u01/app/12.2.0.1/grid/rootupgrade.sh
# If not, do it:sudo su - oracle dcli -g /tmp/dbs_group -l oracle /u01/app/12.2.0.1/grid/bin/skgxpinfo => the command should return "rds" for each node
2.9 Check the status of the cluster and the installed version.dcli -g /tmp/dbs_group -l oracle ORACLE_HOME=/u01/app/12.2.0.1/grid \ make -C /u01/app/12.2.0.1/grid/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle
2.10 Modify ASM.sudo su - oracle /u01/app/12.2.0.1/grid/bin/crsctl check cluster -all /u01/app/12.2.0.1/grid/bin/crsctl query crs activeversion
2.11 Update compatible.asm on the diskgroups.sudo su - oracle srvctl modify asm -count ALL
2.12 Update inventory.$ . oraenv <<< +ASM1 $ sqlplus / as sysasm SQL> select name, COMPATIBILITY from v$asm_diskgroup ; SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.asm' = '12.2.0.1.0'; SQL> ALTER DISKGROUP DBFS_DG SET ATTRIBUTE 'compatible.asm' = '12.2.0.1.0'; SQL> ALTER DISKGROUP RECO SET ATTRIBUTE 'compatible.asm' = '12.2.0.1.0'; SQL> select name, COMPATIBILITY from v$asm_diskgroup ;
3 Known issues$ . oraenv <<< +ASM1 $ /u01/app/12.2.0.1/grid/oui/bin/runInstaller -ignoreSysPrereqs -updateNodeList ORACLE_HOME=/u01/app/12.2.0.1/grid "CLUSTER_NODES={exadatadb01,exadatadb02,exadatadb03,exadatadb04}" CRS=true LOCAL_NODE=exadatadb01
3.1 The spfile was not being used by the ASM instances.
ASMCMD> spget +DATA/exadata-clu1/ASMPARAMETERFILE/registry.253.909449003
# If the command above doesn't return anything, which means the spfile is not being used. We need to set the spfile, restart the cluster if needed.
ASMCMD> DATA/exadata-clu1/ASMPARAMETERFILE/registry.253.909449003
3.2 The password file of ASM instances was not in ASM diskgroup, you will see the failures in prerequisites check.
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...FAILED PRCT-1011 : Failed to run "asmcmd". Detailed error: ASMCMD-8001: diskgroup 'u01' does not exist or is not mounted
[oracle@exadatadb01 grid]$ asmcmd pwcopy /u01/app/12.1.0.2/grid/dbs/orapw+ASM +DBFS_DG/orapwASM [oracle@exadatadb01 grid]$ asmcmd pwset --asm +DBFS_DG/orapwASM [oracle@exadatadb01 grid]$ asmcmd pwget --asm
3.3 The patch 21255373 had been applied to GI, while when doing the prerequisites, it always reports the failures. According to Oracle support, this can be ignored. So the binary installation will have the option "-skipPrereqs".
[oracle@exadatadb01 grid]$ ./gridSetup.sh -silent -skipPrereqs -responseFile /tmp/giresponse.rsp -J-Doracle.install.mgmtDB=false -J-Doracle.install.crs.enableRemoteGIMR=false -applyPSU /software/27475857/Database/12.2.0.1.0/12.2.0.1.180417GIRU/27850694
3.4 The postscripts was not successful as the SCAN and SCAN_LISTENERS were not up.
/u01/app/12.2.0.1/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp -silentINFO: [Jun 16, 2018 2:48:41 AM] Read: Verifying Single Client Access Name (SCAN) ...FAILED (PRVF-5056) INFO: [Jun 16, 2018 2:48:41 AM] Verifying Single Client Access Name (SCAN) ...FAILED (PRVF-5056) WARNING: [Jun 16, 2018 2:48:41 AM] Skipping line: Verifying Single Client Access Name (SCAN) ...FAILED (PRVF-5056) Need to manaully start the SCAN and SCAN_LISTENERS then run the post scripts [oracle@exadatadb01 grid]$ srvctl start scan [oracle@exadatadb01 grid]$ srvctl start scan_listener
3.5 Add the ASM instance entry into /etc/oratab.
For some reason, the ASM instance entry had been removed from /etc/oratab and never been added back. We need to add it manually to all nodes.
+ASM[1..4]:/u01/app/12.2.0.1/grid:N