3: The patching procedure

3.1 Patching the Cells (aka Storage Servers)

3.1.0 - Information

• All actions must be done as root
• Patching a cell takes around one hour and thirty minutes (it may take longer in the event of heavy I/O activity, we experienced some 3 hours per cell patching sessions on an heavy I/O loaded Exadata)
• You can connect to a cell console to check what is happening during a patch application. Please find the procedure on how to connect to an ILOM console. Once connected, you will see everything that is happening on the server console like the reboot sequence, etc...:

[root@myclusterdb01 dbserver_patch_5.170131]# ssh root@myclustercel01-ilom
  Password:
  Oracle(R) Integrated Lights Out Manager
  Version 3.1.2.20.c r86871
  Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
  > start /sp/console
  Are you sure you want to start /SP/console (y/n)? y
  Serial console started. To stop, type ESC (
 

3.1.1 - Check the Version of Each Cell Before Patching

[root@myclusterdb01 ~]# dcli -g ~/cell_group -l root imageinfo -ver
 myclustercel01: 12.1.2.3.3.160119
 myclustercel02: 12.1.2.3.3.160119
 myclustercel03: 12.1.2.3.3.160119
 myclustercel04: 12.1.2.3.3.160119
 myclustercel05: 12.1.2.3.3.160119
 myclustercel06: 12.1.2.3.3.160119
 [root@myclusterdb01 ~]#
 

3.1.2 - Apply the Patch

• You may use screen instead of nohup if it is installed
• You can avoid the -patch_check_prereq step as it should has already been done previously, but I personally like to do it right before the patch to be absolutely sure.
• You can also use the -smtp_to and the -smtp_from options to receive email notifications: -smtp_from "dba@pythian.com" -smtp_to "myteam@pythian.com dba@myclient.com"
• Ensure you are connected on the database server node 1 (myclusterdb01)

[root@myclusterdb01 ~]# cd /patches/OCT2016_bundle_patch/24436624/Infrastructure/12.1.2.3.3/ExadataStorageServer_InfiniBandSwitch/patch_12.1.2.3.3.161013/
 [root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -reset_force
 [root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -cleanup
 [root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -patch_check_prereq -rolling
 [root@myclusterdb01 ~]# nohup ./patchmgr -cells ~/cell_group -patch -rolling &
 [root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -cleanup

Non-Rolling Manner

• • Stop the clusterware

[root@myclusterdb01 ~]#crsctl stop cluster -all
 [root@myclusterdb01 ~]#crsctl stop crs
 [root@myclusterdb01 ~]#crsctl check crs
 -- If the cluster is not stopped properly at this step, use the -f option : crsctl stop crs -f
 

• • Stop the cells

[root@myclustercel01 ~]# cellcli -e 'alter cell shutdown services all'
 

[root@myclusterdb01 ~]# dcli -g ~/cell_group -l root "cellcli -e alter cell shutdown services all"
 

• • Apply the patch

[root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -reset_force
 [root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -cleanup
 [root@myclusterdb01 ~]# nohup ./patchmgr -cells ~/cell_group -patch &
 [root@myclusterdb01 ~]# ./patchmgr -cells ~/cell_group -cleanup
 

3.1.3 - Check the Version of Each Cell After the Patch

[root@myclusterdb01 ~]# dcli -g ~/cell_group -l root imageinfo -ver
 myclustercel01: 12.1.2.3.3.161013
 myclustercel02: 12.1.2.3.3.161013
 myclustercel03: 12.1.2.3.3.161013
 myclustercel04: 12.1.2.3.3.161013
 myclustercel05: 12.1.2.3.3.161013
 myclustercel06: 12.1.2.3.3.161013
 [root@myclusterdb01 ~]#
 

3.2 : Patching the IB Switches

3.2.0 - Information

• Patching an IB Swicth takes around 45 minutes
• All steps have to be executed as root
• It is a 100% online operation
• I've become accustomed to using the database node 1 (myclusterdb01) to patch the IB Switches, which is why I have deployed the root SSH keys from the DB node 1 to the IB Switches in the pre-requisites section
• Nothing tricky here, we have never faced any issue.
• Please find a procedure to create the ib_group file.

3.2.1 - Check the Version of Each IB Switch Before the Patch

[root@myclusterdb01 ~]# dcli -g ~/ib_group -l root version | grep "version"
 myclustersw-ib2: SUN DCS 36p version: 2.1.3-4
 myclustersw-ib3: SUN DCS 36p version: 2.1.3-4
 [root@myclusterdb01 ~]#
 

3.2.2 - Apply the Patch

• You can use screen instead of nohup if it is installed on your system
• Be sure to be connected to the myclusterdb01 server

[root@myclusterdb01 ~]# cd /patches/OCT2016_bundle_patch/24436624/Infrastructure/12.1.2.3.3/ExadataStorageServer_InfiniBandSwitch/patch_12.1.2.3.3.161013/
 [root@myclusterdb01 ~]# nohup ./patchmgr -ibswitches ~/ib_group -upgrade &
 

3.2.3 - Check the Version of Each IB Switch After the Patch

[root@myclusterdb01 ~]# dcli -g ~/ib_group -l root version | grep "version"
 myclustersw-ib2: SUN DCS 36p version: 2.1.8-1
 myclustersw-ib3: SUN DCS 36p version: 2.1.8-1
 [root@myclusterdb01 ~]#
 

3.3 Patching the DB nodes (aka Compute Nodes)

3.3.0 - Information

• • All actions must be done as root
  • Patching a database node takes around one hour
  • It is not possible to start the patch from a database node that will be patched (which makes sense). The official way to apply this patch in a rolling manner is to:
    • Start the patch from the database node 1 to patch all the other nodes
    • Once done, copy patchmgr and the ISO file to an already patched node and then start the patch to patch the remaining node (node 1 in my example)

• I use /tmp to save patchmgr and the ISO on the cell node 1 as /tmp exists on 100% of the Unix boxes and ensure that I can write in it. An important thing to know here is that /tmp on the cells is regularly purged as described in this documentation. The dbnodeupdate.zip file could then be deleted by this purge mechanism if there is too much time between the copy of it and when you use it and then you won't be able to launch patchmgr as dbnodeupdate.zip is mandatory. There are few workarounds to that though:
  • Copy patchmgr and the ISO file just before you apply the patch (this is the solution I use)
  • Copy patchmgr and the ISO file outside of /tmp.
  • The directories with SAVE in the name are ignored, then you could create a /tmp/SAVE directory to put patchmgr and the ISO file in

3.3.1 - Check the Image Versions Before the Patch

[root@myclusterdb01 ~]# dcli -g ~/dbs_group -l root imageinfo -ver
 myclusterdb01: 12.1.2.2.1.160119
 myclusterdb02: 12.1.2.2.1.160119
 myclusterdb03: 12.1.2.2.1.160119
 myclusterdb04: 12.1.2.2.1.160119
 [root@myclusterdb01 ~]#
 

3.3.2 - Check Which Instance is Up and On Which Node

3.3.3 - Apply the Patch

• We first need to umount the NFS on each DB node, this is a pre-requisite of the patch You can generate all the umount commands with this command : "df -t nfs | awk '{if ($NF ~ /^\//){print "umount " $NF}}'"; You can also do it automatically by adding a "| bash" at the end like that "df -t nfs | awk '{if ($NF ~ /^\//){print "umount " $NF}}' | bash"
• You may use screen instead of nohup if it is installed
• You can avoid the -patch_check_prereq step as it should has already been done previously but I personnally like to do it right before the patch to be 100% sure.
• Be sure to be connected to the cell node 1 (myclustercel01)

Copy patchmgr and the ISO

[root@myclusterdb01 ~]# scp /patches/OCT2016_bundle_patch/24436624/Infrastructure/SoftwareMaintenanceTools/DBServerPatch/5.161014/p21634633_121233_Linux-x86-64.zip root@myclustercel01:/tmp/. # This is patchmgr
 [root@myclusterdb01 ~]# scp /patches/OCT2016_bundle_patch/24436624/Infrastructure/12.1.2.3.3/ExadataDatabaseServer_OL6/p24669306_121233_Linux-x86-64.zip root@myclustercel01:/tmp/. # This is the ISO file, do NOT unzip it
 [root@myclusterdb01 ~]# ssh root@myclustercel01
 [root@myclustercel01 ~]# cd /tmp
 [root@myclustercel01 ~]# nohup unzip p21634633_121233_Linux-x86-64.zip &
 

Rolling Manner

[root@myclustercel01 ~]# cd /tmp/dbserver_patch_5.161014
 [root@myclustercel01 dbserver_patch_5.161110]# ./patchmgr -dbnodes ~/dbs_group -precheck -iso_repo /tmp/p24669306_121233_Linux-x86-64.zip -target_version 12.1.2.3.3.161013
 [root@myclustercel01 dbserver_patch_5.161110]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /tmp/p24669306_121233_Linux-x86-64.zip -target_version 12.1.2.3.3.161013 -rolling &
 

Non-Rolling Manner

[root@myclustercel01 ~]# cd /tmp/dbserver_patch_5.161014
 [root@myclustercel01 dbserver_patch_5.161110]# ./patchmgr -dbnodes ~/dbs_group -precheck -iso_repo /tmp/p24669306_121233_Linux-x86-64.zip -target_version 12.1.2.3.3.161013
 [root@myclustercel01 dbserver_patch_5.161110]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /tmp/p24669306_121233_Linux-x86-64.zip -target_version 12.1.2.3.3.161013 &
 

3.3.4 - Check the Image Version on Each Node

[root@myclustercel01 ~]# dcli -g ~/dbs_group -l root imageinfo -ver
 myclusterdb01: 12.1.2.3.3.161013
 myclusterdb02: 12.1.2.3.3.161013
 myclusterdb03: 12.1.2.3.3.161013
 myclusterdb04: 12.1.2.3.3.161013
 [root@myclusterdb01 ~]#
 

3.3.5 - Check the Status of Each Instance on Each Node