Upgrade Exadata to 11.2.0.3

During last couple of month I was seeing some discussion and question in different online conferences and user groups about upgrade RAC and exadata to 11.2.0.3. The questions were mostly about upgrade procedure, timing, what can happen during the upgrade and how a system behaves after upgrade.

I’ve recently upgrade couple of exadata to 11.2.0.3 and want to share the experience. I hope this short note will help someone to make the decision, calculate estimation and prepare for maintenance. I am going to talk about upgrade from the version 11.2.0.2 BP10 to 11.2.0.3 BP2.

I. Preparation:
First, you need to read thoroughly oracle support note [ID 1373255.1] (strongly recommended as a primary guidance for the upgrade), make a general plan and calculate estimation time for every upgrade step. Most of the steps can be done in rolling mode and don’t require full downtime for the environment.

The second step is about gathering information about your current system and checking if your firmware and exadata software versions fulfill the requirements for 11.2.0.3.

There are several ways to get all necessary information from your exadata. The simplest way is to run exachk software and it will provide you all the information compiled together. Go to a database box on your exadata, set up user equivalence for user oracle to root the account on all database boxes, storage cells and infiniband switches or just keep the passwords handy and provide them when the script request them. Another way is to check the firmware/software by running on your exadata database box some scripts:

a) –will show you version for your exadata software on a cell box
# imageinfo
here is sample output:
Kernel version: 2.6.18-194.3.1.0.4.el5 #1 SMP Sat Feb 19 03:38:37 EST 2011 x86_64
Cell version: OSS_11.2.2.3.3_LINUX.X64_110616
Cell rpm version: cell-11.2.2.3.3_LINUX.X64_110616-1

Active image version: 11.2.2.3.3.110616
Active image activated: 2011-07-23 17:57:58 -0700
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

In partition rollback: Impossible

Cell boot usb partition: /dev/sdm1
Cell boot usb version: 11.2.2.3.3.110616

Inactive image version: undefined
Rollback to the inactive partitions: Impossible

b) — will show your version for your infiniband switch
/opt/oracle.SupportTools/CheckSWProfile.sh -I
here is sample output:
[INFO] SUCCESS Switch swtch-ib2 has correct software and firmware version:
SWVer: 1.3.3-2
[INFO] SUCCESS Switch swtch-ib2 has correct opensm configuration:
controlled_handover=TRUE polling_retry_number=5 routing_engine=ftree sminfo_polling_timeout=1000 sm_priority=5

[INFO] SUCCESS All switches have correct software and firmware version:
SWVer: 1.3.3-2
[INFO] SUCCESS All switches have correct opensm configuration:
controlled_handover=TRUE polling_retry_number=5 routing_engine=ftree sminfo_polling_timeout=1000 sm_priority=5 for non spine and 8 for spine switch5

c) — will show you version for your box
dmidecode -s system-product-name
here is sample output:
SUN FIRE X4270 M2 SERVER

You need to make sure that your Datacenter InfiniBand Switch 36 is running software release 1.3.3-2 or later and exadata Storage Server software is release 11.2.2.4.0 or later.

In my case I had proper firmware version for infiniband switched but I still needed to upgrade oracle exadata sofware.

I chose 11.2.2.4.2 version for exadata storage software which was latest on that moment and used it for upgrade my cell and database boxes.

You have to check if you have applied fix for ‘Synchronization problem in the IPC state’ unpublished bug 12539000. Also I needed to apply the patch 13404001 to my existing GI and RDBMS software before the upgrade. I recommend to prepare a list of all one-off patches applied to you RDBMS or GI software and check all those to make sure they have been fixed in 11.2.0.3.

II. Upgrade oracle exadata software:
1. Exadata storage software had been updated on all cell storage boxes in rolling mode. It was done online during the day without impact of any running service.
It took about 1.5 hours per cell box and completed successfully without any issues for most of them.
Only issue I hit during upgrade one of the cells was ora-600 which broken the upgrade during inactivation for grid disks.
As result the patching process had been aborted in the middle and was not able to continue on the cell box. I found a workaround for the issue.
Here is steps done to fix MS application and start patching for the cell from scratch:
a) I stopped MS services on the cell
cellcli>alter cell shutdown services MS

b) Deployed MS
cd $OSS_SCRIPTS_HOME/unix
sh setup_dynamicDeploy -D

c) Started to apply the patch from scratch
patchmgr -cells cell_group -cleanup
patchmgr -cells cell_group -patch -rolling

2. The minimal Pack for database hosts was also applied in rolling mode.
It was done without any warning and took about 1 hour per db box.
According to my observation most time was spent to upgrading ILO software to the new version.

Now we have new exadata software on cell and db boxes and able to start upgrade for Grid Infrastructure and Database homes.

Here are steps to upgrade Grid Infrustructure to 11.2.0.3:

1. First, you need to check your BP level and install the patch for bug 12539000 if you have BP11 and lower. The bug leads to an oracle fatal error during the rolling upgrade.
It can be done in the rolling mode and takes about 1 – 1.5 hours to complete patching for 4 database nodes.

2. Second perform all pre-installation steps described in oracle support note [ID 1373255.1].

3. The next step is to install the new 11.2.0.3 GI software. Oracle recommends “out of place” upgrade for 11gr2.
I was using silent installation for GI 11.2.0.3 with “UPGRADE” option. It can be done during normal business hours without any impact to the environment. The software installation will not take too much time. It took about 1 hour for me. Potentially we can install BP to the GI in advance before running rootupgrade.sh script and it can save you some time during database upgrade.
Don’t forget to relink the software with RDS after installation and don’t run rootupgrade.sh script!

4. The upgrade of the GI itself should be scheduled when the impact from the rolling upgrade would be minimal. What you need now is only to run the rootupgrade.sh on each node in rolling mode. It took for me about 10-20 minutes per node and completed without any errors.

Now we have the 11.2.0.3.0 GI and can continue with database software.

Upgrade the database software and databases to 11.2.0.3:

1. Install the new database software to a new directory and recompile it with RDS option. I used silent install with “INSTALL_DB_SWONLY” and recompiled the libs to use RDS after it.
You can verify if the database using RDS for IB by running command $ORACLE_HOME/bin/skgxpinfo (for version starting from 11.2.0.2)
The installation was simple and took about 1 hour.

2. The next step is to install the latest Bundle Patch (BP) to the GI and RDBMS software.
We can do it in roling mode for GI and the new installed software 11.2.0.3.
I didn’t have any issues with the BP and it was installed successfully and took about one hour to apply.

3. The next step is to upgrade the databases. You need to verify all parameters and run all pre-upgrade checks for the databases.
The oracle support note [ID 1373255.1] will help you to go through all steps. You can use dbua or manual upgrade for the databases and it will require downtime for upgraded databases.
It should not take more than one hour to perform upgrade and postupgrade steps on a database. The upgrade ran smoothly for me and completed successfully.

4. I hit couple of internal errors after upgrade. One was related to materialized view and I resolved it by rebuilding the mmatview.
Another was related to shared server connection and eventually it was resolved.  One more problem we hit with AWR reports. There were no completed reports after upgrade and the problem was fixed only after gathering statistics for fixed objects.

Finally you need to adjust database compatible parameter, ASM and disk group compatibility, verify all processes to make sure no one used old binaries, detach and move out old binaries to clear the space.
Good luck!