Applying Oracle 11.2 April 2010 PSU for Single-Instance ASM and DBMS

May 4, 2010 / By Don Seiler

Tags: ,

When news of the April 2010 PSU for Oracle 11.2 came out, I was excited to see it, since it marked the first non-one-off patch release for the 11.2 database software. I happened to have an 11gR2 test system running on 11gR2 ASM via standalone Grid Infrastructure. I applied PSU 9352237 to the DBMS home and fired it up, only to see the folly of my ways when any ASM file operations like disk resizing (or auto-extending) failed with ORA-1653. This was due to the DBMS component now having a higher version number than the ASM component, which ASM does not allow. The Grid Infrastucture PSU would need to be applied to bring the ASM component up to snuff, but that patch (9343627) was, at that time, only “announced” with no ETA. Alas, the patch was rolled back and we continued testing without it.

Then this week I checked again and saw that PSU 9343627 was released and gave it a whirl. I was a little confused when the README seemed to contain a lot of instructions that always assumed it to be on a clustered, RAC install. My setup was a single-instance Grid Infrastructure installation just to provide ASM. I soon met problem upon problem when going through first this setup step:

# GRID_HOME/crs/install/rootcrs.pl -unlock
2010-05-03 11:40:42: Parsing the host name
2010-05-03 11:40:42: Checking for super user privileges
2010-05-03 11:40:42: User has super user privileges
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
The Oracle Clusterware stack failed to stop.
You should stop the stack with 'crsctl stop crs' and rerun the command


I decided to try the PSU anyway, thinking Oracle would certainly have noted in the README if this was a problem for single-node installations. Turns out I was wrong:

$ GRID_HOME/OPatch/opatch napply -local -oh GRID_HOME -id 9343627,9352237
Invoking OPatch 11.2.0.1.2

... snip ...

Running prerequisite checks...
Prerequisite check "CheckApplicable" failed.
The details are:

Patch 9343627:
Copy Action: Desctination File "GRID_HOME/bin/oradnssd" is not writeable.
'oracle.crs, 11.2.0.1.0': Cannot copy file from 'oradnssd' to 'GRID_HOME/bin/oradnssd'

UtilSession failed: Prerequisite check "CheckApplicable" failed.

OPatch failed with error code 73

I created an SR with Oracle support, and after reviewing my case agreed that the instructions were not right for single-instance Grid Infrastructure installations. They will be updating the README, but in the meantime I’d like to share the revised instructions that got me through two successful GI/DBMS PSU applications today.

Naming Conventions
First a few naming conventions to avoid confusion:

  • DBMS_HOME – The Oracle DBMS installation directory.
  • GRID_HOME – The Oracle Grid Infrastructure installation directory.

I’ve replaced all references to the directories in my examples with DBMS_HOME and GRID_HOME. Whenever you see those strings, substitute the actual path to your DBMS install or GRID install, whichever the case may be.

Download the PSU and OPatch 11.2
First, you’ll obviously need to download patch 9343627 that contains the PSU for your platform. In my case that’s Linux x86_64. As of this writing, I believe it’s only available for Linux x86 and x86_64. Note that patch 9343627 also includes patch 9352237. You do not have to, and should not, download that PSU separately. Everything you need is in the 9343627 download.

Second, choose a working directory to unzip that file. In my case, this is /home/oracle/software/11gR2/psu/gi, which looked like this after unzipping the PSU zip file:

$ ls
9343627  9352237  p9343627_112010_Linux-x86-64.zip

Unlike normal one-off patches, where you would descend into the individual patch-number-named directories, you’ll stay at this level.

Note also that you’ll need OPatch 11.2 or higher. See MOS Doc ID 6880880 and get the latest version of OPatch for your platform. Then just unzip the file in the DBMS_HOME and GRID_HOME directories, overwriting the old OPatch directory.

Pre-Patch Steps
First I’m assuming that you have shut down all DBMS instances on this server. We will shut down ASM later in this process, so be sure to shutdown the DBMS instances now.

Now we’ll start the pre-patch steps. First we stop the database on this server, saving the state of the database configuration in /tmp/oracle_home.stat:

$ DBMS_HOME/bin/srvctl stop home -o DBMS_HOME -s /tmp/oracle_home.stat

Repeat this for any additional Oracle database home installs on the server.

Next we save the database home configuration. This only needs to be done if we’re also going to patch the DBMS home in addition to the GI home. This is done from the directory where we unzipped the PSU:

$ ./9343627/custom/server/9343627/custom/scripts/prepatch.sh -dbhome DBMS_HOME
./9343627/custom/server/9343627/custom/scripts/prepatch.sh completed successfully.

After that, we shut down ASM as one would normally do it:

$ . oraenv
+ASM
$ sqlplus / as sysasm
SQL> shutdown immediate;
SQL> quit

Then, as root, we unlock and shutdown HAS (not CRS, as the original README would have you do):

# GRID_HOME/crs/install/roothas.pl -unlock
2010-05-04 17:14:08: Checking for super user privileges
2010-05-04 17:14:08: User has super user privileges
2010-05-04 17:14:08: Parsing the host name
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost'
CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock GRID_HOME

Now we’re ready to do some patching.

Applying PSU 9343627 and 9352237 to Oracle 11.2 Grid Infrastructure
As I mentioned earlier, the PSU contains both 9343627 and 9352237, and it is advised that you apply both. I did so successfully with these commands:

$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9343627
$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9352237

For each one, you’ll go through the normal opatch [y|n] confirmation prompts, and also prompted for an email address. I entered in my MOS login email, and left the password blank (it is optional). The rest was automatic and finished without error for both cases.

Applying PSU 9343627 and 9352237 to Oracle 11.2 DBMS
Here is something slightly different when applying the 9343627 patch to the DBMS_HOME, we specify a subdirectory after the “napply” parameter:

$ DBMS_HOME/OPatch/opatch napply 9343627/custom/server/ -oh DBMS_HOME -id 9343627

For patch 9352237, we use the regular syntax:

$ DBMS_HOME/OPatch/opatch napply -oh DBMS_HOME -id 9352237

Note that I use OPatch from DBMS_HOME when patching DBMS_HOME, and from GRID_HOME when patching GRID_HOME. I can’t say if it is actually necessary, but I like to keep things tidy. This is why I advised installing the new OPatch into both homes earlier.

As of this moment, both homes are patched, but we aren’t done yet!

Post-Patch Steps
First we run the postpatch.sh script included in patch 9343627 to reset some file permissions on the DBMS_HOME:

$ ./9343627/custom/server/9343627/custom/scripts/postpatch.sh -dbhome DBMS_HOME

Now we re-lock the GRID_HOME and restart the GI stack. As root:

# GRID_HOME/crs/install/roothas.pl -patch
2010-05-04 17:33:41: Checking for super user privileges
2010-05-04 17:33:41: User has super user privileges
2010-05-04 17:33:41: Parsing the host name
Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.

Verify that the services we need are started (again as root):

# crsctl check has
CRS-4638: Oracle High Availability Services is online
# crsctl check css
CRS-4529: Cluster Synchronization Services is online

If you don’t see CSS started, then you aren’t alone. On both of my servers, CSS was not set to auto-start from OHASD. This was fixed by running these commands as the GRID_HOME owner (oracle):

$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
$ crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"

Then, as root:

# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost'
CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.

# crsctl start has
CRS-4123: Oracle High Availability Services has been started.

# crsctl check has
CRS-4638: Oracle High Availability Services is online

# crsctl check css
CRS-4529: Cluster Synchronization Services is online

Moving on in the post-patch steps, we start the DBMS_HOME from the state file we created earlier:

$ DBMS_HOME/bin/srvctl start home -o DBMS_HOME -s /tmp/oracle_home.stat

At this point, we are done. You can start ASM and the DBMS instances. For some affirmation, run this for both GRID_HOME and DBMS_HOME (just using GRID_HOME in this example)


$ GRID_HOME/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : GRID_HOME
Central Inventory : /opt/oracle/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : GRID_HOME/oui
Log file location : GRID_HOME/cfgtoollogs/opatch/opatch2010-05-04_18-36-23PM.log

Patch history file: GRID_HOME/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : GRID_HOME/cfgtoollogs/opatch/lsinv/lsinventory2010-05-04_18-36-23PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (2) :

Patch  9352237      : applied on Tue May 04 18:22:41 EDT 2010
Unique Patch ID:  12381846
   Created on 25 Mar 2010, 00:05:17 hrs PST8PDT
   Bugs fixed:
     8661168, 8769239, 8898852, 8801119, 9054253, 8706590, 8725286, 8974548
     8778277, 8780372, 8769569, 9027691, 9454036, 9454037, 9454038, 8761974
     7705591, 8496830, 8702892, 8639114, 8723477, 8729793, 8919682, 8818983
     9001453, 8475069, 9328668, 8891929, 8798317, 8820324, 8733749, 8702535
     8565708, 9036013, 8735201, 8684517, 8870559, 8773383, 8933870, 8812705
     8405205, 8822365, 8813366, 8761260, 8790767, 8795418, 8913269, 8897784
     8760714, 8717461, 8671349, 8775569, 8898589, 8861700, 8607693, 8642202
     8780281, 9369797, 8780711, 8784929, 8834636, 9015983, 8891037, 8828328
     8570322, 8832205, 8665189, 8717031, 8685253, 8718952, 8799099, 8633358
     9032717, 9321701, 8588519, 8783738, 8796511, 8782971, 8756598, 9454385
     8856497, 8703064, 9066116, 9007102, 8721315, 8818175, 8674263, 9352237
     8753903, 8720447, 9057443, 8790561, 8733225, 9197917, 8928276, 8991997, 8837736

Patch  9343627      : applied on Tue May 04 18:18:31 EDT 2010
Unique Patch ID:  12381846
   Created on 15 Apr 2010, 11:28:38 hrs PST8PDT
   Bugs fixed:
     9343627, 9262748, 9262722

--------------------------------------------------------------------------------

OPatch succeeded.

As I said, Oracle support will be updating the README for PSU 9343627, but in the meantime I hope this guide helps you as much as it helped me. I’d like to thank my Pythian colleague Alex Gorbachev for his help in diagnosing some of the GI problems after the first broken patching, and also Esteban B. at Oracle Support for working closely with us to get a new single-node action plan.

18 Responses to “Applying Oracle 11.2 April 2010 PSU for Single-Instance ASM and DBMS”

  • Thanks Don, this has been a great help. We ran into exactly the same issues when trying to apply 11.2 April 2010 PSU to single-instance grid infrastructure on one of our key user test databases. We have had an SR open with Oracle for 5.5 days, but it was your blog that provided the solution.
    Fantastic stuff
    Patrick

  • Don Seiler says:

    Patrick, glad I could help!

  • Jay Pharis says:

    Fantastic workaround, thank you.

  • Phil says:

    That was awesome.
    But my v$version still say 11.2.0.0

    think i need to run the catbundle.sql…need to look that up.

  • Phil says:

    that got cut off
    it says 11.2.0.1.0 sorry….

  • Frank says:

    Hi Don,

    Thanks! I had been trying to get this PSU installed for 2 days and i was about to give up until i found your post.

    Just one question though, do i need to run the catbundle.pl script after the installation as the original read me requires for each database?

  • Applying Oracle 11.2 April 2010 PSU for Single-Instance ASM and DBMS « die Seilerwerks says:

    [...] leave a comment » Note: This post originally appeared at The Pythian Group blog. [...]

  • twisted illusionz says:

    can anybody give me a link to psu demo patch pls no surveys though

  • wissem says:

    Hello,

    Reading the Patch 12311357 – 11.2.0.2.2 GI Patch Set Update (Includes Database PSU 11.2.0.2.2) readme document.

    I don’t understand why should I unzip the GI PSU 11.2.0.2.2 patch in a shared location since I have installed the cluster in different Grid Infrastructure homes.

    Is it ok to unzip the patch in each cluster node and update the nodes separately?

    Extract of the read me file:

    “The patch application requires explicit user actions to run ‘opatch auto’ command on each node of Oracle clusterware. So, it is recommended that you download and unzip the GI PSU 11.2.0.2.2 patch in a shared location to be able to access it from any node in the cluster and then as the Grid home owner execute the unzip command.”

    Thank you

  • Don Seiler says:

    @wissem

    I’m afraid I can’t answer your question as I have no experience with clusters. Perhaps one of my colleagues will be able to answer this.

  • wissem says:

    Thank you. Who is able to answer me?

  • Ken says:

    Thanks, that was very helpful!

  • Paul says:

    Thanks, mate – life saver. I’ve just used your example here to complete a similar apply process for CPU patch 9706490, where Oracle’s README.txt is still detailing step for RAC installations, and not single node GI.

  • Andy Klock says:

    Looks like this blog entry is still relevant for 11.2.0.2 GI PSU 4 12827731. Thanks Don.

  • Stephen Wood says:

    Excellent post Don, Oracle have updated their readme but it’s hardly clear. Since when has the GI standalone home been referred to as a “Restart home” very confusing…

    “If this is an Oracle Restart Home, as the root user execute:

    # /crs/install/roothas.pl -unlock”

  • Janpreet says:

    Thanks Mate!! It helped me big time thus saving my important time for the same. I ran into the same issue and it was your blog that saved my day.
    Superb

  • Wei Shan says:

    Thanks mate! Life saver post!

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>