Pythian Group

This fifth post assumes that you want to add a new node to your cluster and database. It describes most of the associated “silent” syntaxes. Even if you don’t leverage RAC’s ability to add or remove nodes to gain in agility, it’s still very likely you’ll come to these techniques when you want to upgrade some of your Servers or Operating Systems. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database (this post!)
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Adding a node involves adding all the components of the RAC in their order of appearance, i.e.: (1) The Clusterware and the associated resources, (2) the database software, (3) the listeners, (4) the ASM instance if necessary, and (5) the database instance. The last is the subject of this post.

For a complete reference to this procedure, see to the corresponding section of the 10.2 documentation.

Make Sure You Can Add The New Node To The Cluster

Before you start with anything else, proceed with the hardware and OS configuration. The new node must access the shared storage, the networks, the packages, parameter, users, etc. There is no difference if you add a node later or at the time of the initial install. You can use Oracle CVU and RDA as described in the corresponding section of the previous post to check that all the nodes can be part of a single cluster.

Adding a Node to the Clusterware

This step is actually pretty easy if the prerequisites are met. It’s also the one that can have the deepest impact on the cluster and the one that can end up badly. A lot of components are impacted by this addition: the voting disk, the cluster registry (OCR), and the inventories from all the nodes. Make sure you know how to revert any changes you are going to make. Make sure also that you have everything you need to revert the changes.

Note
Before you start with the node addition, make sure you’ve backed up the voting disk. There is no need to backup the OCR because it’s done automatically but it’s probably a good idea to locate the backup and make sure it contains the latest changes you’ve made.

You should also make sure that the Clusterware owner, oracle, or crs, or whatever it is, can write into the Clusterware ORACLE_HOME and the Oracle Inventory on the new server. Create the associated directory if necessary.

We’ll assume we want to add a new node rac-server5 to the cluster we’ve build in the previous post. In order to proceed, connect as the Clusterware owner on any of the existing nodes and run the set of commands below:

Read the rest of this entry . . .

This fourth post introduces the fundamental silent installation commands for a 10.2 RAC. For a complete series agenda, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database (this post!)
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

As the title suggests, this post will dig into how to (1) install the 10.2 Clusterware, (2) apply the latest Patch Set on top of it, (3) install the 10.2 database, (4) apply the latest Patch Set on top of it, and (5) create a RAC database. These operations will be performed with the Oracle Universal Installer, NETCA and DBCA in silent mode. Before you start, just in case you’re not familiar yet with Oracle Silent Installation, have a look at the first post of the series

Checking the prerequisites

Before you start the installation, make sure all the prerequisites are met. Use the 10.2 Clusterware and Real Application Clusters Installation Guide for your platform to set them up. You should also refer to Metalink Note 169706.1 for the latest updates and make sure the prerequisites for non-RAC databases are met by running the RDA HCVE module as described in the 1st post of the series.

To check that you haven’t missed anything, you should also run the Cluster Verify Utility (CVU). This utility is part of the Clusterware distribution or can be downloaded — get the latest release from here (note that you don’t have to be connected to OTN to download the latest release; you can just wget it once you’ve got its URL).

To run the latest Cluster Verify Utility, create a directory and unzip the CVU there. You can run the check as below:

Read the rest of this entry . . .

This post is the third of the series of ten posts that explore some of the Oracle Universal Installer (OUI), Network Assistant (NETCA), Database Creation Assistant (DBCA), Database Upgrade Assistant (DBUA) and other syntaxes you can use to script or speed up Oracle Installations. The complete series should look like this:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases (this post!)
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

In the first post and second post, we focused on how to leverage these tool to perform a standard installation and apply patches on top of 10.2 and 11.1 databases. This post will dig into the cloning features of both the Universal Install (OUI) and the Database Configuration Assistant (DBCA).

Foreword

The Universal Install and Database Configuration Assistant Cloning features enable you to:

• Install the database software once, apply to it all the patches you need, including Patch Set, CPU, and One-Off patches, and create a “Gold Image” of that software you’ll be able to deploy on all the servers you need as soon as they run the same Operating System.
• Create a database, apply all the scripts you need including the catupgrd.sql, the cpu.sql or any other script associated with patches. You can also run any script that will create a skeleton for installing your application, create tablespaces, create objects, or set parameters. Once done, you’ll be able to create a template from that database and to use it on any servers that run the same operating system and the same database software.

How to clone Oracle database software

You can refer to the product documentation for the whole database software cloning process:

• Oracle Universal Installer and OPatch User’s Guide 10g Release 2 (10.2), Chapter 7 Oracle Software Cloning Using Oracle Universal Installer
• Oracle Universal Installer and OPatch User’s Guide 11g Release 1 (11.1), Chapter 6 Cloning Oracle Software

The process is straightforward and consists of three different steps:

Read the rest of this entry . . .

This post is the second in a series of ten posts exploring some of the Oracle Universal Installer (OUI), Network Assistant (NETCA), Database Creation Assistant (DBCA), Database Upgrade Assistant (DBUA), and many more syntaxes you can use to script or speed up Oracle Installations. The complete series should look like:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases (this post!)
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

In the first post, you can find syntaxes to install a 10.2 or a 11.1 database, and how to apply a Patch Set on top of them. This post is way shorter and digs into a couple OPatch, DBUA, and OUI syntaxes. It explains how to apply a one-off patch, how to upgrade a database and how to uninstall a previous ORACLE_HOME.

Foreword

There are basically two ways to upgrade your Oracle Database Software to a new Patch Set level:

1. The In-Place Way: reuse the same ORACLE_HOME
2. The Out-of-Place Way necessitates that you create a new ORACLE_HOME for the new Patch Set

If the 10g OFA standard contains only the Base Release version, you will be able to perform an In-Place Upgrade. Thi approach, however, has several drawbacks:

• It requires that you stop all the components (Listeners, ASM, Instances, Database Console) during the software upgrade.
• It doesn’t leave the previous ORACLE_HOME install intact, and makes more complex the build of a rollback scenario.
• If you’ve installed a one-off patch, it is very likely the patchset will erase them, but Inventory will keep track of them.

For all those reasons, it’s safer to use a new ORACLE_HOME and so I won’t cover how to perform an In-Place update. Actually it’s not really different, so you should easily be able to build that scenario by yourself. And if you think: “Well, what the use of keeping only the major Database version in the OFA standard if you advise using a new ORACLE_HOME ?”, the answer is: “Once you’ve release the 10.2.0.3 ORACLE_HOME to use a 10.2.0.4 ORACLE_HOME, with that changed in the OFA standard, you’ll be able to use that ORACLE_HOME for the next Patch Set without reinstalling the software and the names will stay consistent!”. But enough of this foreword.

Read the rest of this entry . . .

This post is the first of a series of ten posts that will explore some of the Oracle Universal Installer (OUI), Network Assistant (NETCA), Database Creation Assistant (DBCA), Database Upgrade Assistant (DBUA), and many more syntaxes you can use to script or speed up Oracle Installations. The agenda should follow the one below:

1. Installation of 10.2 And 11.1 Databases (this post!)
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Actually, I may have to split the post #10 into ten more posts to cover all the other syntaxes you could use with Oracle Enterprise Manager, Application Server, or on Windows. Anyway, for now let’s focus on the very beginning: how to install 10.2 or 11.1 non-RAC database; how to apply the latest patch set; and how to create a instance database from a template.

Foreword

First, just because it’s on the Internet doesn’t make it true — even if it’s on the Pythian Blog. There will be a lot of syntaxes in those ten posts and even if they’ve been all tested, (1) the testing conditions are probably very different from your environment, and (2) the commands have been customized so that they appear generic. Be careful; it’s very likely that the syntaxes will be wrong for you. Test them yourself on a test environment and don’t execute them if you don’t understand what every part of them is supposed to do.

One of the reasons for these posts is that it’s kind of difficult to figure out by yourself how you should run one particular tool. The information is spread across the reference manuals, the response files, the online help, and sometimes Oracle Metalink or people that managed to make it work.

In addition, if the syntax looks similar for all the tools, they differ more than we can guess first. Let’s take some examples to illustrate that and to begin with the syntaxes:

Read the rest of this entry . . .

RDA 4.11 is out, with a couple of new features. “Oracle Database Diagnostics Collector” (ORADDC) is one of those. It allows you to easily activate all kinds of traces, dumps, or stack collections.

Once RDA 4.11 installed, run ./rda.pl -T oraddc to collect whatever you need. The tool will guide you. To learn more about its features, you can display its man page: run ./rda.pl -M oraddc or ./rda.pl -M.

This may become one of the most used RDA modules for Oracle Support Services and Oracle database administrators stuck in different situations. For now, let’s start with a more basic question: “How to leverage RDA in a RAC environment ?”

About RDA

RDA (Remote Diagnostic Agent) is one of the primary tools used by Oracle Support Services to diagnose any problem related to the Oracle stacks (from the Virtual Machine to the CRM). You can leverage this tool for your own needs and to speed up your problem resolution.

Using RDA before you install the Clusterware

Installing the Oracle Clusterware is an error-prone process, and in most cases, the errors come from unmet prerequisites, Fortunately, the CVU (Cluster Verification Utility) helps diagnose any problem in the initial setup you’ve performed. The best way to use it is probably to download the latest version and to validate the prerequisites according to the release you want to install. Once you’ve downloaded and unzipped the latest release from OTN, you can for example check if the prerequisites are met before you install Clusterware, with a command like the one below: Read the rest of this entry . . .

As you already know, I’ve spent a whole week near Redwood, California. Saying that the week has been difficult to manage is a euphemism:

• My heater engine broke at home (I didn’t even know it had an engine!), almost leaving my children in the cold for a night. Thank you, people have been so helpful that day!
• The snow storm in Canada prevented my plane from taking off from Chicago to Ottawa Saturday afternoon.
• The same snow storm prevented even the bus from leaving Boston after I decided this could have been the best, and only way to go, and found a plane to go there.
• And many more bad stories I don’t want to talk about . . .

However, the week has been more than useful. I have worked very hard, and had some very good fun too. As always, I’m amazed what people are able to do with Oracle technologies. We’ve learned a lot from the good and bad experiences of several Oracle customers. Unfortunately, there is not a lot I can tell you.

Here is something I can share anyway: one of the things I’ve liked the most is to spend some time (not enough!) with Kuassi Mensah. The guy is awesome! As a Product Manager at Oracle, he knows probably everything about JServer (the JVM in Oracle 11g), and he is one of the best guys on the subject of some of the key connection layer to access an Oracle database, including JDBC, OCI, and Web Services.

If you’re interested by any of those subjects, you should subscribe to his 360° blog and read his book. If you’re lucky and can attend the Conférence PHP Québec in Montréal this week, rush to his presentations, especially the one about the Database Resident Connection Pool.

Hey Kuassi! I hope the weather will be nice in Canada now that I’m back home. However, you may have to rename your blog “36°” for the end of the week — which is not so bad in this winter.

Like Alex G., I’ve left Ottawa and its wonderful snowy landscape this morning for, let’s say, a sunnier place.

But that’s all we have in common! I know the guy told you he is feeling some stress about his presentation but honestly, I doubt it! He has been preparing it for so long, he knows the subject so well, and he is so smart — just like so many people at Pythian, which is why I like it here so much. And to tell you the truth, I wouldn’t like to be at the Hotsos Symposium this year and have to choose between Alex and Cary Millsap.

So Alex, while you’ll be off socializing (and probably drinking more beers than you should), I will be working very hard, as always, this time with a couple of Oracle product managers.

The good news is everything went perfectly in Ottawa, Chicago, and here in San Francisco. I did not lose anything, and everything has just been on-time. The not-so-good — my hotel in Belmont seems about 800 miles away from Oracle Headquarters in Redwood City. I snapped this photo as I flew by on the highway:

When I arrived there, I had a couple of emails in my inbox; I should be able to meet the people I want to (except for Chuck and Larry, who are obviously too busy).

What I’ll be doing in the next five days is kind of a secret. Unfortunately, that’s all I’m allowed to tell you, except maybe that I know many people who would love to be in my shoes this week.

ASM is definitely one of the coolest technologies inside the Oracle Database. On the other hand, the ability of the storage arrays to provide a read/write access to a copy or a “snapshot” of its content is something we can easily leverage as an Oracle DBA. For a couple of weeks, I wanted to copy a database stored in an ASM Disk Group with one of those storage technologies and mount it on the same server; unfortunately, this is not supported even with 11.1.0.6. The good news is that I finally overcame all the obstacles to do it in a specific case. This post relates a couple of the tips I used to get to that result.

What You May Want to Know Before You Start

Several people helped me, even if most of them don’t know. Luca Canali did an awesome presentation about ASM at UKOUG. Alexandro Vargas’s Blog is a must-have in the list of your RSS feed if you work with ASM or RAC. Last but not least, Alex Fatkulin rocks when you have a problem with Oracle. I’m very lucky to work 10 metres away from him and other amazing folks at Pythian.

N.B.: What I describe below is among the worst things you can ever do with ASM. You can use it to play around, but never use it with anything other than test data. If you lose something because of me, you’ll be the only one to blame!

I’ve tested the whole process with Oracle 11.1.0.6 on Ubuntu Gutsy Gibbon, based on a previous post I made on this blog. In order to copy the ASM disks, I’ve used the dd command, but it should not make any difference with a storage array command.

Read the rest of this entry . . .

Recreating the Voting disk may not be as easy as written in Metalink. If you work with RAC, you know about Metalink Note 399482.1 : “How to recreate OCR/Voting disk accidentally deleted”. Of course, you back up the voting disk every time you change your RAC configuration, or on a regular basis. You probably played with the procedure and it worked just fine. Like you, I did all of that.

Yesterday, I had to recreate this precious file when it was lost a couple of hours after the whole software stack had been installed. It was, I guess, just before we would have setup our monitoring on the server that would have backed up the voting disk. When recreating the voting disk, I was amazed that the root.sh on the second node failed with the message below:

Failed to upgrade Oracle Cluster Registry configuration

But, what amazed me even more is that, despite several issues showing up on my Metalink Search with the same error, I wasn’t able to find the fix. At least nobody wanted to share one with me. However, it might frequently occur that you haven’t all nodes installed together, and it’s very likely that some of you will run into this if you are asked to recreate the Voting disk file. It happens when you haven’t installed all the nodes with the first Clusterware runInstaller command, but instead installed only some of them and then added some nodes with the addNodes.sh script.

How You Can Tell This is Your Problem

Read the rest of this entry . . .