If a MySQL DBA from Pythian goes to Oracle Open World, it would be a shame not to send an Oracle bloke, so there I am — presenting a 90-minute session on the first day of the OOW 08 entitled Under the Hood of Oracle Clusterware.

I gave it during RAC Attack in Chicago and I’m pretty satisfied with how it went, so there should be no significant changes to the presentation. The session is in “User Group Forum,” thanks to RAC SIG and Dan Norris.

When the session was first added to the agenda it was misspelled as “Under the Good of Oracle Clusterware.” That’s hilarious and I thought I should have left it as is. Too late now — it’s been fixed.

I’m pretty sure that many of you will be at the OOW as well, so I’ll be glad to meet you in person. I’m getting back on Twitter slowly, so it might be a good way to track me down in SF. No guarantee I’ll keep it up to the minute if it takes too much effort, but I’ll try.

This seventh post digs into some of the silent installation commands of an 11.1 RAC. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database (this post!)
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

As for the Installation of a 10.2 RAC Database, this post shows how to (1) install the 11.1 clusterware, (2) install the 11.1 database, and (3) create a RAC database. It doesn’t explore any Patch Set upgrade since 11.1.0.7 is not out for now. Another interesting question, however, is how to upgrade the 10.2 clusterware to 11.1, since it has to be done in place.

So let’s get into it.

(more…)

This fifth post assumes that you want to add a new node to your cluster and database. It describes most of the associated “silent” syntaxes. Even if you don’t leverage RAC’s ability to add or remove nodes to gain in agility, it’s still very likely you’ll come to these techniques when you want to upgrade some of your Servers or Operating Systems. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database (this post!)
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Adding a node involves adding all the components of the RAC in their order of appearance, i.e.: (1) The Clusterware and the associated resources, (2) the database software, (3) the listeners, (4) the ASM instance if necessary, and (5) the database instance. The last is the subject of this post.

For a complete reference to this procedure, see to the corresponding section of the 10.2 documentation.

Make Sure You Can Add The New Node To The Cluster

Before you start with anything else, proceed with the hardware and OS configuration. The new node must access the shared storage, the networks, the packages, parameter, users, etc. There is no difference if you add a node later or at the time of the initial install. You can use Oracle CVU and RDA as described in the corresponding section of the previous post to check that all the nodes can be part of a single cluster.

Adding a Node to the Clusterware

This step is actually pretty easy if the prerequisites are met. It’s also the one that can have the deepest impact on the cluster and the one that can end up badly. A lot of components are impacted by this addition: the voting disk, the cluster registry (OCR), and the inventories from all the nodes. Make sure you know how to revert any changes you are going to make. Make sure also that you have everything you need to revert the changes.

Note
Before you start with the node addition, make sure you’ve backed up the voting disk. There is no need to backup the OCR because it’s done automatically but it’s probably a good idea to locate the backup and make sure it contains the latest changes you’ve made.

You should also make sure that the Clusterware owner, oracle, or crs, or whatever it is, can write into the Clusterware ORACLE_HOME and the Oracle Inventory on the new server. Create the associated directory if necessary.

We’ll assume we want to add a new node rac-server5 to the cluster we’ve build in the previous post. In order to proceed, connect as the Clusterware owner on any of the existing nodes and run the set of commands below:

(more…)

I recently installed Oracle DB 10.2.0.1 RAC (two nodes) on Oracle Enterprise Linux x86_64 and faced three issues. Since I had to dig out at several places, I thought it might benefit others to have my notes all here.

Facts

1. OS : OEL 4 — 2.6.9-67.0.0.0.1.ELsmp #1 SMP Sun
2. 2 Node RAC Install
3. Clusterware / RDBMS - 10.2.0.1

Issue 1: Error while loading shared libraries: libstdc++.so.5 during root.sh run on first node.

root@linux1 oracle]# /u02/app/oracle/product/crs/root.sh
WARNING: directory '/u02/app/oracle/product' is not owned by root
WARNING: directory '/u02/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u02/app/oracle/product' is not owned by root
WARNING: directory '/u02/app/oracle' is not owned by root
/u02/app/oracle/product/crs/bin/crsctl.bin: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or
directory
Failure initializing entries in /etc/oracle/scls_scr/linux1.
[root@liux1 oracle]# chown -R root:dba /u02/app/oracle/
chown: cannot dereference `/u02/app/oracle/product/crs/lib32/libagtsh.so': No such file or
directory

On Node 2:

[root@linux2  VOTINGDISK]# /u02/app/oracle/product/crs/root.sh
WARNING: directory '/u02/app/oracle/product' is not owned by root
WARNING: directory '/u02/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Failed to upgrade Oracle Cluster Registry configuration

ldd /u02/app/oracle/product/crs/bin/crsctl.bin
libclntsh.so.10.1 => not found
libhasgen10.so => not found
libocr10.so => not found
libocrb10.so => not found
libocrutl10.so => not found
libskgxn2.so => not found
libdl.so.2 => /lib64/libdl.so.2 (0x0000003c58900000)
libm.so.6 => /lib64/tls/libm.so.6 (0x0000003028900000)
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003c59100000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003c60400000)
libstdc++.so.5 => /usr/lib64/libstdc++.so.5 (0x0000002a95577000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c5b700000)
libcxaguard.so.5 => not found
libc.so.6 => /lib64/tls/libc.so.6 (0×0000003c58600000)
/lib64/ld-linux-x86-64.so.2 (0×0000003c58400000)

Ref: Metalink Note 400565.1 / Bug.5671963. Install compat-libstdc++-33-3.2.3-47.3.ia64.rpm and repeat the Oracle Clusterware installation.

Issue 2: PRKN-1008: Unable to load the shared library srvmhas10 during root.sh run on first node: (more…)

Recreating the Voting disk may not be as easy as written in Metalink. If you work with RAC, you know about Metalink Note 399482.1 : “How to recreate OCR/Voting disk accidentally deleted”. Of course, you back up the voting disk every time you change your RAC configuration, or on a regular basis. You probably played with the procedure and it worked just fine. Like you, I did all of that.

Yesterday, I had to recreate this precious file when it was lost a couple of hours after the whole software stack had been installed. It was, I guess, just before we would have setup our monitoring on the server that would have backed up the voting disk. When recreating the voting disk, I was amazed that the root.sh on the second node failed with the message below:

Failed to upgrade Oracle Cluster Registry configuration

But, what amazed me even more is that, despite several issues showing up on my Metalink Search with the same error, I wasn’t able to find the fix. At least nobody wanted to share one with me. However, it might frequently occur that you haven’t all nodes installed together, and it’s very likely that some of you will run into this if you are asked to recreate the Voting disk file. It happens when you haven’t installed all the nodes with the first Clusterware runInstaller command, but instead installed only some of them and then added some nodes with the addNodes.sh script.

How You Can Tell This is Your Problem

(more…)