This fifth post assumes that you want to add a new node to your cluster and database. It describes most of the associated “silent” syntaxes. Even if you don’t leverage RAC’s ability to add or remove nodes to gain in agility, it’s still very likely you’ll come to these techniques when you want to upgrade some of your Servers or Operating Systems. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database (this post!)
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Adding a node involves adding all the components of the RAC in their order of appearance, i.e.: (1) The Clusterware and the associated resources, (2) the database software, (3) the listeners, (4) the ASM instance if necessary, and (5) the database instance. The last is the subject of this post.

For a complete reference to this procedure, see to the corresponding section of the 10.2 documentation.

Make Sure You Can Add The New Node To The Cluster

Before you start with anything else, proceed with the hardware and OS configuration. The new node must access the shared storage, the networks, the packages, parameter, users, etc. There is no difference if you add a node later or at the time of the initial install. You can use Oracle CVU and RDA as described in the corresponding section of the previous post to check that all the nodes can be part of a single cluster.

Adding a Node to the Clusterware

This step is actually pretty easy if the prerequisites are met. It’s also the one that can have the deepest impact on the cluster and the one that can end up badly. A lot of components are impacted by this addition: the voting disk, the cluster registry (OCR), and the inventories from all the nodes. Make sure you know how to revert any changes you are going to make. Make sure also that you have everything you need to revert the changes.

Note
Before you start with the node addition, make sure you’ve backed up the voting disk. There is no need to backup the OCR because it’s done automatically but it’s probably a good idea to locate the backup and make sure it contains the latest changes you’ve made.

You should also make sure that the Clusterware owner, oracle, or crs, or whatever it is, can write into the Clusterware ORACLE_HOME and the Oracle Inventory on the new server. Create the associated directory if necessary.

We’ll assume we want to add a new node rac-server5 to the cluster we’ve build in the previous post. In order to proceed, connect as the Clusterware owner on any of the existing nodes and run the set of commands below:

(more…)

This fourth post introduces the fundamental silent installation commands for a 10.2 RAC. For a complete series agenda, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database (this post!)
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

As the title suggests, this post will dig into how to (1) install the 10.2 Clusterware, (2) apply the latest Patch Set on top of it, (3) install the 10.2 database, (4) apply the latest Patch Set on top of it, and (5) create a RAC database. These operations will be performed with the Oracle Universal Installer, NETCA and DBCA in silent mode. Before you start, just in case you’re not familiar yet with Oracle Silent Installation, have a look at the first post of the series

Checking the prerequisites

Before you start the installation, make sure all the prerequisites are met. Use the 10.2 Clusterware and Real Application Clusters Installation Guide for your platform to set them up. You should also refer to Metalink Note 169706.1 for the latest updates and make sure the prerequisites for non-RAC databases are met by running the RDA HCVE module as described in the 1st post of the series.

To check that you haven’t missed anything, you should also run the Cluster Verify Utility (CVU). This utility is part of the Clusterware distribution or can be downloaded — get the latest release from here (note that you don’t have to be connected to OTN to download the latest release; you can just wget it once you’ve got its URL).

To run the latest Cluster Verify Utility, create a directory and unzip the CVU there. You can run the check as below:

(more…)

In the recent month there were several Oracle community web sites created. Well, I remember I registered on one or two but I couldn’t really keep an eye on many so I decided to wait and see which one wins. Turns out that Oracle Mix came out as a winner. Maybe it’s just my impression.

But I digress… I just wanted to make a quick note that Oracle Mix organized an interesting hybrid between call for papers and abstract judging. Anyone registered at Oracle Mix can propose a session abstract to present themselves or as an idea for others. Everyone can give their votes to the proposed sessions. At the end of the voting deadline (25th of June) Oracle will select the top sessions to be included in the Oracle Open World schedule.

So what? Well, I did send mine few days ago — Demystifying Workload Management with Oracle RAC based on my Hotsos Symposium 2008 presentation. I’m not sure how wide is the potential audience for this session — it’s far from beginners session and is specific to RAC. However, I do believe that this topic is often misunderstood and there is very good potential to spread the knowledge. So if you are coming to the Oracle Open World and interested in that topic - go ahead and vote. If not mine, there are plenty of others.

PS: I managed to do 2 (!) typos in the title and can’t edit it anymore. I don’t have any error on update but the title comes back unchanged. I have already filled in a bug report - let’s see if it gets fixed.

According to the wikipedia page,

“The rack is a medieval torture . . . which induces excruciating pain as the victim’s joints slowly dislocate.”

Per the Oracle website,

“Oracle RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide highly scalable and available database solutions for all your business applications.”

Which is more painful, you might ask? I cannot say for certain, as I have never been subjected to the torture of a medieval rack, but I have experienced some pain at the hands of the Oracle RAC. My first encounter was about five months ago when I first became an “official” DBA. Being eager to jump into solving problems in my new job (as that’s what most DBAs do, solve problems), I relished the chance to get my hands dirty and work on a “real” DBA task — a database lock.

Even though I had never been officially titled a DBA before, I was somewhat familiar with the concepts as I have been working around them for years (and still chose to join their ranks, if that tells you anything). Theoretically, I knew exactly what a database lock was, but I had no clue how to practically diagnose or kill one off.

Checking with a few knowledgeable co-workers, I was directed to a set of common database diagnostic scripts affectionately known as the “Pythian Kit”. (more…)

I guess I have only one and a half posts about COLLABORATE 08 this year. It’s a bit unfortunate that I couldn’t make the whole conference, but only last day-and-a-half. In addition, I’ve been speaking at the two lasts slots of the conference so it’s been quite busy for me.

As I mentioned already, the scheduling for my presentation got a bit screwed-up due to the last minute call to participate in the speaker panel — “To RAC or Not To RAC: What’s Best for HA”. Thanks to Dan Norris, my session was finally moved to 11:00, which is right after the panel. On the other hand, 11:00 slot is the last IOUG session of the conference. This fact, coupled with last minutes re-scheduling, brought the number of participants down. Even though there were about 30 people and the hall (Korbel 1C) was small enough not to seem empty.

Back to the RAC speaker panel, I enjoyed the discussion and especially the fact that speaker’s opinions on how wide RAC adoption should be were a little different. I took the liberty of starting the discussion with a quite provocative quote — “complexity is the enemy of availability”. Interesting that the other panel speakers seemed to disagree to it to some extent by countering it — use knowledge to fight complexity. Not that I don’t agree with it (on the contrary) but my point was rather, “why make it complex when you can keep it simple?” Some of the reasons for over-complicating systems can be found here.

(more…)

For the past few months I’ve been using a query that I refer to as “ash report - recent spike”. That’s the second thing I do when I get a call of the “the system is slow” type. The first thing I do is run “top” (or whichever alternative for the OS) and check the overall CPU usage.

The script is fully RAC-aware, and although it’s not 100% perfect, I use this imperfection to see if any particular node is doing something stupid. Although it is primarily targeted for OLTP systems, it can be useful for data warehouses as well, especially if they use the parallel option.

The query is to the database what “load” (uptime) for a Linux/Unix machine is, except it has much more detail. It is basically a summarization query of the v$active_session_history table. NOTE: you need to have the performance pack license to use it. It is not designed to be aligned, or read. The best is to leave it on just a few lines and concentrate on the results.

It has two “variables” that you can adjust: how far back to look (I use two hours), and how aggressively to look for problems (having count(*) >= 2).

An explanation of how to make sense of the results follows the query.

(more…)

I recently installed Oracle DB 10.2.0.1 RAC (two nodes) on Oracle Enterprise Linux x86_64 and faced three issues. Since I had to dig out at several places, I thought it might benefit others to have my notes all here.

Facts

1. OS : OEL 4 — 2.6.9-67.0.0.0.1.ELsmp #1 SMP Sun
2. 2 Node RAC Install
3. Clusterware / RDBMS - 10.2.0.1

Issue 1: Error while loading shared libraries: libstdc++.so.5 during root.sh run on first node.

root@linux1 oracle]# /u02/app/oracle/product/crs/root.sh
WARNING: directory '/u02/app/oracle/product' is not owned by root
WARNING: directory '/u02/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u02/app/oracle/product' is not owned by root
WARNING: directory '/u02/app/oracle' is not owned by root
/u02/app/oracle/product/crs/bin/crsctl.bin: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or
directory
Failure initializing entries in /etc/oracle/scls_scr/linux1.
[root@liux1 oracle]# chown -R root:dba /u02/app/oracle/
chown: cannot dereference `/u02/app/oracle/product/crs/lib32/libagtsh.so': No such file or
directory

On Node 2:

[root@linux2  VOTINGDISK]# /u02/app/oracle/product/crs/root.sh
WARNING: directory '/u02/app/oracle/product' is not owned by root
WARNING: directory '/u02/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Failed to upgrade Oracle Cluster Registry configuration

ldd /u02/app/oracle/product/crs/bin/crsctl.bin
libclntsh.so.10.1 => not found
libhasgen10.so => not found
libocr10.so => not found
libocrb10.so => not found
libocrutl10.so => not found
libskgxn2.so => not found
libdl.so.2 => /lib64/libdl.so.2 (0x0000003c58900000)
libm.so.6 => /lib64/tls/libm.so.6 (0x0000003028900000)
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003c59100000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003c60400000)
libstdc++.so.5 => /usr/lib64/libstdc++.so.5 (0x0000002a95577000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003c5b700000)
libcxaguard.so.5 => not found
libc.so.6 => /lib64/tls/libc.so.6 (0×0000003c58600000)
/lib64/ld-linux-x86-64.so.2 (0×0000003c58400000)

Ref: Metalink Note 400565.1 / Bug.5671963. Install compat-libstdc++-33-3.2.3-47.3.ia64.rpm and repeat the Oracle Clusterware installation.

Issue 2: PRKN-1008: Unable to load the shared library srvmhas10 during root.sh run on first node: (more…)

RDA 4.11 is out, with a couple of new features. “Oracle Database Diagnostics Collector” (ORADDC) is one of those. It allows you to easily activate all kinds of traces, dumps, or stack collections.

Once RDA 4.11 installed, run ./rda.pl -T oraddc to collect whatever you need. The tool will guide you. To learn more about its features, you can display its man page: run ./rda.pl -M oraddc or ./rda.pl -M.

This may become one of the most used RDA modules for Oracle Support Services and Oracle database administrators stuck in different situations. For now, let’s start with a more basic question: “How to leverage RDA in a RAC environment ?”

About RDA

RDA (Remote Diagnostic Agent) is one of the primary tools used by Oracle Support Services to diagnose any problem related to the Oracle stacks (from the Virtual Machine to the CRM). You can leverage this tool for your own needs and to speed up your problem resolution.

Using RDA before you install the Clusterware

Installing the Oracle Clusterware is an error-prone process, and in most cases, the errors come from unmet prerequisites, Fortunately, the CVU (Cluster Verification Utility) helps diagnose any problem in the initial setup you’ve performed. The best way to use it is probably to download the latest version and to validate the prerequisites according to the release you want to install. Once you’ve downloaded and unzipped the latest release from OTN, you can for example check if the prerequisites are met before you install Clusterware, with a command like the one below: (more…)

Have you ever heard the one about throwing hardware at a software problem?

In one of my previous blog posts, I mentioned something along the lines of—well I’ll just cut and paste . . .

In my experience, the solution to most problems (the ones the caller refers to as “it’s running slow”) are not rooted in hardware, because hardware problems generally cause things to not run at all. It always baffles me when developers and architects prescribe hardware upgrades to make things run faster, because about 80% of the performance-related problems and subsequent solutions I’ve dealt with were resolved by tuning the application.

Well yes, I know you can buy new hardware, and it’s easier. But when it comes to hardware, how many of you have a ten-node RAC cluster running Enterprise Edition with 8GB of RAM on each node, running off a massive SAN?

I’ve been on so many systems that have been running for years–poorly–the way they were, and in a week we can take them apart and have them running without a hitch. We’ve even managed to fix problems that turned out to be the business case to go from RAC back down to a single instance. How much did those customers save on licensing costs?

Back to the example at hand. I have this nifty RAC system that supports some very public and very mission-critical apps, and one day (it was Sunday night) it starts choking. We’re getting enqueues. Slowly they start climbing. Ten nodes came to a crashing halt. I have now seen a ten-node RAC cluster come to crashing halt and completely lock up.

Why, you ask? (more…)

Today is Hotsos Symposium 2008 Training Day — one full day with Tom Kyte. I haven’t registered for it so I took the chance to sleep until 10 this morning which was excellent idea considering that last night we were quite late going to bed thanks in parts to the joined demo that James Morle and Mike Erwin organized at the last presentation yesterday. I was in the James’ session and he was demonstrating how to hide latency problem with batching. I suspect that Mike, in the next hall, was showing the impact of MTU settings on cluster interconnect. The end result is that beer bottles travelled between the presentation halls and James ended up with about 3 packs of Guinness and Shiner Bock. That what kept us up longer last night.

James’ presentation itself was excellent — he explained that all performance problems can be caused by either skew or latency. You can’t normally fix skew issue so you just need to be aware and account for it. Latency can sometimes be shortened but usually insignificantly or it’s impractical (i. e. very expensive). It’s also very important to distinguish bandwidth and latency. I like his idea that the efficient way to solve latency is hiding it and there are generally two ways to do that — batching and threading. Improving bandwidth often doesn’t cause any performance improvement without taking latency into account. Very insightful talk. Thanks James.
(more…)