Last year, I had very pleasant experience presenting at AUSOUG Conference 2007 in Melbourne. It was a long way from Canada but no regrets. Since I moved to Sydney more than a month ago, it’s much closer to travel and this year I will be presenting on both AUSOUG conferences — in Perth and at Gold Coast.

I arrived to Perth late Friday night and Saturday evening we had a very nice dinner with the bunch of Oracle ACE’s (and some of them ACE Directors). Local ACE’s were presented by Chris Muir, Connor McDonald and Penny Cookson. Tim Hall represented UK ACE’s while me being semi-local ACE from Sydney. Two of us were double agents — me and Connor are also members of OakTable Network. Seven is definitely a better number than 5 so my family brought number of lunch guests to 7. It’s been great to catch up with everyone before the start of the conference. Food was also very nice. Unfortunately, I had to hurry up at the end — my junior was falling asleep on the table as it was already close to midnight in Sydney timezone.
(more…)

Larry Ellison is announcing a major new feature this Wednesday at Open World. For the first time in a while, his keynote is dedicated to the “database” as opposed to the usual high level ERP/Apps/Fusion. Even the title of his keynote is catchy — “Extreme Performance”.

Oracle has been keeping the new feature a secret. Even the 11gR2 beta program had very few participants to prevent information leaking out. It’s, “Something’s coming, but I am not telling what.”

Okay, it worked on me, I’m excited about it. Let’s think what it could be. What single database feature is so major, that Larry himself will announce it during OpenWorld?

What do we know so far?

• Starting with the obvious, Larry’s keynote is “Extreme Performance”, so it’s related to performance.
• We know Kevin Closson has worked on it - he had a blog entry saying “I am working on something big” that got pulled off the web. (Here’s Google’s cache.)

Given these two point, let’s further think about it. What do we know about Kevin?

• He worked for PolyServe — a company whose main product is a cluster file system.
• He worked for Sequent on NUMA systems, which in today’s world is pretty close to cluster software with a very fast, low latency interconnect.
• He is an expert in storage systems and disk performance.
• He joined Oracle recently, possibly to work on this secret project.
• He must be really excited about it, to post anything on his blog under radio silence.

I think it’s something related to storage, something new and revolutionary about storage. But what?

We already know, from leaks on certain websites, that ASM will become a cluster filesystem which will allow storing OCR files, as well as user files, on the ASM disks.

But is this big enough? It’s definitely significant. Now you get a “free” reliable, cluster file system with Oracle. I don’t think it’s big enough though. Oracle already had OCFS and OCFS2. So it’s not something new to release a filesystem. And even if ASM becomes a true filesystem, that would not provide such a significant performance boost to warrant a keynote called “Extreme Performance”. An ASM filesystem would be a major manageability feature, not so much a performance feature.

That being ruled out, what could it be?

Recently, when setting up a new 11g database on a server with 128gb of RAM, I was setting up hugepages as usual, and thinking about how big my cache would be. It struck me that the cache will be bigger than the database for quite a while. Why do we even need the SAN/Datafiles?!

Then it hit me.

(more…)

This seventh post digs into some of the silent installation commands of an 11.1 RAC. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database (this post!)
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

As for the Installation of a 10.2 RAC Database, this post shows how to (1) install the 11.1 clusterware, (2) install the 11.1 database, and (3) create a RAC database. It doesn’t explore any Patch Set upgrade since 11.1.0.7 is not out for now. Another interesting question, however, is how to upgrade the 10.2 clusterware to 11.1, since it has to be done in place.

So let’s get into it.

(more…)

This fifth post assumes that you want to add a new node to your cluster and database. It describes most of the associated “silent” syntaxes. Even if you don’t leverage RAC’s ability to add or remove nodes to gain in agility, it’s still very likely you’ll come to these techniques when you want to upgrade some of your Servers or Operating Systems. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database (this post!)
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Adding a node involves adding all the components of the RAC in their order of appearance, i.e.: (1) The Clusterware and the associated resources, (2) the database software, (3) the listeners, (4) the ASM instance if necessary, and (5) the database instance. The last is the subject of this post.

For a complete reference to this procedure, see to the corresponding section of the 10.2 documentation.

Make Sure You Can Add The New Node To The Cluster

Before you start with anything else, proceed with the hardware and OS configuration. The new node must access the shared storage, the networks, the packages, parameter, users, etc. There is no difference if you add a node later or at the time of the initial install. You can use Oracle CVU and RDA as described in the corresponding section of the previous post to check that all the nodes can be part of a single cluster.

Adding a Node to the Clusterware

This step is actually pretty easy if the prerequisites are met. It’s also the one that can have the deepest impact on the cluster and the one that can end up badly. A lot of components are impacted by this addition: the voting disk, the cluster registry (OCR), and the inventories from all the nodes. Make sure you know how to revert any changes you are going to make. Make sure also that you have everything you need to revert the changes.

Note
Before you start with the node addition, make sure you’ve backed up the voting disk. There is no need to backup the OCR because it’s done automatically but it’s probably a good idea to locate the backup and make sure it contains the latest changes you’ve made.

You should also make sure that the Clusterware owner, oracle, or crs, or whatever it is, can write into the Clusterware ORACLE_HOME and the Oracle Inventory on the new server. Create the associated directory if necessary.

We’ll assume we want to add a new node rac-server5 to the cluster we’ve build in the previous post. In order to proceed, connect as the Clusterware owner on any of the existing nodes and run the set of commands below:

(more…)

Back in April I was at COLLABORATE 08 and delivered a presentation on 11g — Oracle 11g New Features Out of the Box, including Oracle 11g ASM features. The first ASM slide was about diskgroup compatibility, and I have a bit more to share than I said back then.

Every diskgroup in ASM has two compatibility attributes — compatible.asm and compatible.rdbms. In 10.2, the V$ASM_DISKGROUP view has a couple new columns added — COMPATIBILITY and DATABASE_COMPATIBILITY, but only in 11g did Oracle introduce the concept of diskgroup attributes and the V$ASM_ATTRIBUTE view. Thus, there are two ways to check the diskgroup attributes in 11g:

SQL> col COMPATIBILITY form a10
SQL> col DATABASE_COMPATIBILITY form a10
SQL> col NAME form a20
SQL> select group_number, name,
compatibility, database_compatibility from v$asm_diskgroup;

GROUP_NUMBER NAME                 COMPATIBIL DATABASE_C
------------ -------------------- ---------- ----------
           1 DG1                  11.1.0.0.0 11.1.0.0.0
           2 DG2                  10.1.0.0.0 10.1.0.0.0

SQL> col value form a10
SQL> select group_number, name, value from v$asm_attribute;

GROUP_NUMBER NAME                 VALUE
------------ -------------------- ----------
           1 disk_repair_time     3.6h
           1 au_size              1048576
           1 compatible.asm       11.1.0.0.0
           1 compatible.rdbms     11.1

Note that V$ASM_ATTRIBUTES is filled only when compatible.asm is set to 11.1. What’s important is that you can only change compatibility level upwards; there is no way to reset it back to the lower value. Compatibility attributes can be changed online one at a time:

(more…)

Since the introduction of ASM in Oracle 10g Release 1, every ASM administrator has been dreaming of a simple command line tool to copy files between ASM diskgroups and other filesystems. Oracle ASM 10g Release 2 added the handy asmcmd utility, but even though everyone expected a copy command there, it had not been implemented.

The only way to copy files to or from an ASM diskgroup was either to use RMAN, to configure XDB for FTP access, or use the DBMS_FILE_TRANSFER package. No wonder that the cp command is the most popular addition to asmcmd tool in Oracle ASM 11g: the hardest barrier to convincing my customers to use ASM has been the inability to access the files and copy them to the OS filesystem using the command-line copy command. Customers wanted to “feel” the files and be able to easily manipulate them.

While working on a Collaborate 08 presentation on Oracle 11g new features out-of-the-box, I was verifying new commands in Oracle ASM 11g’s asmcmd utility. It turned our that copying files from or to ASM is still a problem.

First, I tried to copy a single text file to an ASM diskgroup:

ASMCMD> cp /home/oracle/.bash_profile +dg2/test.file
source /home/oracle/.bash_profile
target +dg2/test.file
ASMCMD-08012: can not determine file type for file->’/home/oracle/.bash_profile’
ORA-15056: additional error message
ORA-17503: ksfdopn:DGGetFileAttr15 Failed to open file /home/oracle/.bash_profile
ORA-27046: file size is not a multiple of logical block size
Additional information: 1
ORA-06512: at “SYS.X$DBMS_DISKGROUP”, line 207
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)

Hmm . . . Okay. Let’s try to do it in multiples of diskgroup blocks: (more…)

ASM is definitely one of the coolest technologies inside the Oracle Database. On the other hand, the ability of the storage arrays to provide a read/write access to a copy or a “snapshot” of its content is something we can easily leverage as an Oracle DBA. For a couple of weeks, I wanted to copy a database stored in an ASM Disk Group with one of those storage technologies and mount it on the same server; unfortunately, this is not supported even with 11.1.0.6. The good news is that I finally overcame all the obstacles to do it in a specific case. This post relates a couple of the tips I used to get to that result.

What You May Want to Know Before You Start

Several people helped me, even if most of them don’t know. Luca Canali did an awesome presentation about ASM at UKOUG. Alexandro Vargas’s Blog is a must-have in the list of your RSS feed if you work with ASM or RAC. Last but not least, Alex Fatkulin rocks when you have a problem with Oracle. I’m very lucky to work 10 metres away from him and other amazing folks at Pythian.

N.B.: What I describe below is among the worst things you can ever do with ASM. You can use it to play around, but never use it with anything other than test data. If you lose something because of me, you’ll be the only one to blame!

I’ve tested the whole process with Oracle 11.1.0.6 on Ubuntu Gutsy Gibbon, based on a previous post I made on this blog. In order to copy the ASM disks, I’ve used the dd command, but it should not make any difference with a storage array command.

(more…)

I’ve recently moved to Ubuntu Linux, and this post describes my attempt to play around with Oracle ASM on Ubuntu. For this demonstration, I used Oracle 11.1.0.6 on Gutsy Gibbon. I hope it will be useful to somebody out there.

Important Notice: What I describe below is among the worst thing you can ever do with ASM. You can use it to play around but never use it with anything other than test data. If you lose something because of me, you’ll be the only one to blame !

Question #1: How do you simulate a disk from a file?

(See here for more details.)

If you have a free partition or disk to be used as an ASM disk, just skip this step. If you don’t, you can create a file with the dd command and create a device that actually loops to the file with the losetup command.

Let’s assume you’ve created a directory named /asmdisks (and you have write access to it). Run the command below to create a file named disk1 that is 3GB in size:

$ dd if=/dev/zero of=/asmdisks/disk1 bs=1024k count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 80.9113 seconds, 39.8 MB/s

Once you’ve created the file, map it to a device named loopN in /dev. You can list the used loop devices with the losetup -a command.

Once you’ve made sure the one you plan to used is free, e.g. /dev/loop1, you can map the device to the file with the following commands (you have to be root) :

# losetup /dev/loop1 /asmdisks/disk1
# losetup -a
/dev/loop1: [0802]:7438407 (/asmdisks/disk1)

Question #2: How do you create an interface to the disk that is usable by ASM?

(more…)

After weeks of not having the time I wanted for this, it’s finally done! Today, I installed RAC on Oracle Enterprise Linux 5 (OEL5), and I can tell you that there’s nothing exceptional about the process.

The only trouble I encountered had nothing directly to do with the installation. In order for the device rights to be assigned when RHEL5 or OEL5 start up, you have to create a file in /etc/udev/rules.d, but I have already spoken about that in my last post on the subject of raw devices.

Besides that, it’s all quite simple once 10g’s prerequisites are met. After an hour and a half, it was all wrapped up, at least for two nodes. I didn’t see anything revolutionary differences, but nonetheless there are some points worth mentioning.

(more…)