Build an 11gR2 RAC Cluster in VirtualBox Using OVM Templates

After reviewing my blog post about running EBS OVM templates in VirtualBox, two of my teammates suggested that I work on something with potentially broader appeal. Their basic message was: "This is really cool for us EBS nerds, but what about the Core DBAs?" So how does "11gR2 RAC in an hour" sound? :-)

In this post, I'll demonstrate how to deploy the pre-built Oracle VM templates to create a two-node 11gR2 RAC cluster in Oracle VirtualBox.

Why Deploy 11gR2 RAC with OVM Templates?

There are already several high-quality "How to run RAC on your workstation" HOW-TO's out there, including the well-known RAC Attack (by Pythian's own Jeremy Schneider, and others) and Tim Hall's super-straightforward article on ORACLE-BASE. Does the internet really need another screenshot-heavy blog post about installing Oracle RAC? Maybe not, but I'm doing it anyway, because:

• The OVM templates come with the software pre-installed/patched, and scripts that configure the networking, Grid Infrastructure, and database for you. Less fiddling around reduces the possibility of error, and you still have a RAC cluster at the end!
• I claimed in my earlier blog post that it should be possible to convert other OVM templates, so it seemed like a good idea to actually test that claim.
• I wanted an excuse to play around a bit more with the command-line interface to VirtualBox.

Learning by Failure vs. Fun

Some readers might point out that installing and configuring the software is a good way to learn how things work, and that breaking and fixing things along the way helps to learn even more. I actually agree with that sentiment in general, since I'm a "learn by failing doing" kind of guy. However, Oracle is selling a line of high-end products that are supposed to take all of the hard work out of configuring RAC, so why shouldn't we have a bit of fun?

System Requirements and Prerequisites

The Ingredients: What You Will Need

• RAM. Lots of RAM. The OVM template docs specify 2GB per RAC NODE, and that is probably on the small side for any serious work. If you want to do anything else with your workstation while this is running, you will not be able to proceed without at least 6GB of RAM on the host machine. This is less resource-intensive than building your own OVM server, but it is not a lightweight endeavor.
• 80-100GB of disk space, depending on how you size your ASM disks.
• A recent version of VirtualBox. An old one might do, but I didn't test on an old version. :)
• DNS service for the SCAN interface. You might be able to get away without it, but I can't guarantee that the Oracle-supplied cluster build scripts will work if you try to fake it. Tim Hall has a great post on a minimal DNS setup for SCAN, or you can use dnsmasq to convert your local hosts file into a DNS service. I opted for dnsmasq; it's pretty cool.
• A Linux install ISO image. I used Oracle Enterprise Linux 5, Update 6, but any relatively recent OEL or RHEL install image should do the job here.
• An understanding of some basic Linux systems administration tasks and familiarity with configuring storage and network options in Virtualbox.

Important Disclaimers and Licensing

Nothing you're about to read in this post is supported by anyone. Not me, not Pythian, and certainly not Oracle. If you're thinking about using the techniques described here for any sort of production or QA deployment, please stop and question your sanity. Then call a few colleagues over to your desk and ask them to question your sanity.

Licensing Caveats

Please be mindful of your licensing and support status before working with these templates. Content from Oracle's Software Delivery Cloud is subject to a far more restrictive licensing than the more-familiar OTN development license. So far, this is just a proof-of-concept. As Darth Vader once said: "Do not be too proud of this technological terror you've constructed." :-)

HOW-TO: The Short Version

The basic steps are as follows, with details in the next section.

1. Set up your local DNS with IP addresses for both nodes in your future RAC cluster.
2. Download the "Oracle RAC 11.2.0.1.4 for x86_64 (64 bit) with Oracle Linux 5.5 " OVM templates and unzip.
3. Create a single VM in Virtualbox for the first node.
4. Convert OVM disk images to VDI format and attach them.
5. Boot the VM in rescue mode, install a non-Xen kernel, and adjust config files.
6. Clone the VM for the second node.
7. Create shared disks and attach to both VMs.
8. Boot both VMs and run the cluster configuration script.

HOW-TO: The Long Version

Step 1: DNS Configuration for your RAC Cluster

Complete details on DNS setup are beyond the scope of this post. Here are the IPs and hostnames that I will be using. I'm using two separate host-only networks (vboxnet0 and vboxnet1) for the public and private interfaces.

#RAC stuff  #Pub  192.168.56.11 thing1.local.org thing1  192.168.56.12 thing2.local.org thing2  #Priv  192.168.57.11 thing1-priv.local.org thing1-priv  192.168.57.12 thing2-priv.local.org thing2-priv  #VIP  192.168.56.21 thing1-vip.local.org thing1-vip  192.168.56.22 thing2-vip.local.org thing2-vip  #SCAN  192.168.56.31 clu-scan.local.org clu-scan  192.168.56.32 clu-scan.local.org clu-scan  192.168.56.33 clu-scan.local.org clu-scan  

Step 2: Download and Extract the Templates

Connect to Oracle's Software Delivery Cloud and download the files for "Oracle RAC 11.2.0.1.4 for x86_64 (64 bit) with Oracle Linux 5.5" (V25916-01.zip and V25917-01.zip). Unzip the files and unpack the resulting .tgz archives. This will create a directory called OVM_EL5U5_X86_64_11201RAC_PVM.

Step 3: Convert OVM Disk Images to VDI Format

Use the VBoxManage utility to convert the raw disk images (System.img and Oracle11201RAC_x86_64-xvdb.img) to .vdi files.

zathras:OVMRACTempl jpiwowar$ mkdir OVM_EL5U5_X86_64_11201RAC_PVM/Thing1  zathras:OVMRACTempl jpiwowar$ time VBoxManage convertfromraw OVM_EL5U5_X86_64_11201RAC_PVM/System.img OVM_EL5U5_X86_64_11201RAC_PVM/Thing1/RacRoot.vdi  zathras:OVMRACTempl jpiwowar$ time VBoxManage convertfromraw OVM_EL5U5_X86_64_11201RAC_PVM/Oracle11201RAC_x86_64-xvdb.img OVM_EL5U5_X86_64_11201RAC_PVM/Thing1/RacORCL.vdi 

Step 4: Create the Virtual Machine for Node 1

Configure the VM (Thing1) with Oracle Linux (64-bit), 3 NICs (2 Host-only, 1 NAT), 1 CPU, and 2GB RAM. Attach the VDI files to a SATA controller and the Linux ISO to the virtual DVD drive.

Step 5: Boot Node 1 in Rescue Mode

Enter “linux rescue” at the boot prompt. Configure the network interfaces eth0 and eth2 (NAT) using DHCP. Once at the prompt, switch to the root volume:

# chroot /mnt/sysimage # service sshd start 

Step 6: Update System Configuration Files

Update modprobe.conf to remove Xen-specific modules and adjust inittab to prevent console spawning errors.

[root@localhost ~]# vi /etc/modprobe.conf alias eth0 e1000 alias scsi_hostadapter ata_piix alias scsi_hostadapter ahci  [root@localhost ~]# perl -pi.orig -e 's/^(co)/#\1/' /etc/inittab [root@localhost ~]# rm /etc/rc3.d/S99oraclevm-template 

Step 7: Install a "Vanilla" Kernel

Since the OVM template uses a Xen kernel, we need a standard version from the Oracle public yum server.

[root@localhost ~]# wget https://public-yum.oracle.com/public-yum-el5.repo [root@localhost ~]# yum install kernel-2.6.18-194.el5 kernel-devel-2.6.18-194.el5 [root@localhost ~]# mkinitrd -v -f /boot/initrd-2.6.18-194.el5.img 2.6.18-194.el5 

Step 8: Tuning Boot Parameters for VirtualBox

Add divider=10 to the boot parameters in grub.conf. This reduces host CPU utilization and prevents the cluster configuration scripts from bogging down.

[root@localhost yum.repos.d]# perl -pi.orig -e 's/(numa=off)/\1 divider=10/' /boot/grub/grub.conf 

Now that the first node is prepared with a standard kernel and proper disk references, you're ready to clone it and begin the cluster configuration.

Oracle Database Consulting Services

Ready to optimize your Oracle Database for the future?