Oracle RAC on Azure

Microsoft Azure provides an acceptable and affordable platform for a training environment. I am an Oracle DBA, and use it to test functionality, new technologies and features of different Oracle products. Azure supplies a template for Oracle linux and it can be used to run a single database, but when we try to create an Oracle RAC, we hit two major issues. In the first, the Azure virtual network doesn't support multicast and, as result, cannot be used for interconnect. The second issue is shared storage. Azure provides shared file storage, and you can access it using SMB-2 protocol, but it isn't exactly what we need for RAC.

How we can solve or workaround those problems? I will share my experience and show how I can setup a RAC on Azure.

Azure Infrastructure: The Dual Challenge of Multicast and Shared Storage

For a two node RAC we first need to create at least two virtual machines for the cluster nodes. I've chosen Oracle Linux 6.4 from Azure Marketplace. I decided to create the machines with 2 network interfaces where one will be used for public, and another will be used for private interconnect.

Here is my blog post how to create a VM with 2 network interfaces. It may not be necessary since you can fork a virtual interface out of your only public network, but I decided to go this way and create cluster nodes with two interfaces. Here is output for the network from the first node:

[root@oradb5 network-scripts]# ifconfig  eth0 Link encap:Ethernet HWaddr 00:0D:3A:11:A3:71  inet addr:10.0.1.11 Bcast:10.0.1.255 Mask:255.255.254.0  inet6 addr: fe80::20d:3aff:fe11:a371/64 Scope:Link  UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1  RX packets:776 errors:0 dropped:0 overruns:0 frame:0  TX packets:789 errors:0 dropped:0 overruns:0 carrier:0  collisions:0 txqueuelen:1000  RX bytes:96068 (93.8 KiB) TX bytes:127715 (124.7 KiB)   eth1 Link encap:Ethernet HWaddr 00:0D:3A:11:AC:92  inet addr:10.0.2.11 Bcast:10.0.3.255 Mask:255.255.254.0  inet6 addr: fe80::20d:3aff:fe11:ac92/64 Scope:Link  UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1  RX packets:2 errors:0 dropped:0 overruns:0 frame:0  TX packets:9 errors:0 dropped:0 overruns:0 carrier:0  collisions:0 txqueuelen:1000  RX bytes:722 (722.0 b) TX bytes:1166 (1.1 KiB)  

We need to install oracle-rdbms-server-12cR1-preinstall-1.0-14.el6.x86_64 rpm. It will install all required packages and set up kernel and limits for oracle user on our boxes :

yum install oracle-rdbms-server-12cR1-preinstall-1.0-14.el6.x86_64

Implementing the Multicast Workaround

The next step is to enable multicast support on the network for interconnect. You can read how to enable the multicast support in my other blog. As result you are getting a network interface edge0 which can be used now for our private network.

Here is output of the ifconfig after crating virtual interface with support of multicast:

[root@oradb5 ~]# ifconfig edge0  Link encap:Ethernet HWaddr 9E:1A:D8:0B:94:EF  inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0  inet6 addr: fe80::9c1a:d8ff:fe0b:94ef/64 Scope:Link  UP BROADCAST RUNNING MULTICAST MTU:1400 Metric:1  RX packets:0 errors:0 dropped:0 overruns:0 frame:0  TX packets:3 errors:0 dropped:0 overruns:0 carrier:0  collisions:0 txqueuelen:500  RX bytes:0 (0.0 b) TX bytes:238 (238.0 b)  

I've used multicast tool from Oracle support document Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement (Doc ID 1212703.1). The check was successful:

[oracle@oradb5 mcasttest]$ ./mcasttest.pl -n oradb5,oradb6 -i edge0  ########### Setup for node oradb5 ##########  Checking node access 'oradb5'  Checking node login 'oradb5'  Checking/Creating Directory /tmp/mcasttest for binary on node 'oradb5'  Distributing mcast2 binary to node 'oradb5'  ########### Setup for node oradb6 ##########  Checking node access 'oradb6'  Checking node login 'oradb6'  Checking/Creating Directory /tmp/mcasttest for binary on node 'oradb6'  Distributing mcast2 binary to node 'oradb6'  ########### testing Multicast on all nodes ##########  Test for Multicast address 230.0.1.0  Nov 24 16:22:12 | Multicast Succeeded for edge0 using address 230.0.1.0:42000  Test for Multicast address 224.0.0.251  Nov 24 16:22:13 | Multicast Succeeded for edge0 using address 224.0.0.251:42001  

Architecting Shared Storage via NFS

So, we have solved the first obstacle and need to get shared storage for our RAC. We have at least a couple of options here, and I believe somebody can advise us on others. We can use NFS based shared storage, or we can use iscsi for that.

In my case I just fired another Oracle Linux VM, added couple of storage disks to it using portal, and then set up NFS server on that machine. Here is the high level description for that: We create a linux based VM on Azure using Oracle Linux 6.4 template from Marketplace. The size will be dictated by your requirements. I called the machine oradata. I've added a 20 Gb disk to the oradata machine through the Azure portal, and created a partition and filesystem on it:

[root@oradata ~]# fdisk /dev/sdc  [root@oradata ~]# mkfs.ext4 /dev/sdc1  [root@oradata ~]# mkdir /share  [root@oradata ~]# mkdir /share/oradata1  [root@oradata ~]# mount /dev/sdc1 /share/oradata1 

Installed necessary utilities and configured the NFS server:

[root@oradata ~]# yum install nfs-utils  [root@oradata ~]# cat /etc/exports  /share/oradata1 10.0.0.0/23(rw,sync,no_root_squash)  [root@oradata ~]# service nfs restart  

On your cluster nodes you need add the mountpoint for your shared storage to /etc/fstab and mount it:

[root@oradb5 ~]# cat /etc/fstab | grep nfs  oradata:/share/oradata1 /u02/oradata nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,actimeo=0,vers=3,timeo=600 0 0  [root@oradb5 ~]# mount -a  

Silent Installation of Grid Infrastructure

Now we have the required storage for OCR and Voting disks, network for public and interconnect, and can install our cluster. We need to correct /etc/hosts file on both nodes. Of course it is not acceptable for any production implementation or if you want to have more than one scan, so you can use DNS for proper installation.

We copy required software to one of the nodes, unpack it and create a response file for installation. This file can be used for silent installation:

[oracle@oradb5 ~]$ cat grid.rsp  oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v12.1.0  ORACLE_HOSTNAME=oradb5  INVENTORY_LOCATION=/u01/app/oraInventory  SELECTED_LANGUAGES=en  oracle.install.option=CRS_CONFIG  ORACLE_BASE=/u01/app/oracle  ORACLE_HOME=/u01/app/12.1.0/grid  oracle.install.crs.config.gpnp.scanName=oradb-clst-scan  ... oracle.install.crs.config.clusterNodes=oradb5:oradb5-vip,oradb6:oradb6-vip  oracle.install.crs.config.networkInterfaceList=eth0:10.0.0.0:1,eth1:10.0.2.0:3,edge0:192.168.1.0:2  oracle.install.crs.config.storageOption=FILE_SYSTEM_STORAGE  oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=/u02/oradata/voting/vdsk1,/u02/oradata/voting/vdsk2,/u02/oradata/voting/vdsk3  oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=/u02/oradata/ocr/ocrf1  

To run installation in silent mode you just need to go to your unpacked software and run:

[oracle@oradb5 grid]$ ./runInstaller -silent -responseFile /home/oracle/grid.rsp -ignoreSysPrereqs -ignorePrereq

Finalizing the RAC Deployment

Your installation will be completed and all you need to run is a couple of scripts to finish the installation. As a root user, execute the following script(s):

1. /u01/app/12.1.0/grid/root.sh (on both nodes, one by one).
2. /u01/app/12.1.0/grid/cfgtoollogs/configToolAllCommands (on the first node).

Keep in mind the configToolAllCommands should also create the management database in your cluster. If somehow it was failed you can try to recreate it using dbca in silent mode.

The RAC is created and now it can be used for application high availability or for databases tests. I would not recommend installing it as production system, but it is quite suitable for tests or to experiment if you want to verify or troubleshoot some RAC specific features.

Oracle Database Consulting Services

Ready to optimize your Oracle Database for the future?