In part 1 of this series, we talked about some of the challenges of setting up Oracle RAC on a public cloud provider, and went on to order some VMs from provider Gandi, and finally configuring an NFS server for shared storage. In this post, we move on to configuring the RAC servers themselves, rac01 and rac02.
After starting up the RAC nodes in the GUI, we can log in via the SSH key we created. The first order of business is to set up networking. The eth1 and eth2 VLAN interfaces by default have no network configuration at all. We’ll set up a configuration to use a DHCP, and sending a hostname to the DHCP server so it knows which IP address to give us. This ties into the dnsmasq configuration we set up in part 1, to automatically assign IP addresses to the eth1 and eth2 private-VLAN network interfaces.
cd /etc/sysconfig/network-scripts cat > ifcfg-eth1 <<EOF DEVICE=eth1 BOOTPROTO=dhcp DHCP_HOSTNAME=rac01-pub ONBOOT=yes NAME=rac-public EOF cat > ifcfg-eth2 <<EOF DEVICE=eth2 BOOTPROTO=dhcp DHCP_HOSTNAME=rac01-priv ONBOOT=yes NAME=rac-private EOF service network restart ip address list
And if all is working properly, we should see eth1 and eth2 each assigned an IP address.
Since this server is accessible over the Internet, we need a basic firewall, just like server01:
iptables -F INPUT # Allow existing connections iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT # Loopback traffic iptables -A INPUT -i lo -j ACCEPT # RAC public network iptables -A INPUT -i eth1 -j ACCEPT # RAC private network iptables -A INPUT -i eth2 -j ACCEPT # SSH incoming iptables -A INPUT -p tcp --dport 22 -j ACCEPT # Log and reject everything else iptables -A INPUT -m limit --limit 2/minute -j LOG --log-prefix "iptables-input: " --log-level info iptables -A INPUT -j REJECT service iptables save
Fun with hostnames
For DNS resolution, we simply point ourselves to the DNS server on server01, 10.100.0.1:
cat > /etc/resolv.conf <<-EOF server 10.100.0.1 EOF
One issue I ran into with hostnames: the grid infrastructure install expects its public network to be associated with the hostname of the machine. But in the Gandi setup, the hostname is associated with the Internet-facing IP. Modifying /etc/sysconfig/network-scripts to change the hostname to rac01-pub.
perl -pi -e 's/HOSTNAME=rac01\\n/HOSTNAME=rac01-pub\\n/' /etc/sysconfig/network hostname rac01-pub
But even after rebooting the hostname is still rac01. More digging showed it to be part of Gandi’s auto-configuration scripts. But conveniently they provide a file, /etc/sysconfig/gandi, where specific configurations can be turned off. There are a few that won’t play well with RAC: the hostname as mentioned above, but also the mountpoint name: the grid infrastructure expects the same mountpoint names for both RAC nodes, so the default /srv/rac01data mountpoints won’t work. And lastly, we have our DNS server, so we don’t want the Gandi configuration to mangle our resolv.conf.
The relevant section of the /etc/sysconfig/gandi looks like this:
# set to 0 to avoid hostname automatic reconfigure CONFIG_HOSTNAME=1 # set to 0 to avoid nameserver automatic reconfigure CONFIG_NAMESERVER=1 # allow mounting the data disk to the mount point using the disk label CONFIG_ALLOW_MOUNT=1
perl -pi.orig -e 's/CONFIG_HOSTNAME=1/CONFIG_HOSTNAME=0/; s/CONFIG_NAMESERVER=0/CONFIG_NAMESERVER=1/; s/CONFIG_ALLOW_MOUNT=1/CONFIG_ALLOW_MOUNT=0/' /etc/sysconfig/gandi
Now our config won’t be clobbered on next reboot.
And speaking of the mountpoint, we need to undo what Gandi did and create our own mountpoint on /u01:
umount /srv/rac01data mkdir /u01 cat >> /etc/fstab <<-EOF LABEL=rac01data /u01 ext3 rw,nosuid,nodev,noatime 0 0 EOF mount /u01
A bit about the /etc/fstab line: Gandi created our filesystem with a filesystem label, so we can use this to locate the disk even if the actual device node /dev/xvdb changes. The mountpoint options are taken form Gandi’s defaults. Notably, “noatime” avoids doing a disk write every time a file is accessed.
Filessystems and NFS
NFS client config:
mkdir -p /srv/datadisk01/oradata /srv/datadisk01/dl cat >> /etc/fstab <<-EOF 10.100.0.1:/srv/datadisk01/oradata /srv/datadisk01/oradata nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0 10.100.0.1:/srv/datadisk01/dl /srv/datadisk01/dl nfs rw,bg,tcp 0 0 EOF yum -y install nfs-utils service rpcbind start chkconfig rpcbind on mount /srv/datadisk01/dl mount /srv/datadisk01/oradata
And although Gandi gives us a minimal amount of swap, we’ll need 4GB to make the Oracle installer happy. Since the system disk is only 3GB it has to come from the data disk.
dd if=/dev/zero of=/u01/swapfile bs=1048576 count=4096 mkswap /u01/swapfile swapon /u01/swapfile # Make it persistent echo "/u01/swapfile swap swap defaults 0 0" >> /etc/fstab
And Oracle checks for /dev/shm, just in case you want to use memory_target. Adding a config here to make it owned by Oracle to avoid ugly error messages from dbca later on. It you get a “ORA-00600: internal error code, arguments: [SKGMHASH]…” error message, it may very well be permission issues on /dev/shm. The “54321” userid is the numeric ID of the oracle user that the preinstall RPM will create later on.
echo "tmpfs /dev/shm tmpfs user=54321,group=54321,mode=755 0 0" >> /etc/fstab mount /dev/shm
Now that storage is set up, we can start looking at Oracle stuff.
Going from the official Oracle Grid Infrastructure install guide
cd wget http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64.rpm yum -y localinstall oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64.rpm
It complains about a UEK dependency. On this kind of cloud environment we can’t install custom kernels like UEK, so we do need to stay with default kernels. And we’re running CentOS anyway.
flashdbba has an excellent blog post on this subject; he listed a two-line workaround of downloading the full UEK kernel package, and installing it with the –justdb and –nodeps options. I can confirm that it works, but it requires a big UEK package download, plus results in warnings about missing dependencies every time yum is run.
So instead, we’re going to create a dummy RPM package to satisfy the dependency. It won’t have any files or scripts, but will match the name that the preinstall RPM is looking for.
To actually turn this specification into a RPM, we need the the rpm-build package installed as well:
yum -y install rpm-build mkdir -p ~/rpmbuild/SPECS cat > ~/rpmbuild/SPECS/dummy-kernel-uek.spec <<-EOF Summary: Dummy UEK kernel RPM Name: kernel-uek Version: 2.6.39 License: GPLv2 Release: 1.dummy %description Dummy RPM to satisfy oracle preinstall RPM dependencies %build echo OK %files EOF rpmbuild -bb dummy-kernel-uek.spec
And if all went well, we should see a kernel-uek package in ~/rpmbuild/RPMS/x86_64
cd ~/rpmbuild/RPMS/x86_64/ yum -y localinstall kernel-uek*.rpm
Back at the original dependency package:
cd yum -y localinstall oracle-rdbms-server-11cR1-preinstall-*
50m of dependencies. Downloading them all. Setting up the NFS server was so much less work :-)
And we might as well use sudo from the Oracle account rather than using root:
yum -y install sudo visudo
And uncommenting the line
%wheel ALL=(ALL) NOPASSWD: ALL
Adding oracle to the “wheel” group so it can run sudo:
usermod -G wheel oracle
Now that we have an Oracle user, the installer will require passwordless SSH to be set up. For now, it’s just from the Oracle user to itself:
su - oracle mkdir -p .ssh cd .ssh ssh-keygen -t rsa -b 2048 -N "" -f id_rsa cat id_rsa.pub >> authorized_keys chmod 0600 authorized_keys
Testing out the SSH, which lets us add the host key.
ssh rac01-pub exit
And setting a password for Oracle, which requires the passwd tool itself:
sudo yum -y install passwd sudo passwd oracle
Creating directories for the Oracle install:
sudo mkdir -p /u01/app sudo chown oracle:oinstall /u01/app sudo chown oracle:oinstall /srv/datadisk01/oradata
And now it’s time to repeat it all the steps from this post, this time on rac02, replacing rac01 with rac02 where appropriate.
After that, it’s time for part 3
- If you’re using Gandi hosting for anything out of the ordinary, familiarize yourself with /etc/sysconfig/gandi
- Dummy RPMs work very well to circumvent Oracle’s OS compatibility checks
Interested in working with Marc? Schedule a tech call.