How to install a brand new Exadata X5 (Part 1)

Fred Denis

August 17, 2016

Tags: Technical Track, Oracle 18C, 18C Grid Installation

When you buy an Exadata machine, you can ask Oracle to proceed with the installation or let some brilliant DBAs like the one at Pythian to do it :) I will then present in this blog how to install a brand new Exadata X5; this procedure comes from the real life as it has been applied on three X5-2 half rack that are now on production. As the procedure is quite long, I have split it in two distinct parts :

Part 1 (this one) is more about the settings that are needed to proceed with the installation
Part 2 is about the installation procedure itself

Anonymization

I have obviously anonymized the procedures, a little word here on the conventions I used here :

I have replaced the name of the Exadata cluster by "mycluster"
"a_server" is a hostname outside the Exadata I used to reach it (a jump server, an administration server for example)
"a_user" is my own non privileged user I use to reach the Exadata servers from this "jump server" (I could have wrote "fdenis" instead)
I show the client specific IP like this one "<IP_of_an_IB_Switch>"
Sometimes the customer specific IPs appear like that "10.XX.YY.ZZ" (no secret here, 10.x.x.x being the class A private network)
All the clear text IPs are the Exadata defaults

Please let me know in the comments if I missed one or if something is not clear.

What is already configured

When Oracle delivers an Exadata, they take care of the cabling and the first boot. During this first boot, few things are configured :

The Ethernet Switch
The InfiniBand Switches (IB Switches)
The PDUs
An access to the ILOM of the first database server

What is needed to go further

<client>-<cluster>.xml

The DBA needs some customer specific information before heading on to the network configuration and the software installation of the new Exadata machine such as the hostnames , the IPs address of each components, the default gateway , etc… Since Exadata X2, Oracle is kind enough to provide an easy way to do that with a software named Oracle Exadata Deployment Assistant (OEDA). The client then download OEDA on his laptop, fill all the needed information and then generate an XML file (named <client>-<cluster>.xml ) that contains all the required information for the DBA to proceed with the installation. This file is vital as all the installation procedure is based on it.

OEDA for Linux

OEDA is not only used to fill the XML file, it is also used to proceed with the set up (and as OEDA version can move pretty quickly, it is NOT provided with the Exadata machine), we then need to upload on the first database server the following components :

OEDA for Linux
The <client>-<cluster>.xml file
The databasemachine.xml file is nice to have as well as we can then use it to make an Exadata Status Script for example

There’re 2 ways described in the official documentation to do that ; usually, the client plug an USB key and copy the OEDA zip file and the XML configuration file let’s say in /root/config_files .

Exadata set up

What’s in it ?

First, let’s have a look at what’s in this new Exadata ! we can easily know what is in the box by using the ibhosts command from any part of the IB network (here, I did it from an IB switch as there's not much server reachable at this part of the installation) https://gist.github.com/freddenis/9134c435be8c8ed62a9f2a0673915008 Here, we can see that we have 5 cells (5 storage servers) :

node1 elasticNode 172.16.2.37
node2 elasticNode 172.16.2.38
node3 elasticNode 172.16.2.39
node4 elasticNode 172.16.2.40
node6 elasticNode 172.16.2.42

And 4 database servers :

node8 elasticNode 172.16.2.44
node9 elasticNode 172.16.2.45
node10 elasticNode 172.16.2.46
node12 elasticNode 172.16.2.48

As no network configuration has been done yet, all components have the default factory hostnames and IP . Oracle uses the private 172.16 network for the factory IP settings starting with the first database node at 172.16.2.44. Oracle uses the Infiniband port number where the node is plugged (first database node is plugged on the port number 8) + 36 : 36 + 8 = 44 then the default database server node 1 is 172.16.2.44.

Network Configuration

Let’s then configure these IP and hostnames with the customer’s requirements in order to be able to access the database and storage servers from the customer network.

/etc/hosts of the IB Switches

There’s not much to configure on the IB Switches, just to check that the IP and the host are declared in the /etc/hosts ; add it if it’s not already done : https://gist.github.com/freddenis/68b9738c0fdd8fbeb0e9199b24c865bc

ILOM console

To start with, we have to connect to the ILOM of the first database server (the IP is accessible from the customer network). https://gist.github.com/freddenis/b12e46959e843f8588a42389bf2a16ac As we don’t need to do much things on the ILOM itself, let’s head on to the first database server using the ILOM console : https://gist.github.com/freddenis/32b61e598d8282810b3fa46ce017682f Then here press <ENTER> and login as root https://gist.github.com/freddenis/d06cc46908bce8bb3a4b60c835f8017d As we saw previously, the first database server is <node8> and a Linux OS is pre-installed https://gist.github.com/freddenis/cf6cf9c9dbe04f09ab5431e8acf0f8b2

ReclaimSpace

Exadata database servers come with Linux pre-installed but it is also possible to switch to Oracle VM which is also pre-installed on disk. In order to not waste any disk space, we can reclaim that space and any other unused space by executing the below command on every database server (list of IP for db servers can be taken from the ibhosts command like shown earlier) : https://gist.github.com/freddenis/eee5c7bc7bdde289c8d415bf190c34fb

OneCommand

Create a OneCommand directory, copy and unzip OEDA and the XML configuration file provided by the client in it. Note that it is not allowed to run OEDA from a root ( /opt) directory then use /u01 to unzip it. Here is the error you would face if you launch OEDA from /opt : Invoke OEDA from a non root file system. Current directory /opt/oracle.SupportTools/onecommand/linux-x64 is part of the root file system Non-root file systems with required space are: File System /u01 free space 95485 MB

https://gist.github.com/freddenis/ca720d721be4237770ffe57ffc9e15de

applyElasticConfig

We can now proceed with the network configuration (the applyElasticConfig.sh shell script is provided by OEDA and is located under /opt/oracle.SupportTools/onecommand/linux-x64) https://gist.github.com/freddenis/1ac1c7d35d253961cc37fead9997d59e This will apply all the customer’s specifications to the database and storage servers and all servers will be rebooted at the end of their configuration. Once the set-up applied, you will be able to connect to each database and storage servers from the customer network using their 10.x.y.z specific IP and their hostnames if they have already been defined in the corporate DNS.

An Issue with the Cells IP config

It happened that after the previous step, the cells were not well configured and then the cells configuration is different from the real network configuration shown by the “ ifconfig ” command. We have to correct that manually to avoid any issue later on specially when applying some patches; here are the error messages you could face if your cells are misconfigured (“ CELL-01533: Unable to validate the IP addresses from the cellinit.ora file because the IP addresses may be down or misconfigured” and "ORA-00700: soft internal error, arguments: [main_6a], [3], [IP addresses in cellinit.ora not operational], [], [], [], [], [], [], [], [], []" ). Here is the output of the ifconfig command on one cell (is pasted here only the information we need for visibility purpose) : https://gist.github.com/freddenis/d4a8d1d8dabedb1ecba50c37e0b46550 Here is a (bad) output of one cell (is pasted here only the information we need for visibility purpose) : https://gist.github.com/freddenis/9a4c15febe4ad54f7286f8c91a2d54c0 Note : “ /24 ” is the network mask, it should be set to “ /22 ” in the following step Who’s right then ? well, ibhosts doesn’t lie : https://gist.github.com/freddenis/69add9bf3847932e9dd2776f34370d16 We then need to update the configuration manually (even the hostname has to be set), restart the services : https://gist.github.com/freddenis/899fe8ac1e427c33d6affa09ae9e15ff And then a quick check of the new config : https://gist.github.com/freddenis/5e79e2b88a42793b6a4fe67e670b7dd3 Note : this procedure has to be done for every cell -- if your cells were misconfigured by the ./applyElasticConfig.sh -cf <CLIENT>-mycluster.xml command only

dbs_group, cell_group and all_group

You have probablyalready seen many Exadata documentations mentioning files named dbs_group, cell_group and all_group. But where do they come from ? Well, you, the DBA, create them:) As these files are used on our daily basis administration tasks, it is a good idea to put them in a well known directory. Here is few commands to create them quickly -- I also like to have an ib_group file with the name of my IB Switches, this will be very useful when you will patch the IB Switches. https://gist.github.com/freddenis/0e8d6b406b695708f5e019e17f265f93

root SSH equivalence

It is better to have a root SSH equivalence to ease all the installation and the administration tasks. We will choose one server (I use to choose DB server node 1 aka "myclusterdb01") and will deploy its SSH keys to all the other DB, cell nodes and the IB Switches. First we need to generate the root’s SSH environment (press < ENTER> after each question) : https://gist.github.com/freddenis/1ad1ddf040ad8e2310fb42828cdbf717 And we will use the wonderful dcli to deploy the ssh keys to all the database and storage servers from the database node 1: https://gist.github.com/freddenis/8886e89977993368d1f28bf224f605b6 Note : you will be prompted for the password of each server, don’t mess up, root user is locked after one failed attempt :) Let’s do the same for the IB Switches : https://gist.github.com/freddenis/9af3c65e5c722355300e8c2084e745ea You are now able to connect from the first DB server to any host without password. You can quickly test it with this command : "dcli -g ~/all_group -l root date" All is now ready to jump to the installation part which is detailed in the part 2 of this blog !

Insight and analysis of technology and business strategy