Oracle Silent Mode, Part 9: Remove a Node from a RAC database

Written by Pythian Marketing | Oct 28, 2008 4:00:00 AM

This 9 ^th post will describe how to remove a node from a 11.1 RAC cluster in silent mode. It will differ from the associated documentation in that it will allow you to remove the node even if it is not available anymore. This procedure has only a few differences from the 10.2 one in the 6^th post. For a complete series agenda, see below:

In what follows, we'll remove rac-server5 from the configuration. We assume that rac-server5 is not accessible anymore. However, to cover the case when the node is still available, I'll mark specific sections with a (*), meaning that the step should be done only if you still have access to that node; it can just be ignored otherwise.

Backup the Voting Disk and the OCR

Backup the voting disk with dd and the OCR with ocrconfig:

rac-server1$ su -
 rac-server1# cd $ORA_CRS_HOME/bin
 rac-server1# ./crsctl query css votedisk
 rac-server1# mkdir -p /home/oracle/backup
 rac-server1# chown oracle:dba /home/oracle/backup
 rac-server1# dd if=/dev/sdb5 \
  of=/home/oracle/backup/votedisk.bak \
  bs=4k
 rac-server1# ./ocrconfig -manualbackup
 rac-server1# cp /u01/app/crs/cdata/Xluster/backup_20080707_225600.ocr \
  /home/oracle/backup/.

Remove the services from the instance you plan to delete

Update the services so that they don't contain the instance you plan to remove, with srvctl modify service:

rac-server1$ # The command below lists the services
 rac-server1$ srvctl config service -d ORCL
 rac-server1$ # The command below modify the OLTP service
 rac-server1$ srvctl modify service -d ORCL \
  -s OLTP -n \
  -i "ORCL1,ORCL2,ORCL3,ORCL4"

Remove the instance from the node

With DBCA

DBCA is probably the easiest and fastest way to delete an instance from a RAC database (you can also do it manually, of course). DBCA doesn't actually require the node-to-be-removed to be available. Obviously, if that's not the case, the associated files--- oratab, background_dump_dest---won't be deleted. Run the command below from any of the servers:

rac-server1$ . oraenv
  ORCL
 rac-server1$ dbca -silent -deleteInstance \
  -gdbName ORCL \
  -instanceName ORCL5 \
  -sysDBAPassword xxx

You need to use the SYS password or at least to know the one from a SYSDBA user.

Manually

In some situations, it may be useful to know how to delete an instance from a database manually. The steps to perform this operation are similar to the ones for 10.2 described in the 6^th post of this series.

Remove ASM from the node

You cannot delete ASM with DBCA. Deleting an ASM instance, however, is straightforward. It consists in: (1) stopping the ASM instance (*); (2) removing the ASM instance from the OCR; and (3) deleting the ASM init.ora file (*). To proceed, execute the commands below:

rac-server1$ srvctl stop asm -n rac-server5
 
 rac-server1$ srvctl remove asm -n rac-server5
 
 rac-server1$ ssh rac-server5
 rac-server5$ rm $ORACLE_HOME/dbs/init+ASM5.ora

Remove the Listener from the node

With 11.1, you don't need to use NETCA to delete a listener. The good news is that you can now use srvctl instead of crs_unregister and thus use a supported way to remove a listener from a configuration when the node is not available anymore. To proceed run:

rac-server1$ srvctl stop listener -n rac-server5
 rac-server1$ srvctl remove listener -n rac-server5

Once this is done, update all the network files from the remaining nodes and from the node you are removing (*), and remove all references to that node.

Remove the Database Software

By removing the software, there are two separate things to do: (1) Update the Oracle Inventory on all the nodes that will remain. That way you will remove all the software installer links to the server; (2) remove the software from that node (*). This second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

Oracle Universal Installer can do this from any of the remaining nodes. What you'll declare in that specific case is that only rac-server1, rac-server2, rac-server3, and rac-server4 will still be part of the clustered installation. To update the inventories of those nodes, run:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
 rac-server1$ cd $ORACLE_HOME/oui/bin
 rac-server1$ ./runInstaller -silent -updateNodeList \
  ORACLE_HOME=$ORACLE_HOME \
  "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}"

Once you've updated the inventory, Oracle Universal Installer will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Database Software from the node to be removed (*)

One of the limits of the Oracle Universal Installer with RAC is that you can not use it for one node only. It's probably a good thing in a way as you cannot apply a Patch Set on one node only and mess up everything. Unfortunately when it comes time that you want to remove the software from one server only, you have to work around that limitation. The way to do it is to update the inventory of that node only and let it think it is the only node of the cluster. Once you've done this, you'll be able to execute the Universal Installer with various syntaxes like detachHome or deinstall. To change the inventory on the node to be removed, connect to it and run:

rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
 rac-server5$ cd $ORACLE_HOME/oui/bin
 rac-server5$ ./runInstaller -silent -updateNodeList \
  ORACLE_HOME=$ORACLE_HOME \
  CLUSTER_NODES="" -local

Once the inventory updated, all the runInstaller command you'll run will touch only the local ORACLE_HOME (assuming it is not shared). Then, as described in the documentation, you can remove the ORACLE_HOME you want and withdraw it from the cluster:

rac-server5$ cat /etc/oraInst.loc
 rac-server5$ cd /u01/app/oraInventory/ContentsXML
 rac-server5$ grep NAME inventory.xml
 rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
 rac-server5$ cd $ORACLE_HOME/oui/bin
 rac-server5$ ./runInstaller -silent -deinstall -removeallfiles \
  "REMOVE_HOMES={/u01/app/oracle/product/11.1.0/db_1}"

You can also detach the ORACLE_HOME from the inventory with the -detachHome syntax as below:

rac-server5$ cat /etc/oraInst.loc
 rac-server5$ cd /u01/app/oraInventory/ContentsXML
 rac-server5$ grep NAME inventory.xml
 rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
 rac-server5$ cd $ORACLE_HOME/oui/bin
 rac-server5$ ./runInstaller -silent -detachHome \
  ORACLE_HOME="/u01/app/oracle/product/11.1.0/db_1" \
  ORACLE_HOME_NAME="OraDB111Home1"

This second approach allow you to keep the ORACLE_HOME and delete its content only if or when you want:

rac-server5$ rm -rf /u01/app/oracle/product/11.1.0/db_1

If that's the last database software installed on the server, you can delete the oratab file too:

rac-server5# rm /etc/oratab

Remove the ONS Configuration

To remove the ONS subscription from the server, you first have to identify the port that is used. With 11g, that port is stored in the OCR. Dump the OCR and locate the port in the dump file:

rac-server1$ cd $ORA_CRS_HOME/bin
 rac-server1$ ./ocrdump /home/oracle/backup/ocr.dmp
 rac-server1$ grep -A1 "DATABASE.ONS_HOSTS.rac-server5.PORT" \
  /home/oracle/backup/ocr.dmp

Note: If you've installed the 11.1.0.6 clusterware without adding the ONS configuration, the OCR may not store the remote port. If that's the case, just skip the next step.

Then, you can remove the server and its port from the ONS remote configuration:

rac-server1$ cd $ORA_CRS_HOME/bin
 rac-server1$ ./racgons remove_config rac-server5:6250

Remove the NodeApps

The nodeapps includes the GSD, the ONS, and the VIP. You can simply remove them from any of the node with the srvctl remove nodeapps command:

rac-server1$ srvctl stop nodeapps -n rac-server5
 rac-server1$ su -
 rac-server1# ./srvctl remove nodeapps -n rac-server5

You can check that the nodeapps have been removed by querying its status with srvctl, or by checking the resources named ora.rac-server5 as below:

rac-server1$ cd $ORA_CRS_HOME/bin
 rac-server1$ ./crs_stat |grep "ora.rac-server5"

Remove the Clusterware Software

Removing the clusterware software is very similar from removing the Database Software---there are two separate things to do. (1) Update the Oracle Inventory on all the nodes that remain; and (2) remove the clusterware from that node (*). That second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

The syntax is identical to that of database removal, just add CRS=TRUE:

rac-server1$ export ORA_CRS_HOME=/u01/app/crs
 rac-server1$ cd $ORA_CRS_HOME/oui/bin
 rac-server1$ ./runInstaller -silent -updateNodeList \
  ORACLE_HOME=$ORA_CRS_HOME \
  "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}" \
  CRS=TRUE

Once you've updated the inventory, Oracle Universal Install will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Clusterware Software from the node to be removed (*)

To delete the clusterware from the node you want to delete, you must first stop it (if you haven't already):

rac-server5$ su -
 rac-server5# cd /u01/app/crs/bin
 rac-server5# ./crsctl stop crs
 rac-server5# ./crsctl disable crs

Then, update the inventory as below:

rac-server5$ export ORA_CRS_HOME=/u01/app/crs
 rac-server5$ cd $ORA_CRS_HOME/oui/bin
 rac-server5$ ./runInstaller -silent -updateNodeList \
  ORACLE_HOME=$ORA_CRS_HOME \
  CLUSTER_NODES="" \
  CRS=TRUE \
  -local

Finally, run the Universal Installer with the deinstall and CRS=TRUE directives as below:

rac-server5$ cat /etc/oraInst.loc
 rac-server5$ cd /u01/app/oraInventory/ContentsXML
 rac-server5$ grep NAME inventory.xml
 rac-server5$ export ORA_CRS_HOME=/u01/app/crs
 rac-server5$ cd $ORA_CRS_HOME/oui/bin
 rac-server5$ ./runInstaller -silent -deinstall -removeallfiles \
  "REMOVE_HOMES={/u01/app/crs}" \
  CRS=TRUE

Additional cleanup (*)

If the node you are removing is still accessible and you plan to reuse it (say, for another cluster) there is some additional cleanup to do:

Delete any file that remains in the Clusterware Home. For some reason, you may have to remove all the files if the -removeallfiles didn't do its job.
Delete any specific entries in the oracle .profile file.
Delete any specific entries in the crontab.
Delete the oraInv.loc file and the inventory.
Replace the inittab with the one backed up from before the clusterware install named inittab.no_crs.
Delete the /var/tmp/.oracle directory.
Delete the Startup/Shutdown services, i.e., all the /etc/init.d/init* and /etc/rc?.d/*init.crs files with Oracle or Redhat Enterprise Linux.
Delete the clusterware and ocr.loc files, i.e., the /etc/oracle directory with Oracle or Redhat Enterprise Linux
Delete the storage-specific configuration files to prevent altering the shared storage from that node (/etc/fstab for NFS, udev for ASM, or the raw devices)

Remove the Node from the Cluster Configuration

Everything has been removed but if you connect to any of the remaining nodes and run olsnodes, you'll see the server is always registered in the OCR:

rac-server1$ cd /u01/app/crs/bin
 rac-server1$ ./olsnodes -n -i
 rac-server1 1 rac-server1-priv rac-server1-vip
 rac-server2 2 rac-server2-priv rac-server2-vip
 rac-server3 3 rac-server3-priv rac-server3-vip
 rac-server4 4 rac-server4-priv rac-server4-vip
 rac-server5 5 rac-server5-priv

To remove that server from the OCR, connect as root on any of the remaining nodes and use its name and number with the rootdeletenode.sh script as below:

rac-server1$ su -
 rac-server1# cd /u01/app/crs/install
 rac-server1# ./rootdeletenode.sh rac-server5,5
 rac-server1# exit
 rac-server1$ cd /u01/app/crs/bin
 rac-server1$ olsnodes -n -i

More to come

This 9 ^th post is the last one about the fundamentals of the 10g and 11g database and silent syntaxes, though there are many more syntaxes you may want to look at. The 10 ^th and last post will present some of the additional syntaxes. But you'll have to wait again! (*) If you cannot access the server you are removing, don't run this step

Oracle Database Consulting Services

Ready to optimize your Oracle Database for the future?

View full post