Oracle Silent Mode, Part 9: Remove a Node from an 11.1 RAC database

Oct 28, 2008 / By Grégory Guillou

Tags:

This 9th post will describe how to remove a node from a 11.1 RAC cluster in silent mode. It will differ from the associated documentation in that it will allow you to remove the node even if it is not available anymore. This procedure has only a few differences from the 10.2 one in the 6th post. For a complete series agenda, see below:

  1. Installation of 10.2 And 11.1 Databases
  2. Patches of 10.2 And 11.1 databases
  3. Cloning Software and databases
  4. Install a 10.2 RAC Database
  5. Add a Node to a 10.2 RAC database
  6. Remove a Node from a 10.2 RAC database
  7. Install a 11.1 RAC Database
  8. Add a Node to a 11.1 RAC database
  9. Remove a Node from a 11.1 RAC database (this post!)
  10. A ton of other stuff you should know

In what follows, we’ll remove rac-server5 from the configuration. We assume that rac-server5 is not accessible anymore. However, to cover the case when the node is still available, I’ll mark specific sections with a (*), meaning that the step should be done only if you still have access to that node; it can just be ignored otherwise.

Backup the Voting Disk and the OCR

Backup the voting disk with dd and the OCR with ocrconfig:

rac-server1$ su -
rac-server1# cd $ORA_CRS_HOME/bin
rac-server1# ./crsctl query css votedisk
rac-server1# mkdir -p /home/oracle/backup
rac-server1# chown oracle:dba /home/oracle/backup
rac-server1# dd if=/dev/sdb5                        
                of=/home/oracle/backup/votedisk.bak 
                bs=4k
rac-server1# ./ocrconfig -manualbackup
rac-server1# cp /u01/app/crs/cdata/Xluster/backup_20080707_225600.ocr 
             /home/oracle/backup/.

Remove the services from the instance you plan to delete

Update the services so that they don’t contain the instance you plan to remove, with srvctl modify service:

rac-server1$ # The command below lists the services
rac-server1$ srvctl config service -d ORCL
rac-server1$ # The command below modify the OLTP service
rac-server1$ srvctl modify service -d ORCL 
                -s OLTP -n                 
                -i "ORCL1,ORCL2,ORCL3,ORCL4"

Remove the instance from the node

With DBCA

DBCA is probably the easiest and fastest way to delete an instance from a RAC database (you can also do it manually, of course). DBCA doesn’t actually require the node-to-be-removed to be available. Obviously, if that’s not the case, the associated files—oratab, background_dump_dest—won’t be deleted. Run the command below from any of the servers:

rac-server1$ . oraenv
             ORCL
rac-server1$ dbca -silent -deleteInstance 
         -gdbName ORCL                    
         -instanceName ORCL5              
         -sysDBAPassword xxx

You need to use the SYS password or at least to know the one from a SYSDBA user.

Manually

In some situations, it may be useful to know how to delete an instance from a database manually. The steps to perform this operation are similar to the ones for 10.2 described in the 6th post of this series.

Remove ASM from the node

You cannot delete ASM with DBCA. Deleting an ASM instance, however, is straightforward. It consists in: (1) stopping the ASM instance (*); (2) removing the ASM instance from the OCR; and (3) deleting the ASM init.ora file (*). To proceed, execute the commands below:

rac-server1$ srvctl stop asm -n rac-server5

rac-server1$ srvctl remove asm -n rac-server5

rac-server1$ ssh rac-server5
rac-server5$ rm $ORACLE_HOME/dbs/init+ASM5.ora

Remove the Listener from the node

With 11.1, you don’t need to use NETCA to delete a listener. The good news is that you can now use srvctl instead of crs_unregister and thus use a supported way to remove a listener from a configuration when the node is not available anymore. To proceed run:

rac-server1$ srvctl stop listener -n rac-server5
rac-server1$ srvctl remove listener -n rac-server5

Once this is done, update all the network files from the remaining nodes and from the node you are removing(*), and remove all references to that node.

Remove the Database Software

By removing the software, there are two separate things to do: (1) Update the Oracle Inventory on all the nodes that will remain. That way you will remove all the software installer links to the server; (2) remove the software from that node (*). This second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

Oracle Universal Installer can do this from any of the remaining nodes. What you’ll declare in that specific case is that only rac-server1, rac-server2, rac-server3, and rac-server4 will still be part of the clustered installation. To update the inventories of those nodes, run:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server1$ cd $ORACLE_HOME/oui/bin
rac-server1$ ./runInstaller -silent -updateNodeList 
             ORACLE_HOME=$ORACLE_HOME               
             "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}"

Once you’ve updated the inventory, Oracle Universal Installer will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Database Software from the node to be removed (*)

One of the limits of the Oracle Universal Installer with RAC is that you can not use it for one node only. It’s probably a good thing in a way as you cannot apply a Patch Set on one node only and mess up everything. Unfortunately when it comes time that you want to remove the software from one server only, you have to work around that limitation. The way to do it is to update the inventory of that node only and let it think it is the only node of the cluster. Once you’ve done this, you’ll be able to execute the Universal Installer with various syntaxes like detachHome or deinstall. To change the inventory on the node to be removed, connect to it and run:

rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -updateNodeList 
             ORACLE_HOME=$ORACLE_HOME               
             CLUSTER_NODES="" -local

Once the inventory updated, all the runInstaller command you’ll run will touch only the local ORACLE_HOME (assuming it is not shared). Then, as described in the documentation, you can remove the ORACLE_HOME you want and withdraw it from the cluster:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -deinstall -removeallfiles 
              "REMOVE_HOMES={/u01/app/oracle/product/11.1.0/db_1}"

You can also detach the ORACLE_HOME from the inventory with the -detachHome syntax as below:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -detachHome                  
              ORACLE_HOME="/u01/app/oracle/product/11.1.0/db_1" 
              ORACLE_HOME_NAME="OraDB111Home1"

This second approach allow you to keep the ORACLE_HOME and delete its content only if or when you want:

rac-server5$ rm -rf /u01/app/oracle/product/11.1.0/db_1

If that’s the last database software installed on the server, you can delete the oratab file too:

rac-server5# rm /etc/oratab

Remove the ONS Configuration

To remove the ONS subscription from the server, you first have to identify the port that is used. With 11g, that port is stored in the OCR. Dump the OCR and locate the port in the dump file:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./ocrdump /home/oracle/backup/ocr.dmp
rac-server1$ grep -A1 "DATABASE.ONS_HOSTS.rac-server5.PORT" 
                      /home/oracle/backup/ocr.dmp

Note: If you’ve installed the 11.1.0.6 clusterware without adding the ONS configuration, the OCR may not store the remote port. If that’s the case, just skip the next step.

Then, you can remove the server and its port from the ONS remote configuration:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./racgons remove_config rac-server5:6250

Remove the NodeApps

The nodeapps includes the GSD, the ONS, and the VIP. You can simply remove them from any of the node with the srvctl remove nodeapps command:

rac-server1$ srvctl stop nodeapps -n rac-server5
rac-server1$ su -
rac-server1# ./srvctl remove nodeapps -n rac-server5

You can check that the nodeapps have been removed by querying its status with srvctl, or by checking the resources named ora.rac-server5 as below:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./crs_stat |grep "ora.rac-server5"

Remove the Clusterware Software

Removing the clusterware software is very similar from removing the Database Software—there are two separate things to do. (1) Update the Oracle Inventory on all the nodes that remain; and (2) remove the clusterware from that node (*). That second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

The syntax is identical to that of database removal, just add CRS=TRUE:

rac-server1$ export ORA_CRS_HOME=/u01/app/crs
rac-server1$ cd $ORA_CRS_HOME/oui/bin
rac-server1$ ./runInstaller -silent -updateNodeList  
     ORACLE_HOME=$ORA_CRS_HOME                       
     "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}" 
     CRS=TRUE

Once you’ve updated the inventory, Oracle Universal Install will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Clusterware Software from the node to be removed (*)

To delete the clusterware from the node you want to delete, you must first stop it (if you haven’t already):

rac-server5$ su -
rac-server5# cd /u01/app/crs/bin
rac-server5# ./crsctl stop crs
rac-server5# ./crsctl disable crs

Then, update the inventory as below:

rac-server5$ export ORA_CRS_HOME=/u01/app/crs
rac-server5$ cd $ORA_CRS_HOME/oui/bin
rac-server5$ ./runInstaller -silent -updateNodeList 
             ORACLE_HOME=$ORA_CRS_HOME              
             CLUSTER_NODES=""                       
             CRS=TRUE                               
             -local

Finally, run the Universal Installer with the deinstall and CRS=TRUE directives as below:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORA_CRS_HOME=/u01/app/crs
rac-server5$ cd $ORA_CRS_HOME/oui/bin
rac-server5$ ./runInstaller -silent -deinstall -removeallfiles 
              "REMOVE_HOMES={/u01/app/crs}"                    
              CRS=TRUE

Additional cleanup (*)

If the node you are removing is still accessible and you plan to reuse it (say, for another cluster) there is some additional cleanup to do:

  • Delete any file that remains in the Clusterware Home. For some reason, you may have to remove all the files if the -removeallfiles didn’t do its job.
  • Delete any specific entries in the oracle .profile file.
  • Delete any specific entries in the crontab.
  • Delete the oraInv.loc file and the inventory.
  • Replace the inittab with the one backed up from before the clusterware install named inittab.no_crs.
  • Delete the /var/tmp/.oracle directory.
  • Delete the Startup/Shutdown services, i.e., all the /etc/init.d/init* and /etc/rc?.d/*init.crs files with Oracle or Redhat Enterprise Linux.
  • Delete the clusterware and ocr.loc files, i.e., the /etc/oracle directory with Oracle or Redhat Enterprise Linux
  • Delete the storage-specific configuration files to prevent altering the shared storage from that node (/etc/fstab for NFS, udev for ASM, or the raw devices)

Remove the Node from the Cluster Configuration

Everything has been removed but if you connect to any of the remaining nodes and run olsnodes, you’ll see the server is always registered in the OCR:

rac-server1$ cd /u01/app/crs/bin
rac-server1$ ./olsnodes -n -i
rac-server1    1    rac-server1-priv rac-server1-vip
rac-server2    2    rac-server2-priv rac-server2-vip
rac-server3    3    rac-server3-priv rac-server3-vip
rac-server4    4    rac-server4-priv rac-server4-vip
rac-server5    5    rac-server5-priv

To remove that server from the OCR, connect as root on any of the remaining nodes and use its name and number with the rootdeletenode.sh script as below:

rac-server1$ su -
rac-server1# cd /u01/app/crs/install
rac-server1# ./rootdeletenode.sh rac-server5,5
rac-server1# exit
rac-server1$ cd /u01/app/crs/bin
rac-server1$ olsnodes -n -i

More to come

This 9th post is the last one about the fundamentals of the 10g and 11g database and silent syntaxes, though there are many more syntaxes you may want to look at. The 10th and last post will present some of the additional syntaxes. But you’ll have to wait again!

(*) If you cannot access the server you are removing, don’t run this step

One Response to “Oracle Silent Mode, Part 9: Remove a Node from an 11.1 RAC database”

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>