Cleaning up PID files when Oracle GRID/RAC upgrades and patches fail

Introduction:

Unfortunately is it not all that unusual for a patch or upgrade to Oracle Grid or RAC to fail, requiring a rollback. Frequently when that is necessary, the rollback actions then also fail. The reason for the rollback failure is often to 'dirty' PID files that Oracle does not clean up. Oracle keeps a number of PID files around to keep track of the Process ID for various processes.

Example:

Following is an example from an Oracle 11gR1 RAC node:

 [root@oravm01 ~]# find /u01/ -name \*.pid
 /u01/app/11.2.0/grid/crf/admin/run/crfmond/soravm01.pid
 /u01/app/11.2.0/grid/crf/admin/run/crflogd/loravm01.pid
 /u01/app/11.2.0/grid/gipc/init/oravm01.pid
 /u01/app/11.2.0/grid/mdns/init/oravm01.pid
 /u01/app/11.2.0/grid/gpnp/init/oravm01.pid
 /u01/app/11.2.0/grid/ctss/init/oravm01.pid
 /u01/app/11.2.0/grid/ologgerd/init/oravm01.pid
 /u01/app/11.2.0/grid/ohasd/init/oravm01.pid
 /u01/app/11.2.0/grid/evm/init/oravm01.pid
 /u01/app/11.2.0/grid/osysmond/init/oravm01.pid
 /u01/app/11.2.0/grid/log/oravm01/agent/crsd/oraagent_oracle/oraagent_oracle.pid
 /u01/app/11.2.0/grid/log/oravm01/agent/crsd/orarootagent_root/orarootagent_root.pid
 /u01/app/11.2.0/grid/log/oravm01/agent/ohasd/oraagent_oracle/oraagent_oracle.pid
 /u01/app/11.2.0/grid/log/oravm01/agent/ohasd/orarootagent_root/orarootagent_root.pid
 /u01/app/11.2.0/grid/log/oravm01/gpnpd/oravm01.pid
 

[root@oravm01 ~]# cat /u01/app/11.2.0/grid/crs/init/oravm01.pid
 4999
 [root@oravm01 ~]#
 [root@oravm01 ~]# ps -p 4999 -o cmd
 CMD
 /u01/app/11.2.0/grid/bin/crsd.bin reboot
 

PID Files

When patching and the patch process fails, a number of these PID files may remain behind, even though the processes they represent may no longer be running. When an attempt is made to rollback the patch, Oracle will not restart these processes, as it reads the PID file and believes that process is already running. Why Oracle does not do a cleanup of dead PID files is something of a mystery to me. In any case, here is a small script to rename all PID files that do not have a corresponding Process.

#!/bin/bash
 
 # chkpid.sh
 
 for pidfile in $(find /u01/ -name \*.pid)
 do
 
  pid=$(cat $pidfile)
 
 
  ps -p $pid > /dev/null
 
  ret=$?
 
 
  if [[ $ret -ne 0 ]]; then
  echo "#######################"
  echo " PID: $pid"
  echo " Pid not found for file:"
  ls -ld $pidfile
  mv $pidfile ${pidfile}.old
  fi
 done
 

Conclusion

This script has been used a number of times now when it became impossible to rollback a failed patch. This doesn't always work, but frequently it does. Why do I not just delete the files? Because I may want to verify some process ID's with some log and trace files. How often does this actually happen? I have personally seen this occur a number of times. The most recent was a 2 node RAC. The script made it possible to restart one of the nodes. The other node however required RAC reconfiguration. Even so it was a sigh of relief to get one node up immediately and ensure all was OK with the database.

Oracle Database Consulting Services

Ready to optimize your Oracle Database for the future?