In this post and some upcoming posts, I’m going to write more about Oracle application servers, a subject we have addressed too little on the Pythian blog.
In this post, I am addressing how to bounce a whole application server, including all tiers and databases from one location. The reason being, I have a request from a client to have the application server be bounced automatically during the weekend to release swaps and to address memory leaks. The application server on this client includes one Mid tier, one Infra tier, and one database (a metarepository database) in three different Oracle homes on two boxes.
As you know, for application server, tiers should be stopped and started up in a specific order. On startup, the sequence is like: database –> listener –> Infra tier –> Mid tier.
For shutdown, the sequence is vice versa. It is not safe to shutdown the database or Infra tier before making sure that the Mid tier is totally done.
So, in order to address client’s request, the basic plan was to have a script to shutdown Mid tier, then handshake with a tscript on the Infra tier to let it know that it is safe to shutdown the Infra tier. We would use the same approach for the Infra shutdown script and database shutdown script.
Is there a simpler way?
opmnctl, the main tool for startup/shutdown of application server components, is able to bounce the whole farm. However, first time you try to run opmnctl status @farm, you may just see the status of AS component only for single box rather than for the whole farm. Why?
For security reasons, in AS 10g, opmn requests (start, restart, and stop) to the farm work only if SSL is enabled in the opmn.xml file and a wallet file is configured. If neither SSL nor a wallet file are configured, OPMN will reject any remote process control request with HTTP code 403. Interestingly, OPMN in AS 10g is not SSL-enabled by default.
Here are the steps to enable SSL on OPMN and make it ready for a remote bounce.
- Edit
$OH/opmn/conf/opmn.xml on all tiers and add the following lines to the <notification-server> section. <ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>
- Copy the existing wallet on each box to defined location for opmn. By default the wallet file is
OH/Apache/Apache/conf/ssl.wlt/default. Copy this file to OH/opmn/conf/ssl.wlt/default.
- Run the following command to pick up the changes in each Oracle Home.
$ORACLE_HOME/dcm/bin/dcmctl updateconfig -ct opmn
- Bounce the tiers.
- Run the following statement from one location, and you should see status of all components in all tiers.
opmnctl status @farm
opmnctl stopall @farm
opmnctl startall @farm
In my case:
oracle(/home/oracle):opmnctl status @farm
Processes in Instance: infraAS10g.##################
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
HTTP_Server | HTTP_Server | 471476 | Alive
LogLoader | logloaderd | 462866 | Alive
dcm-daemon | dcm-daemon | 430186 | Alive
OC4J | OC4J_SECURITY | 1089626 | Alive
OC4J | oca | 1069158 | Alive
DSA | DSA | N/A | Down
OID | OID | 1200560 | Alive
Processes in Instance: Mid10gAS_###########################
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
HTTP_Server | HTTP_Server | 978974 | Alive
LogLoader | logloaderd | 471280 | Alive
dcm-daemon | dcm-daemon | 962758 | Alive
OC4J | home | 585772 | Alive
OC4J | OC4J_BI_Forms | 946312 | Alive
OC4J | OC4J_Portal | 790612 | Alive
WebCache | WebCache | 704698 | Alive
WebCache | WebCacheAdmin | 524378 | Alive
Discoverer | ServicesStatus | 774258 | Alive
Discoverer | PreferenceServer | 811190 | Alive
wireless | performance_server | N/A | Down
wireless | messaging_server | N/A | Down
wireless | OC4J_Wireless | N/A | Down
DSA | DSA | N/A | Down
When everything is ready, it is just a matter of having a shell script to be run on the InfraStructure tier and bouncing the whole AS environment.
Here is the script I implemented for this client. In any steps if startup/shutdown fails, it will page the person on-call for manual intervention.
### Script to recycle IAS and DB ###
### 3 Envs should be set : INSTANCE_INFRA, INSTANCE_MIDAS,META_LISTENERAS_env=$1
DB_env=$2
##################************************
##################************************
##################************************
##################************************
##################************************
. ~/$AS_env
export ENVIRONMENT=`hostname`
export cnt=0
## Check number of AS component "
echo ******************##################************************
echo `date` "=> Checking number of AS components "
echo ******************##################************************
Mid_comps=`opmnctl @instance:$INSTANCE_MIDAS status -noheaders | grep "|" |wc -l`
Inf_comps=`opmnctl @instance:$INSTANCE_INFRA status -noheaders | grep "|" | wc -l`
## Starting AS/Database Recycle ##
echo ******************##################************************
echo `date` "=>Shutting down the dcm-daemon and LogLoader first..."
echo ******************##################************************
opmnctl @farm stopproc ias-component=dcm-daemon
opmnctl @farm stopproc ias-component=LogLoader
## Shutdown Mid AS on Foo ##
echo ******************##################************************
echo `date` "=>Shutting down mid tier AS ...."
echo ******************##################************************
opmnctl @instance:$INSTANCE_MIDAS stopproc
## Wait until the AS processes actually end up in a 'DOWN' state
while opmnctl @instance:$INSTANCE_MIDAS status | grep -i "stopped" > /dev/null
do
sleep 5
((cnt=cnt+1))
if [ $cnt = 1000 ]; then
break;
fi
echo `date` "=>Waiting for the Mid Tier to go down.."
done
echo ******************##################************************
echo `date` "=>Status of mid tier components..."
echo ******************##################************************
opmnctl @instance:$INSTANCE_MIDAS status
Mid_Downcomps=`opmnctl @instance:$INSTANCE_MIDAS status -noheaders |grep "Down" |wc -l`
if [ "$Mid_Downcomps" = "$Mid_comps" ]; then
echo `date` "Mid-tier is down .... Safe to go for Infra"
else
/usr/sbin/sendmail oncall@example.com << EOF
From: $ENVIRONMENT@example.com
To: oncall@example.com
Subject: Automatic AS bounce failed, It needs manual intervention.
Message:
Manual intevention is required to fix AS bounce
EOF
exit 4
fi
## Shutdown Infra on Bar ##
cnt=1
echo ******************##################************************
echo `date` "=>Shutting down infra ...."
echo ******************##################************************
opmnctl @instance:$INSTANCE_INFRA stopproc
## Wait until the AS processes actually end up in a 'DOWN' state
while opmnctl @instance:$INSTANCE_INFRA status | grep -i "stopped" > /dev/null
do
sleep 5
((cnt=cnt+1))
if [ $cnt = 1000 ]; then
break;
fi
echo `date` "=>Waiting for the Infra Tier to go down.."
done
echo ******************##################************************
echo `date` "=>Status of infra components..."
echo ******************##################************************
opmnctl @instance:$INSTANCE_INFRA status
Inf_Downcomps=`opmnctl @instance:$INSTANCE_INFRA status -noheaders |grep "Down" |wc -l`
if [ "$Inf_Downcomps" = "$Inf_comps" ]; then
echo `date` "Infra is down .... Safe to go for Database"
else
/usr/sbin/sendmail oncall@example.com << EOF
From: $ENVIRONMENT.up@example.com
To: oncall@example.com
Subject: Automatic AS bounce failed, It needs manual intervention.
Message:
Manual intevention is required to fix AS bounce
EOF
exit 4
fi
##################************************
##################************************
##################************************
##################************************
. ~/$DB_env
echo ******************##################************************
echo `date` "=>Shutting down the listener $META_LISTENER ..."
echo ******************##################************************
lsnrctl stop $META_LISTENER
echo ******************##################************************
echo `date` "=>Bouncing the Database $ORACLE_SID ..."
echo ******************##################************************
sqlplus -s /nolog << EOF
connect / as sysdba
alter system flush shared_pool;
shutdown immediate;
startup;
EOF
echo ******************##################************************
echo `date` "=>Startup up the listener $META_LISTENER ..."
echo ******************##################************************
lsnrctl start $META_LISTENER
##################************************
##################************************
##################************************
##################***********************
. ~/$AS_env
## Startup Infra on Bar ##
echo ******************##################************************
echo `date` "=>Starting up infra ...."
echo ******************##################************************
opmnctl @instance:$INSTANCE_INFRA startproc
while opmnctl @instance:$INSTANCE_INFRA status | grep -i "init" > /dev/null
do
sleep 5
echo `date` "=>Waiting for the Infra to come up completely..."
done
## Startup Mid AS on Foo ##
echo ******************##################************************
echo `date` "=>Starting up mid tier AS on Foo ...."
echo ******************##################************************
opmnctl @instance:$INSTANCE_MIDAS startproc
while opmnctl @instance:$INSTANCE_MIDAS status | grep -i "init" > /dev/null
do
sleep 5
echo `date` "=>Waiting for the Mid Tier to come up completely.."
done
echo ******************##################************************
echo `date` "=>Bringing up dcm-daemon and LogLoader for infra and mid tier"
echo ******************##################************************
opmnctl @farm startproc ias-component=dcm-daemon
opmnctl @farm startproc ias-component=LogLoader
echo ******************##################************************
echo `date` "=>Status of mid tier components..."
echo ******************##################************************
opmnctl @instance:$INSTANCE_INFRA status
echo ******************##################************************
echo `date` "=>Status of mid tier components..."
echo ******************##################************************
opmnctl @instance:$INSTANCE_MIDAS status
##################************************
##################************************
##################************************
##################************************
No Comments Yet
Let us know what you think