Oracle Application Server: How to Bounce AS from One Location

May 6, 2008 / By Shervin Sheidaei

Tags: ,

In this post and some upcoming posts, I’m going to write more about Oracle application servers, a subject we have addressed too little on the Pythian blog.

In this post, I am addressing how to bounce a whole application server, including all tiers and databases from one location. The reason being, I have a request from a client to have the application server be bounced automatically during the weekend to release swaps and to address memory leaks. The application server on this client includes one Mid tier, one Infra tier, and one database (a metarepository database) in three different Oracle homes on two boxes.

As you know, for application server, tiers should be stopped and started up in a specific order. On startup, the sequence is like: database –> listener –> Infra tier –> Mid tier.

For shutdown, the sequence is vice versa. It is not safe to shutdown the database or Infra tier before making sure that the Mid tier is totally done.

So, in order to address client’s request, the basic plan was to have a script to shutdown Mid tier, then handshake with a tscript on the Infra tier to let it know that it is safe to shutdown the Infra tier. We would use the same approach for the Infra shutdown script and database shutdown script.

Is there a simpler way?

opmnctl, the main tool for startup/shutdown of application server components, is able to bounce the whole farm. However, first time you try to run opmnctl status @farm, you may just see the status of AS component only for single box rather than for the whole farm. Why?

For security reasons, in AS 10g, opmn requests (start, restart, and stop) to the farm work only if SSL is enabled in the opmn.xml file and a wallet file is configured. If neither SSL nor a wallet file are configured, OPMN will reject any remote process control request with HTTP code 403. Interestingly, OPMN in AS 10g is not SSL-enabled by default.

Here are the steps to enable SSL on OPMN and make it ready for a remote bounce.

  1. Edit $OH/opmn/conf/opmn.xml on all tiers and add the following lines to the <notification-server> section.
    <ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>
  2. Copy the existing wallet on each box to defined location for opmn. By default the wallet file is OH/Apache/Apache/conf/ssl.wlt/default. Copy this file to OH/opmn/conf/ssl.wlt/default.
  3. Run the following command to pick up the changes in each Oracle Home.
    $ORACLE_HOME/dcm/bin/dcmctl updateconfig -ct opmn
  4. Bounce the tiers.
  5. Run the following statement from one location, and you should see status of all components in all tiers.
    opmnctl status  @farm
    opmnctl stopall @farm
    opmnctl startall @farm

    In my case:

    oracle(/home/oracle):opmnctl status @farm
    Processes in Instance: infraAS10g.##################
    -------------------+--------------------+---------+---------
    ias-component      | process-type       |     pid | status
    -------------------+--------------------+---------+---------
    
    HTTP_Server        | HTTP_Server        |  471476 | Alive
    
    LogLoader          | logloaderd         |  462866 | Alive
    
    dcm-daemon         | dcm-daemon         |  430186 | Alive
    
    OC4J               | OC4J_SECURITY      | 1089626 | Alive
    
    OC4J               | oca                | 1069158 | Alive
    
    DSA                | DSA                |     N/A | Down
    
    OID                | OID                | 1200560 | Alive
    
    Processes in Instance: Mid10gAS_###########################
    
    -------------------+--------------------+---------+---------
    
    ias-component      | process-type       |     pid | status
    
    -------------------+--------------------+---------+---------
    
    HTTP_Server        | HTTP_Server        |  978974 | Alive
    
    LogLoader          | logloaderd         |  471280 | Alive
    
    dcm-daemon         | dcm-daemon         |  962758 | Alive
    
    OC4J               | home               |  585772 | Alive
    
    OC4J               | OC4J_BI_Forms      |  946312 | Alive
    
    OC4J               | OC4J_Portal        |  790612 | Alive
    
    WebCache           | WebCache           |  704698 | Alive
    
    WebCache           | WebCacheAdmin      |  524378 | Alive
    
    Discoverer         | ServicesStatus     |  774258 | Alive
    
    Discoverer         | PreferenceServer   |  811190 | Alive
    
    wireless           | performance_server |     N/A | Down
    
    wireless           | messaging_server   |     N/A | Down
    
    wireless           | OC4J_Wireless      |     N/A | Down
    
    DSA                | DSA                |     N/A | Down

    When everything is ready, it is just a matter of having a shell script to be run on the InfraStructure tier and bouncing the whole AS environment.

    Here is the script I implemented for this client. In any steps if startup/shutdown fails, it will page the person on-call for manual intervention.

    ### Script to recycle IAS and DB ###
    ### 3 Envs should be set : INSTANCE_INFRA, INSTANCE_MIDAS,META_LISTENERAS_env=$1
    DB_env=$2
    ##################************************
    ##################************************
    ##################************************
    ##################************************
    ##################************************
    . ~/$AS_env
    export ENVIRONMENT=`hostname`
    export cnt=0
    ## Check number of AS component "
    echo ******************##################************************
    echo `date` "=> Checking number of AS components "
    echo ******************##################************************
    Mid_comps=`opmnctl @instance:$INSTANCE_MIDAS  status -noheaders | grep "|" |wc -l`
    Inf_comps=`opmnctl @instance:$INSTANCE_INFRA status -noheaders | grep "|" | wc -l`
    ## Starting AS/Database Recycle  ##
    echo ******************##################************************
    echo `date` "=>Shutting down the dcm-daemon and LogLoader first..."
    echo ******************##################************************
    opmnctl @farm stopproc ias-component=dcm-daemon
    opmnctl @farm stopproc ias-component=LogLoader
    ## Shutdown Mid AS on Foo ##
    echo ******************##################************************
    echo `date` "=>Shutting down mid tier AS ...."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_MIDAS stopproc
    ## Wait until the AS processes actually end up in a 'DOWN' state
    while opmnctl @instance:$INSTANCE_MIDAS status | grep -i "stopped"  > /dev/null
    do
    sleep 5
    ((cnt=cnt+1))
    if [ $cnt = 1000 ]; then
       break;
    fi
    echo `date` "=>Waiting for the  Mid Tier to go down.."
    done
    echo ******************##################************************
    echo `date` "=>Status of mid tier components..."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_MIDAS status
    Mid_Downcomps=`opmnctl @instance:$INSTANCE_MIDAS status -noheaders |grep "Down" |wc -l`
    if [ "$Mid_Downcomps" = "$Mid_comps" ]; then
       echo `date` "Mid-tier is down .... Safe to go for Infra"
    else
    /usr/sbin/sendmail oncall@example.com << EOF
    From: $ENVIRONMENT@example.com
    To: oncall@example.com
    Subject: Automatic AS bounce failed, It needs manual intervention.
    Message:
    Manual intevention is required to fix AS bounce
    EOF
    exit 4
    fi
    ## Shutdown Infra on Bar ##
    cnt=1
    echo ******************##################************************
    echo `date` "=>Shutting down infra ...."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_INFRA stopproc
    ## Wait until the AS processes actually end up in a 'DOWN' state
    while opmnctl @instance:$INSTANCE_INFRA status | grep -i "stopped" > /dev/null
    do
    sleep 5
    ((cnt=cnt+1))
    if [ $cnt = 1000 ]; then
       break;
    fi
    echo `date` "=>Waiting for the  Infra Tier to go down.."
    done
    echo ******************##################************************
    echo `date` "=>Status of infra components..."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_INFRA status
    Inf_Downcomps=`opmnctl @instance:$INSTANCE_INFRA status -noheaders |grep "Down" |wc -l`
    if [ "$Inf_Downcomps" = "$Inf_comps" ]; then
       echo `date` "Infra is down .... Safe to go for Database"
    else
    /usr/sbin/sendmail oncall@example.com << EOF
    From: $ENVIRONMENT.up@example.com
    To: oncall@example.com
    Subject: Automatic AS bounce failed, It needs manual intervention.
    Message:
    Manual intevention is required to fix AS bounce
    EOF
    exit 4
    fi
    ##################************************
    ##################************************
    ##################************************
    ##################************************
    . ~/$DB_env
    echo ******************##################************************
    echo `date` "=>Shutting down the listener $META_LISTENER ..."
    echo ******************##################************************
    lsnrctl stop $META_LISTENER
    echo ******************##################************************
    echo `date` "=>Bouncing the Database $ORACLE_SID ..."
    echo ******************##################************************
    sqlplus -s /nolog << EOF
    connect / as sysdba
    alter system flush shared_pool;
    shutdown immediate;
    startup;
    EOF
    echo ******************##################************************
    echo `date` "=>Startup up the listener $META_LISTENER ..."
    echo ******************##################************************
    lsnrctl start $META_LISTENER
    ##################************************
    ##################************************
    ##################************************
    ##################***********************
    . ~/$AS_env
    ## Startup Infra on Bar ##
    echo ******************##################************************
    echo `date` "=>Starting up infra ...."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_INFRA startproc
    while opmnctl @instance:$INSTANCE_INFRA status | grep -i "init"  > /dev/null
    do
    sleep 5
    echo `date` "=>Waiting for the Infra to come up completely..."
    done
    ## Startup Mid AS on Foo ##
    echo ******************##################************************
    echo `date` "=>Starting up mid tier AS on Foo ...."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_MIDAS startproc
    while  opmnctl @instance:$INSTANCE_MIDAS status  | grep -i "init"  > /dev/null
    do
    sleep 5
    echo `date` "=>Waiting for the  Mid Tier to come up completely.."
    done
    echo ******************##################************************
    echo `date` "=>Bringing up dcm-daemon and LogLoader for infra and mid tier"
    echo ******************##################************************
    opmnctl @farm startproc ias-component=dcm-daemon
    opmnctl @farm startproc ias-component=LogLoader
    echo ******************##################************************
    echo `date` "=>Status of mid tier components..."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_INFRA status
    echo ******************##################************************
    echo `date` "=>Status of mid tier components..."
    echo ******************##################************************
    opmnctl @instance:$INSTANCE_MIDAS status
    ##################************************
    ##################************************
    ##################************************
    ##################************************

One Response to “Oracle Application Server: How to Bounce AS from One Location”

  • kiran kumar says:

    Hi,
    I am working on bouncing the application server on 2 nodes using an ant script. I am using opmnctl start and startproc options. but how to use these in an ant script which lies in an windows environment and can run these commands inside ant script on remote servers. Any help would be greatly needed.

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>