If your configuration matches the following setup, then this blog could be helpful to you.

OS: Redhat Enterprise Linux 3 or 4
JDK: Sun Java 1.3 or 1.4
Apps: 11.5.9 or 11.5.10
Users: many Oracle Self Service Web Applications Users e.g., iProc, iRec, Timecard, and HR self-service

With this setup, you might have already faced issues like the Apps login page not responding, or browsers timing out in loading SSWA pages. You might have raised numerous long-running TARs with Oracle support on this and ended up uploading lot of Apache and Jserv debug logs, and you always end up recycling or bouncing the Apache service to fix the issue. Don’t worry—you are not alone here.

Read the rest of this entry »

As an Apps DBA, you might have been asked to clear the JSP cache many times, either by Developers or Oracle Support. The usual directory that holds the JSP cache is $COMMON_TOP/_pages in 11i and R12 releases. In older versions of 11i it used to be $OA_HTML/_pages.

What is this JSP Cache? It’s the compiled version of JSP files. When JSP gets compiled by Apache Jserv, it gets internally gets converted into a .java file (servlet) and then that .java file gets compiled into a .class file

Why do we need to clear the JSP cache? Read the rest of this entry »

Recently, an Oracle-L discussion on Sun T2000 servers got me thinking. The T2000 servers have Sun’s new line of processors, UltraSPARC-T1. These processors come with 8 physical cores, and each core has 4 threads (similar to hyperthreading in Intel Xeon Processors). So each UltraSPARC-T1 processor shows up as 32 Processors (8 cores * 4 threads) at the operating system (OS) level. Sun termed this technology “Cool Threads”. It is supposed to give high-volume throughput along with saving millions on power and cooling costs.

But many discussion forums have more complaints against these T2000 servers than praises. Read the rest of this entry »

Anybody who has tried this multi-node to single-node cloning in 11i knows that it’s difficult and very error-prone. If we outline the Apps Tier cloning process, it looks like this (supposing we have a two-node instance with the DB and CM on one node, and Web and Forms on the other):

# CM Node
# copy all files to the target
$ perl adpreclone.pl appsTier merge

# Web Node
# copy $COMMON_TOP/clone/appl directory only to the target
$ perl adpreclone.pl appltop merge

# Target Node
$ perl adcfgclone.pl appsTier

This process is called merging appltops. It’s not as easy as it looks. Many times, the production environment might not have proper values in the fnd_nodes table, which makes adcfgclone.pl fail to properly recognize the appltops for merging.

But in R12, life is made easy. Read the rest of this entry »

One of the main goals in architecting a Disaster Recovery (DR) solution is to make a DR failover transparent to the end users. Too often, users must reboot their desktops, clear their browser cache and the jinitiator jar cache, and so on, even when we have made sure that the post-failover URL of the 11i instance is the same. After a failover of an 11i instance from a primary site to a DR site, if the user can operate without changing anything in his desktop, only then can we say that the goal is achieved.

In most cases the culprits are: forgetting the DNS setup for the hostnames of Middle Tiers, or the load balancer, if one is used; and the caching of DNS entries at the different levels in the network. A quick look at the caching section of Wikipedia’s page on DNS gives some idea of I’m talking about. Because of the default settings, the old IP address gets cached in the user’s desktop and in caching DNS servers in the network. As a result, the user’s desktop is still trying to reach the old server, which is now offline.

The best fix for these kind of DNS side effects is to change the TTL (Time To Live) parameter of the DNS entry for the hostname from the default value to a smaller one. I prefer setting it to a value a little smaller than the time you take to failover. That is, if you take 60 minutes to failover from Primary to Secondary datacenter, then set the TTL to 50 minutes.

Let’s take an example here. Let’s say our 11i instance has the URL http://apps.example.com:8000, the primary instance being windsor, the secondary ottawa. And we have two load balancers: one at primary site and one at the secondary, with hostnames lb.windsor.example.com and lb.ottawa.example.com respectively. If the DNS is set up with default values, it will look like this:

Read the rest of this entry »

Oracle R12 is certified to install on 64-bit Linux for X86_64 platforms. Yeah, I know that’s old news. The interesting part here is it’s not “pure” 64-bit software like the Oracle Database on Linux 64-bit: R12 is taking advantage of the OS’s capability to run 32-bit executables.

If we look at metalink note 416305.1, there is a big list of 32-bit RPMs in the pre-req packages list for R12 on 64-bit Linux. This is not something new — 11i on Solaris, HP-UX, and AIX also had the same behaviour.

11i doesn’t take advantage of 64-bit capabilities of the OS at full scale. It runs in 32-bit mode even on a 64-bit OS, as all its executables are 32-bit. One place where 11i does use the 64-bit feature is the “Memory Based Planner” program, which is part of the Advanced Supply Chain Planning (ASCP) module. Oracle shipped a 64-bit version of the program only for the Solaris, AIX, and HP-UX platforms, while the Linux version had just the 32-bit version. I have seen customers who added a Solaris Middle Tier to their Linux 11i instances just to run this 64-bit Planner program, spending all the dollars they saved with Linux middle tiers thus far, in consulting fees to add that extra Solaris middle tier node.

The good news for R12 customers is that, we don’t need a Solaris, AIX or HP-UX middle tier any more to run the 64-bit Planner program. Oracle has started shipping 64-bit Planner (MSONWL64.exe) for Linux 64-bit from version 12.0.3 onwards. Check Metalink note 254810.1 for details.

If you are a early adopter of R12 with a version earlier than 12.0.3, its time to patch up to 12.0.3 or 12.0.4. and migrate to the Linux 64-bit platform. This migration to Linux 64-bit should not be a big hassle, as it is binary-compatible with Linux 32-bit. I expect it be as simple as 1) copy, 2) relink, and 3) startup. I will, however, find out the exact method, and post here in the blog.

From my previous post on TXK rollup patch, you already know the significance of adctxinf.tmp file in $AD_TOP/admin/template directory. It has wealth of information about different XML tags in the Context XML file of an Apps 11i instance. In relation to the same file, now I want to share with you all a small XSL (XML style sheet) file I wrote back in 2005. It makes adctxinf.tmp much more readable; all tags are presented in a tabular format in the browser.

For users of Microsoft Internet Explorer, Firefox, and Safari, go directly to this link to view the file. Those of you who want to have an offline copy, download that file and vasu.xsl to same directory and open adctxinf.xml in any app of your choice. Its just a very simple XSL file, but it should make life a little easier. My goal when I wrote this stylesheet was to make the Context XML file itself much more human-readable. But I never got much time to work on that.

It’s true — XML is for machines; HTML is for humans.

You might have already seen the blog update on Steven Chan’s site. TXK AutoConfig and Templates Rollup Patch S (6372396) was released on May 5th.

This patch differs from traditional TXK autoconfig template patch releases in that the ATG team decided to include some other important TXK patches also with this release. One of these is TXK Advanced Utilities Rollup Patch C (5011249).

This Advanced Utilities patch brings some import scripts that can be used to implement advanced topologies. The important benefit here is that these scripts can be run from command line. For complete details, refer to Metalink note 277574.1, Running Configuration Wizards from the Command Line in Oracle Applications 11i.

As a side effect of this generous inclusion of import updates, the patch size has increased from 16mb (RUP R) to 65mb (RUP S).

This Rollup Patch has also brought in some new Context variables. These new XML tags in Rollup Patch S are:

1. s_jdbc_connect_descriptor_generation: The value of this variable determines whether the jdbc connect descriptor is regenerated and whether the value of the context variable s_apps_jdbc_connect_descriptor is updated. Acceptable values for this context variable are true (the default) or false. Set this value to false when using a custom apps jdbc connect descriptor
2. s_apcprestart: This variable is used to specify the complete path of the script to be executed before the Apache service is started.
3. s_ccmmaxsyspathlen: (Windows-specific) This context variable is used for setting the maximum length of the Windows System Path.

Read the rest of this entry »

Today I ran into an 11i E-Business Suite instance which is using Apache/JServ to do forms load-balancing. In fact, it’s my first customer to implement and successfully use Apache/JServ for load-balancing. Here is quick sketch of the instance architecture:

                  Users
                    | 11i URL:
                    | http://appsrv1.pythian.com:8000/
                    |
             ----------------
             |   appsrv1    |
             | apache/jserv |
             |  formservlet |
             ----------------
            /                \
           /                  \
----------------             ----------------
|   appsrv2    |             |   appsrv3    |
| apache/jserv |             | apache/jserv |
|  formservlet |             |  formservlet |
----------------             ----------------

            -------------------
            |    dbsrv01      |
            | CM/RepSrv       |
            |  Database       |
            -------------------

Almost all the customers that I have worked on till now, with a requirement for multiple middletier, have implemented load balancing using hardware load-balancers like Cisco Content Switch or F5 Big-IP.

This customer is running forms in servlet mode, a prerequisite for using Jserv to do load-balancing. If somebody wants to do forms load-balancing in socket mode and opts not to use any hardware LBRs, then the only option is a Metrics server.

Here is brief overview on how forms load-balancing happens in above architecture. Read the rest of this entry »

It is quite common for Oracle Tech support, while troubleshooting any 11i E-Business Suite Self Service Applications (SSWA) related error messages, to ask to enable debug logging in JServ configuration files.

The procedure to enable debug logging in JServ is to update following line in file $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties from log=false to log=true, and then bounce Apache using:

$ cd $COMMON_TOP/admin/scripts/$CONTEXT_NAME
$ adapcctl.sh stop
$ adapcctl.sh start

After the bounce, we start seeing debug messages in the log file $IAS_ORACLE_HOME/Apache/Jserv/logs/jserv.log.

Please note, the logfile’s and config file’s locations can be different from those mentioned above. In case you have a shared tech stack implemented, replace $IAS_ORACLE_HOME with $CONF_TOP/iAS.

The main problem with this procedure is the bounce of Apache. In cases where the client is not a heavy SSWA user and the issue isn’t affecting a significant number of users, then often we will be asked to wait until off-business hours to do the bounce of Apache. These delays can make your metalink SR hop between different Oracle support offices in different timezones, which can lead to delays in resolution of the issue.

Let’s have quick review of JServ configuration files which are of interest to us. jserv.conf — this is the configuration file of mod_jserv module in Apache. Here’s a snippet from jserv.conf: Read the rest of this entry »