I decided to build my first sand box ever. I always wanted one, but there was no time or reason until now. Fortunately, hardware is ridiculously cheap these days compared to the past, so you can actually run full-blown Oracle Applications on your home PC without compromises.

I built a Dual Core box with 8G of RAM for an install of E-Business Suite R12. For OS I chose Oracle EL5U2, mainly because I’m used to the RH/OEL clones, and also because I expected it to be less painful than other distributions.

I started slowly, as time allowed, spending some time chasing various libraries and packages on the Net. Eventually it took me a week to realize how tedious it is to go the manual way. I got so frustrated by the dependencies that I gave up and purchased access to the ULN network.

Just to illustrate the type of struggle I went through, I found four distinct lists of required packages for 10g installation (in Oracle documentation and metalink combined). I’d recommend Note 421308.1 - Requirements For Installing Oracle10gR2 On RHEL/OEL 5 (x86_64), which summarizes the database part in a neat way. I also encourage you not to underestimate requirements of related Note 376183.1 - Defining a “default RPMs” installation of the RHEL OS.

Nevertheless, it’s just way more easy to use the oracle-validated package, which will download all dependencies for you. Well, almost all.

The next step was to prepare the machine. The only tricky part (from the DBA point of view) is to setup a DNS server, but even that went well. There are plenty of nice howtos on the web — search for keyword “rndc”.

Once I had all the required packages installed, users created, kernel parameters set, domain name resolution working, and Xvnc started, it was time to call a wizard.

(more…)

About once a week, our team gets a request from the server admins to provide some information on why a database server is running slowly. This typically begins a painstaking process of finding the main processes on the OS side and tracing them back to database sessions (and possibly even to Oracle Apps Concurrent Requests).

As we all know, writing SQL to do this tie-back-when-the-issue-is-happening is not a good practice — there is always the chance to mis-type something or to forget a join and have the data come back as unusable for good decision-making. Not to mention that there is usually a time constraint involved when diagnosing high server load, etc.

We started fairly small with a solution to our problem and simply joined the v$process view to the v$session to get the OS process ID tied out to a database session. Once you have that information from v$session, you can start to going in other views such as v$sort_usage, v$sql, v$session_wait, v$transaction, and so on. Some of these additional views may have relevant information for your specific issue.

We had an additional issue in our case — the databases that sit on our servers run Oracle Applications, so simply tying to a database session provides only half of the picture. You also need to be able to pull information from the fnd_concurrent_requests table to be able to see if the OS process may be generated from a Concurrent Request within Oracle Apps.

Attached to this post (see bottom for link) is a general diagram that the team came up with for relating our tables together, taking RAC into consideration. (more…)

Anybody who has tried this multi-node to single-node cloning in 11i knows that it’s difficult and very error-prone. If we outline the Apps Tier cloning process, it looks like this (supposing we have a two-node instance with the DB and CM on one node, and Web and Forms on the other):

# CM Node
# copy all files to the target
$ perl adpreclone.pl appsTier merge

# Web Node
# copy $COMMON_TOP/clone/appl directory only to the target
$ perl adpreclone.pl appltop merge

# Target Node
$ perl adcfgclone.pl appsTier

This process is called merging appltops. It’s not as easy as it looks. Many times, the production environment might not have proper values in the fnd_nodes table, which makes adcfgclone.pl fail to properly recognize the appltops for merging.

But in R12, life is made easy. (more…)

I’m pleased to announce that there will be the formal launch of Pythian Europe at the premises of the Canadian Embassy in Prague on Wednesday the 6th of August from 17:00 to 18:30. This historic event will be announced by Mrs. Sameena Qureshi, Trade Counsellor, Embassy of Canada; and Paul Vallée, President and Founder, The Pythian Group. Present will be various members from the press (IT and Business), as well as representatives from Oracle and Sun Microsystems, the Canadian Chamber of Commerce in Prague, and many more. We will prepare some unusual and very tasty snacks and refreshments.

We would love for readers of this blog to join us, so please consider this your special, personal invitation from me. Please come if you’re in Prague on Wednesday. If you plan to attend, please contact Dan at elbl@pythian.com.

One of the main goals in architecting a Disaster Recovery (DR) solution is to make a DR failover transparent to the end users. Too often, users must reboot their desktops, clear their browser cache and the jinitiator jar cache, and so on, even when we have made sure that the post-failover URL of the 11i instance is the same. After a failover of an 11i instance from a primary site to a DR site, if the user can operate without changing anything in his desktop, only then can we say that the goal is achieved.

In most cases the culprits are: forgetting the DNS setup for the hostnames of Middle Tiers, or the load balancer, if one is used; and the caching of DNS entries at the different levels in the network. A quick look at the caching section of Wikipedia’s page on DNS gives some idea of I’m talking about. Because of the default settings, the old IP address gets cached in the user’s desktop and in caching DNS servers in the network. As a result, the user’s desktop is still trying to reach the old server, which is now offline.

The best fix for these kind of DNS side effects is to change the TTL (Time To Live) parameter of the DNS entry for the hostname from the default value to a smaller one. I prefer setting it to a value a little smaller than the time you take to failover. That is, if you take 60 minutes to failover from Primary to Secondary datacenter, then set the TTL to 50 minutes.

Let’s take an example here. Let’s say our 11i instance has the URL http://apps.example.com:8000, the primary instance being windsor, the secondary ottawa. And we have two load balancers: one at primary site and one at the secondary, with hostnames lb.windsor.example.com and lb.ottawa.example.com respectively. If the DNS is set up with default values, it will look like this:

(more…)

Oracle R12 is certified to install on 64-bit Linux for X86_64 platforms. Yeah, I know that’s old news. The interesting part here is it’s not “pure” 64-bit software like the Oracle Database on Linux 64-bit: R12 is taking advantage of the OS’s capability to run 32-bit executables.

If we look at metalink note 416305.1, there is a big list of 32-bit RPMs in the pre-req packages list for R12 on 64-bit Linux. This is not something new — 11i on Solaris, HP-UX, and AIX also had the same behaviour.

11i doesn’t take advantage of 64-bit capabilities of the OS at full scale. It runs in 32-bit mode even on a 64-bit OS, as all its executables are 32-bit. One place where 11i does use the 64-bit feature is the “Memory Based Planner” program, which is part of the Advanced Supply Chain Planning (ASCP) module. Oracle shipped a 64-bit version of the program only for the Solaris, AIX, and HP-UX platforms, while the Linux version had just the 32-bit version. I have seen customers who added a Solaris Middle Tier to their Linux 11i instances just to run this 64-bit Planner program, spending all the dollars they saved with Linux middle tiers thus far, in consulting fees to add that extra Solaris middle tier node.

The good news for R12 customers is that, we don’t need a Solaris, AIX or HP-UX middle tier any more to run the 64-bit Planner program. Oracle has started shipping 64-bit Planner (MSONWL64.exe) for Linux 64-bit from version 12.0.3 onwards. Check Metalink note 254810.1 for details.

If you are a early adopter of R12 with a version earlier than 12.0.3, its time to patch up to 12.0.3 or 12.0.4. and migrate to the Linux 64-bit platform. This migration to Linux 64-bit should not be a big hassle, as it is binary-compatible with Linux 32-bit. I expect it be as simple as 1) copy, 2) relink, and 3) startup. I will, however, find out the exact method, and post here in the blog.

We recently had an issue with a client while cloning a huge database. The result was that we had to restore the whole database as the post-clone corrupted the existing database. Pain! It took another fourteen hours to restore.

This may help you to troubleshoot the issue.

/sbox/oracle/product/920/appsutil/clone/bin$ perl adcfgclone.pl dbTier

Errors in the Alert.log after Clone failure

Completed: ALTER DATABASE ENABLE PUBLIC THREAD 3Shutting down instance: further logons disabled
Shutting down instance (normal)License high water mark = 5Mon Apr 14 14:49:00 2008

Errors in file /sbox/oracle/product/920/admin/Prod/bdump/sbox1_j001_2453612.trc:

ORA-12012: error on auto execute of job 16303
ORA-12008: error in materialized view refresh path
ORA-28112: failed to execute policy function
ORA-06512: at “SYS.DBMS_SNAPSHOT”, line 820
ORA-06512: at “SYS.DBMS_SNAPSHOT”, line 877
ORA-06512: at “SYS.DBMS_IREFRESH”, line 683
ORA-06512: at “SYS.DBMS_REFRESH”, line 195ORA-06512: at line 1~