Pythian Group

Solaris 11 has been release a few days ago. I was anxious to upgrade as I was using Solaris Express 2010.11 for some time and I was hitting a couple of bugs. One was a nasty IP layer bug (BAD TRAP: type=e (#pf Page fault) rp=ffffff005c9b1040 addr=20 occurred in module “ip” due to a NULL pointer dereference) causing kernel panics – not a good thing for a storage server.

Since I was using a version of 11 already, an experimental upgrade was not a problem. With the BE (boot environments) feature, one could boot into any version safely. BE is an awesome feature. Need to install a patch? Install into a boot environment – any problems reboot into the old environment. BEs leverage ZFS snapshots to create a clone of your boot disk, install any patches onto it and allow you to switch flawlessly between the two.

The upgrade process
The upgrade was extremely easy. With the pkg manager – everything is fully automated. Simply run the update and wait. It downloads everything as needed, creates a clone, upgrades it by installing and removing any packages required and makes it current. The next restart brings you the new upgraded release.
So I gave it a try – and it worked – flawlessly. I was pleasantly surprised and happy. Of course, it did give me a scare after the first reboot. Took nearly 15 minutes (compared to 2) as it had to initialize something about the packages. It even converted by /etc/hostname* network config files to the new ipadm method – which I love. Read the rest of this entry . . .

Hi! Recently I was working on a task wherein I had to confirm if the direct IO is in use or not.

filesystem_io_option database parameter was set to “DIRECTIO” to make use of directio.

Now in Linux it becomes very easy.you just need to read /proc/slabinfo :

cat /proc/slabinfo | grep kio

In the SLAB allocator there are three different caches involved. The kioctx and kiocb are Async I/O data structures that are defined in aio.h header file. If it shows a non zero value that means async io is enabled.
Read the rest of this entry . . .

This post has bee updated live from the Oracle OpenWorld as I’m learning what’s new. Last update done on 28-Sep-2010.

Oracle Exadata v2 has been transformed into x2-2 and x2-8. x2-2 is just slightly updated while x2-8 is a much more high-end platform. Please note that Exadata x2-2 is not just an old Exadata v2 — it’s a fully refreshed model. This is a huge confusion here at the OOW and even at the Oracle web site.

The new Exadata pricing list is released and Exadata x2-2 costs exactly the same as old Exadata v2. Exadata x2-8 Full Rack (that’s the only x2-8 configuration — see below why) is priced 50% higher then Full Rack x2-2. This is hardware price only to clarify the confusion (updated 18-Oct-2010).

Exadata Storage Server Software pricing is the same and licensing costs per storage server and per full rack is the same as for Exadata v2 because number of disks didn’t change. Note that storage cells got upgraded but priced the same when it comes to Exadata Server software and hardware. Nice touch but see implications on databases licensing below.

This comparison is for Full-Rack models Exadata x2-2 and x2-8 and existing v2 model.

Finally, data-sheets are available for both x2-2 (Thx Dan Norris for the pointers):

http://www.oracle.com/technetwork/database/exadata/dbmachine-x2-2-datasheet-175280.pdf

and x2-8:

http://www.oracle.com/technetwork/database/exadata/dbmachine-x2-8-datasheet-173705.pdf

It means that live update of this post is probably over (27-Sep-2010).

Read the rest of this entry . . .

It’s Sunday morning early in San Francisco and the biggest ever Oracle OpenWorld is about to start. It looks like it’s also going to be the busiest ever OpenWorld for me — my schedule looks crazy and I still need to do the slides for my Thursday sessions (one on ASM and one on cloud computing). Fortunately, my slides for today’s presentation are all ready to go.

OK. Don’t let me carry away — I started this post with the intention to write about what I expect Oracle to announce at this OpenWorld and it seems like the most important announcements happen at tonight’s keynote. I hasn’t been at the Oracle ACE Directors briefing so unlike them, all I can say is pure speculation-based and my wishes of what should be covered. Actually, unlike them, I actually CAN say at least something. :)

1. Oracle Exadata Database Machine v3 (x2-8) — well, that shouldn’t come as a surprise to anybody by now. I fully expect upgrade of the hardware — new Intel CPUs (probably with more cores), more memory, possibly more flash (this technology moves really quick these days). Maybe 10GbE network can be introduced to address some of the customers demands but I don’t think it’s needed that much. InfiniBand might just stay as it is — I think there is enough throughput but Marc Fielding noted that moving InfiniBand to the next speed level shouldn’t be very expensive. Other then cosmetic upgrade, I believe that hardware architecture will largely stay the same — it works very well, it’s proven and very successful. Maybe something should be done to let customers integrate Exadata better into their data-centers — folks keep complaining of inflexibility (and I think Oracle should stay firm on this and don’t let customer screw themselves up but who knows).
   On the software side, Read the rest of this entry . . .

At Kscope this year, I attended a half day in-depth session entitled Data Warehousing Performance Best Practices, given by Maria Colgan of Oracle. In that session, there was a section on how to determine I/O throughput for a system, because in data warehousing I/O per second (iops) is less important than I/O throughput (how much actual data goes through, not just how many reads/writes).

The section contained an Oracle-specific in-database tool, and a standalone tool that can be used on many operating systems, regardless of whether or not a database exists:
Read the rest of this entry . . .

Update 04-Aug-10: I’m happy to see The Illumos Project announcement.

Update 17-Aug-10: So Oracle seems to drop OpenSolaris project. Let’s see if Illumos pulls it off.

I’m tired reading all over the internet — Oracle taking back OpenSolaris, Open Solaris May Die
?, Solaris Is Dead, Save Open Solaris, Oracle taking back OpenSolaris.

I’m so sick of it!

I see that some don’t even know the difference between OpenSolaris and commercial Oracle Solaris (former Sun Solaris 10)!

Wake up people! Oracle did make commercial Solaris 10… eh… commercial, that is. They (well, Sun but Oracle paid big $$ for it) have invested lots into Solaris IP and they have full rights to actually charge money for it and they probably should. Struggling Sun made commercial Solaris free to use in desperation to maintain their rapidly shrinking market share. Oracle doesn’t need that – they are not desperate. You’ve made the right decision Oracle – keep Solaris commercial and use these funds to continue developing this great operating system (or whatever makes business sense).

Having said all this, what does it have to do with OpenSolaris? Nothing!

OpenSolaris was and is free. I have just quickly skimmed through the licensing (Binary License and CDDL) and there are no caveats that I can see like 90 days limitation or whatsoever. All the OpenSolaris goodies are still available to everyone for free.

Whining starts that Oracle will not contribute to OpenSolaris anymore. Come on people! Couldn’t you just appreciate what’s been done already and what a great product OpenSolaris is? If you forgot what open-source is about, it’s about community contributions and not about a single vendor giving away its IP so that everyone around can scream how great open-source movement is what great products it produces. If one vendor pulls out and community can’t sustain product development, then the product cannot live its normal open-source life.

Get over it! Want a high quality software with great support without any fuss? Pay $$. Want a high quality free open-source software? Make it happen!

Update 13-Jul-10: I’m keeping an eye on the OpenSolaris community news and it’s been so far discouraging..

When Oracle announced Oracle 11.2 for Solaris (x86-64), I decided to try a silent installation of Oracle on OpenSolaris, even though it is not certified.

I downloaded the Solaris ISO and installed it as 64-bit on one of my virtual machines with 1Gb of RAM. Installation of the OS went without a problem and I was soon enjoying a look at OpenSolaris. I eventually stopped admiring the good-looking interface (interesting—do they sell OpenSolaris-style doughnuts somewhere in Sun’s offices?), and connected through a plain old black-and-white ssh terminal to execute the silent installation.

Based on the silent installation that I had earlier executed on Linux for 11.2, I started the following: Read the rest of this entry . . .

I recently performed a migration from Oracle 10gR2 on Solaris to the same version on Linux, immediately followed by an upgrade to 11g. Both platforms were x86-64. Migrating to Linux also included migrating to ASM, whereas we had been using ZFS to hold the datafiles on Solaris. Restoring files into ASM meant we would have to use RMAN (which we would probably choose to use anyway).

As with many databases, the client wanted minimal downtime. It was obvious to us that the most time-consuming operation would be the restore and recovery into the new instance. We were basically doing a restore and recovery from production backups and archived redo logs. It quickly dawned on me that we could start this operation well before the scheduled cutover time and downtime window, chopping at least six hours from the downtime window. The client would only need to keep the new instance in mount mode after the initial restore/recovery finished, periodically re-catalog the source instance’s FRA (which was mounted via NFS), and then re-run the recover database command in RMAN. Once the time comes to cutover, simply archivelog current the original instance and shutdown immediate. Then open the new instance with the RESETLOGS option, and voila! Migration complete!

I’ll try to recreate a simple example here. Read the rest of this entry . . .

In the process of scripting a database migration, I was in need of something akin to the GNU basename utility that I know and love on Linux. basename is most famous for taking a full file path string and stripping away the leading path component, returning just the name of the file. This can be emulated in PL/SQL with calls to SUBSTR and INSTR, like this:

substr(dirname,instr(dirname,'/',-1)+1)

(Thanks to Ian Cary, who shared this logic on oracle-l)

As you can see, this simply finds the last occurence of /, which is our directory separator on *nix and Solaris operating systems. On Windows, it would be \. It then returns a substring beginning one character after that last separator until the end of the string. Voila, a basic basename routine!

Upon reading the basename man page again, I found that basename also takes an optional parameter, a suffix string. If this suffix string is provided, basename will also truncate that string from the end. For example:

Read the rest of this entry . . .

We’ve been running into a problem with one client:

SELECT COUNT(*) FROM tbl;

takes 0.25 seconds on one db, and 0.06 seconds on another.

Consistently. That’s a fourfold difference.

There aren’t any significant configuration differences (like query cache, etc.), the software versions are the same, and the table fits into memory. This has been looked at by at least 3 in-house MySQL experts, and the only thing we can determine is that it’s a hardware difference.

The table fits into memory so it’s not a disk issue, and the only other difference among the hardware is that the slower machine has Solaris virtualization in place in the form of “containers” (cpu is the same, etc). Is this something that’s known to cause issues with speed? The “tbl” in question is an InnoDB table, if that means anything. Is there something like the “speed” of RAM?

Note that Sun has already been called in, and scripts have been run and analyzed on their side and they say “must be a db config issue” even though the configs have no significant differences (and neither does the hardware and OS spec).