Pythian Group

Solaris 11 has been release a few days ago. I was anxious to upgrade as I was using Solaris Express 2010.11 for some time and I was hitting a couple of bugs. One was a nasty IP layer bug (BAD TRAP: type=e (#pf Page fault) rp=ffffff005c9b1040 addr=20 occurred in module “ip” due to a NULL pointer dereference) causing kernel panics – not a good thing for a storage server.

Since I was using a version of 11 already, an experimental upgrade was not a problem. With the BE (boot environments) feature, one could boot into any version safely. BE is an awesome feature. Need to install a patch? Install into a boot environment – any problems reboot into the old environment. BEs leverage ZFS snapshots to create a clone of your boot disk, install any patches onto it and allow you to switch flawlessly between the two.

The upgrade process
The upgrade was extremely easy. With the pkg manager – everything is fully automated. Simply run the update and wait. It downloads everything as needed, creates a clone, upgrades it by installing and removing any packages required and makes it current. The next restart brings you the new upgraded release.
So I gave it a try – and it worked – flawlessly. I was pleasantly surprised and happy. Of course, it did give me a scare after the first reboot. Took nearly 15 minutes (compared to 2) as it had to initialize something about the packages. It even converted by /etc/hostname* network config files to the new ipadm method – which I love. Read the rest of this entry . . .

Just a quick announcements…

If you didn’t manage to attend my presentation, Oracle 11g ASM — The Evolution, during RMOUG or other conferences, you have a chance to see it online today. I’m doing it a web-cast at RAC SIG. It’s today, 4-Mar-10 at 12:00pm EST (9:00am PST).

A couple weeks ago I did a short blog post about SAN storage failures and how people are blinded by all the bells and whistles that are supposed to make storage arrays 100% reliable and failsafe. My conclusion was that there is no way to avoid storage failures, and that a better way is to anticipate those failures and be ready to handle them with minimal service impact.

I referenced a wake up call from a CTO of an Australian hosting company. Let me quote it again:

The outage, blamed on an IBM storage array, saw the company’s chief technology officer promise “significant changes to the way we deploy and manage our storage environment”.

Today, I stumbled across another article that demonstrates their solution of the storage reliability problem. From Melbourne IT on $18m Oracle revamp:

… to improve the reliability of its operational support systems at a cost of $7 million over three years, which has also seen it switch storage vendors from IBM to EMC. Data corruption that had occurred on its IBM storage systems were blamed for a several day outage experienced at the company’s WebCentral web-hosting business.

So we see that, instead of learning the right lesson, they conclude, “This IBM storage stuff isn’t reliable, EMC sales folks convinced me that they are better. Now my storage will not fail.” The “significant changes to the way we deploy and manage our storage environment” were mere vendor change.

Well, data recovery services will be flourishing!

Oracle ASM 11g Release 2 – The Evolution

Oracle Automatic Storage Management has proven to be one of the most widely adopted new features in Oracle Database 10g and it has been dramatically improved in the later 11g releases. This presentation will explain what changes are solved by ASM, how these challenges are solved, what barriers there are to ASM adoptions, and how 11g Release 2 addresses these barriers.

I shall say that the slides alone are not that helpful without my commentary but if you didn’t manage to attend it on one of the previous conferences, we will be releasing it as a webinar soon so stay tuned.

How many times have we heard the assurance of storage administrators (fueled by the SAN vendor’s claims) that their top-of-the-shelf SAN arrays simply cannot fail. Unfortunately, reality proves this wrong and we see it regularly with our customers.

At the moment of this writing, one of our DBA teams has just completed failover to the standby database as a result of a database crash caused by a SAN issue. A few hours have passed, and parts of these databases are still not available on the formerly primary host, but traffic is being handled just fine on the standby. This customer provides SaaS type of services. Imagine what hours of downtime would do for them and their clients?

Unfortunately, people get bitten by this overestimated (god-like I’d say) SAN reliability. It must, however, be said: SANs do fail!

Do you want such a wake up call for your executives?

The outage, blamed on an IBM storage array, saw the company’s chief technology officer promise “significant changes to the way we deploy and manage our storage environment”.

Since I mentioned one Australian example, here is one more storage failure scenario described by our friends at Open Query. There are many cases from literally any industry, and some of them are rather complicated while others are just plain obvious.

Is there a silver bullet? Well, not as solution but as a concept, yes — simply admit that SANs do fail — this what should drive infrastructure design for business continuity. Actually, I should extrapolate it to another design principle — everything fails, but that’s another story.

On first glance, it looks like TEXT and VARCHAR can store the same information. However, there are fundamental differences between the way TEXT fields and VARCHAR fields work, which are important to take into consideration.

Standard
VARCHAR is actually part of the ISO SQL:2003 standard; The TEXT data types, including TINYTEXT, are non-standard.

Storage
TEXT data types are stored as separate objects from the tables and result sets that contain them. This storage is transparent — there is no difference in how a query involving a TEXT field is written versus one involving a VARCHAR field. Since TEXT is not stored as part of a row, retrieval of TEXT fields requires extra [edited 1/22] memory overhead.

Read the rest of this entry . . .

What: Sydney Oracle Meetup #6 — storage for Oracle databases

When: June 17, 2009 5:30 PM. Please RSVP Yes/No/Maybe.

Where: Our usual location at Sydney CBD. Level 3 this time!

Details:

We will start at 5:30PM with pizza and drinks and roll on from there as usual.
Note that we are meeting at the level 3 this time!

This meetup will be focused on storage technologies for Oracle database. It looks like a short presentation on Oracle Automatic Storage Management is in order – quite a few people are missing the concepts of the Oracle flagman storage storage solution and it’s useful to understand the approach whether you use it now or not.

So the presentation is – Oracle ASM 11g – the Evolution by Alex Gorbachev:
Read the rest of this entry . . .

Last week, I was at the NetApp office in North Sydney for the presentation on NetApp SnapManager for Oracle. It was good opportunity to learn more about NetApp snapshots while working on a project for one of our clients in Sydney. It was an especially interesting topic as I have some experience using Veritas Checkpoints (see my presentation on test systems refreshes), and it was interesting to see what’s different and new in the NetApp implementation. But I digress.

I learned that NetApp can provide access to the same LUNs via either Fiber-Channel (FC) or iSCSI. And this is when the interesting argument surfaced. Apparently, some companies aim to have the technology stack on their disaster-recovery site as different as possible from the primary production site. Their argument is that if one technology fails at the primary site (like FC to access storage), then the DR site using a different technology stack will more likely be unaffected.

Hrm . . .  I had never thought about this, and when I consider it now, it still doesn’t appeal to me. If I design a highly-available solution with a disaster-recovery site in place, one of my priorities would be to switch between the sites comfortably at any time. The more differences two sites have, the lower my comfort level is.

The only reason why I think some companies can “demand” having different storage technology stacks at production and DR is to justify a more convenient (a cheaper?) implementation.

Thoughts? Comments?

In the era of consolidation, storage has not been left out. Different systems are made to share the same storage boxes, fiber-channel switches and networks. Inside a typical storage box, we have front-end and back-end controllers, cache, physical spindles shared amongst different applications, databases, backup destinations, and so on.

The impact of backup on normal database activity . . . batch processing in one database impacting transactional processing — these are two real life examples of the consequences of storage consolidation known to almost every DBA. Of course, it’s easy to suggest separating databases to different physical disks, but what about SAN box controllers and shared cache? And don’t forget about the cost factor and ubiquitous consolidation that forces storage administrators to pack as much data as possible into a single SAN or NAS storage device.

Some of our customers use hosting services — they outsource hardware hosting just like they outsource DBA work to Pythian. In such scenarios, hosting service providers usually have storage hardware shared amongst different customers to provide higher utilization and on-demand storage capacity at a lower cost.

Read the rest of this entry . . .

I was reading the morning newspaper with a cup of coffee, well, actually I was reading slashdot.org, and I tripped across this story about some new 750G disks @ 7200 RPM soon to be released by Seagate. This filled me with a sense of dread about having to, once again, go through the process of convincing purchasing managers at various customer sites that actually, no, they can not just buy three of these and RAID-5 them together into a huge storage area for their terabyte database.

But, but, why, you may ask?

Think of a disk array as a warehouse. No, not a data warehouse, an actual brick and mortar warehouse. Imagine it as a big building in which you store physical stuff, like books or paper forms or cases of wine or something. Visualize the warehouse as having several loading docks for delivering new stuff or for loading up containers to ship stuff out. Then, imagine the access road or, for a large warehouse, various access roads leading to the loading docks. Are you with me so far?
Now, let’s map this analogy back to the array:

The square feet of your warehouse is the size of your array in gigabytes.

The loading docks are the separate disks in your array.

The access roads are the number of controllers in your server servicing these disks.

So now, tell me, what happens when you use very big disks for high-performance applications? You have way, way too many square feet to service with far, far too few loading docks (and usually only one access road!!!).

In the “good old days” when 9G disks were big, we didn’t have this problem. Really, this problem is new since then. Back then, if we wanted 200G of storage RAID1, we needed about 45 of those disks. Controllers could only handle 7 of them, you see (the 8th device on the bus was the controller itself) and that meant we had proportionally lots more access roads, and lots more loading docks per square foot of warehouse space than you typically have today. Now, some may look it up and say that Ultra-320 SCSI has 32 times the bandwidth capability of those ancient controllers! But note the following: In 1998, Storage Review’s editor’s choice for enterprise hard disk was the Seagate Cheetah 9LP This drive featured 10000 RPM and 5.4ms seek time and could deliver 10MB/sec to the controller. Now compare that the the specs of these newly announced Barracudas: 7200 RPM, unpublished seek times and 100MB/sec maximum theoretical peak delivery to the controller. For reference, the 7200.7 had 8.5 ms seek, the 7200.8 had 8ms seek and the 7200.9 had 11ms seeks, so we’re likely to have seeks somewhere in the 1.5x slower range than my 9GB reference.

I note:

• The seek time is actually slower today.
• The bandwidth performance is maximum only 10 times better
• Your controller is no more than 32 times faster
• The disk, however is about 83 times bigger!

Now, I admit things got faster as they got bigger, but they did not get as faster as fast as they got bigger.

That’s why I’m founding a new club, in the spirit of BAARF, the Battle Against Any Raid Five.

I’m calling it the Battle Against Huge Disks for Databases, or “BAHD for DBs”. You can either join me in my battle against huge disks for Databases. Or not. Together, we can relegate these monsters to their intended purpose, whatever that may be. Or not.

To join, simply post a comment to this article with a story of how you have fought the BAHD for DB. I will keep the following list of charter members up to date: