Pythian Group

A couple weeks ago I did a short blog post about SAN storage failures and how people are blinded by all the bells and whistles that are supposed to make storage arrays 100% reliable and failsafe. My conclusion was that there is no way to avoid storage failures, and that a better way is to anticipate those failures and be ready to handle them with minimal service impact.

I referenced a wake up call from a CTO of an Australian hosting company. Let me quote it again:

The outage, blamed on an IBM storage array, saw the company’s chief technology officer promise “significant changes to the way we deploy and manage our storage environment”.

Today, I stumbled across another article that demonstrates their solution of the storage reliability problem. From Melbourne IT on $18m Oracle revamp:

… to improve the reliability of its operational support systems at a cost of $7 million over three years, which has also seen it switch storage vendors from IBM to EMC. Data corruption that had occurred on its IBM storage systems were blamed for a several day outage experienced at the company’s WebCentral web-hosting business.

So we see that, instead of learning the right lesson, they conclude, “This IBM storage stuff isn’t reliable, EMC sales folks convinced me that they are better. Now my storage will not fail.” The “significant changes to the way we deploy and manage our storage environment” were mere vendor change.

Well, data recovery services will be flourishing!

How many times have we heard the assurance of storage administrators (fueled by the SAN vendor’s claims) that their top-of-the-shelf SAN arrays simply cannot fail. Unfortunately, reality proves this wrong and we see it regularly with our customers.

At the moment of this writing, one of our DBA teams has just completed failover to the standby database as a result of a database crash caused by a SAN issue. A few hours have passed, and parts of these databases are still not available on the formerly primary host, but traffic is being handled just fine on the standby. This customer provides SaaS type of services. Imagine what hours of downtime would do for them and their clients?

Unfortunately, people get bitten by this overestimated (god-like I’d say) SAN reliability. It must, however, be said: SANs do fail!

Do you want such a wake up call for your executives?

The outage, blamed on an IBM storage array, saw the company’s chief technology officer promise “significant changes to the way we deploy and manage our storage environment”.

Since I mentioned one Australian example, here is one more storage failure scenario described by our friends at Open Query. There are many cases from literally any industry, and some of them are rather complicated while others are just plain obvious.

Is there a silver bullet? Well, not as solution but as a concept, yes — simply admit that SANs do fail — this what should drive infrastructure design for business continuity. Actually, I should extrapolate it to another design principle — everything fails, but that’s another story.

In the era of consolidation, storage has not been left out. Different systems are made to share the same storage boxes, fiber-channel switches and networks. Inside a typical storage box, we have front-end and back-end controllers, cache, physical spindles shared amongst different applications, databases, backup destinations, and so on.

The impact of backup on normal database activity . . . batch processing in one database impacting transactional processing — these are two real life examples of the consequences of storage consolidation known to almost every DBA. Of course, it’s easy to suggest separating databases to different physical disks, but what about SAN box controllers and shared cache? And don’t forget about the cost factor and ubiquitous consolidation that forces storage administrators to pack as much data as possible into a single SAN or NAS storage device.

Some of our customers use hosting services — they outsource hardware hosting just like they outsource DBA work to Pythian. In such scenarios, hosting service providers usually have storage hardware shared amongst different customers to provide higher utilization and on-demand storage capacity at a lower cost.

Read the rest of this entry . . .

I was reading the morning newspaper with a cup of coffee, well, actually I was reading slashdot.org, and I tripped across this story about some new 750G disks @ 7200 RPM soon to be released by Seagate. This filled me with a sense of dread about having to, once again, go through the process of convincing purchasing managers at various customer sites that actually, no, they can not just buy three of these and RAID-5 them together into a huge storage area for their terabyte database.

But, but, why, you may ask?

Think of a disk array as a warehouse. No, not a data warehouse, an actual brick and mortar warehouse. Imagine it as a big building in which you store physical stuff, like books or paper forms or cases of wine or something. Visualize the warehouse as having several loading docks for delivering new stuff or for loading up containers to ship stuff out. Then, imagine the access road or, for a large warehouse, various access roads leading to the loading docks. Are you with me so far?
Now, let’s map this analogy back to the array:

The square feet of your warehouse is the size of your array in gigabytes.

The loading docks are the separate disks in your array.

The access roads are the number of controllers in your server servicing these disks.

So now, tell me, what happens when you use very big disks for high-performance applications? You have way, way too many square feet to service with far, far too few loading docks (and usually only one access road!!!).

In the “good old days” when 9G disks were big, we didn’t have this problem. Really, this problem is new since then. Back then, if we wanted 200G of storage RAID1, we needed about 45 of those disks. Controllers could only handle 7 of them, you see (the 8th device on the bus was the controller itself) and that meant we had proportionally lots more access roads, and lots more loading docks per square foot of warehouse space than you typically have today. Now, some may look it up and say that Ultra-320 SCSI has 32 times the bandwidth capability of those ancient controllers! But note the following: In 1998, Storage Review’s editor’s choice for enterprise hard disk was the Seagate Cheetah 9LP This drive featured 10000 RPM and 5.4ms seek time and could deliver 10MB/sec to the controller. Now compare that the the specs of these newly announced Barracudas: 7200 RPM, unpublished seek times and 100MB/sec maximum theoretical peak delivery to the controller. For reference, the 7200.7 had 8.5 ms seek, the 7200.8 had 8ms seek and the 7200.9 had 11ms seeks, so we’re likely to have seeks somewhere in the 1.5x slower range than my 9GB reference.

I note:

• The seek time is actually slower today.
• The bandwidth performance is maximum only 10 times better
• Your controller is no more than 32 times faster
• The disk, however is about 83 times bigger!

Now, I admit things got faster as they got bigger, but they did not get as faster as fast as they got bigger.

That’s why I’m founding a new club, in the spirit of BAARF, the Battle Against Any Raid Five.

I’m calling it the Battle Against Huge Disks for Databases, or “BAHD for DBs”. You can either join me in my battle against huge disks for Databases. Or not. Together, we can relegate these monsters to their intended purpose, whatever that may be. Or not.

To join, simply post a comment to this article with a story of how you have fought the BAHD for DB. I will keep the following list of charter members up to date: