Analysis of the Oracle Exadata Storage Server and Database Machine

Pythian has a full-featured Oracle Exadata Practice complete with successful implementations and reference customers.

*Updated* see comments.
Exadata — the smart storage server. I am definitely excited about this product, but my point of view is a bit different.

It’s fast, and much faster than anything out there right now. But how many shops will actually need this? How many shops can spend 2.2 million dollars on hardware and equipment?

What are the products, in a nutshell? The Oracle Exadata Storage Server (Data Sheet, PDF):

• 2U Storage “unit” with either 1 TB SAS or 3.3 TB SATA redundant capacity. There is a query processor in the box that can “offload” tasks from the main database server. Primary filtering, decompression, joins, backups.
• Storage units linked to database servers via dual Infiniband offering 20 Gbit/s (2.5 GBytes/sec) bandwidth

The Database Machine (Data Sheet, PDF):

• A standard 42U rack with 8 database servers and 12 Exadata storage servers.
• Pre-installed Linux and Oracle. Pre-configured.
• In 8 servers — a total of 256GB RAM, 64 Intel cores @ 2.66 Ghz, InfiniBand-ed and gigabit-switched.

The cost for one Database Machine: $2.33M ($650,000 + $1,680,000 in software) as grabbed from Larry’s keynote (thank chet) I called the “call us now” phone mentioned on the Oracle Exadata website to ask them for pricing. They had no idea what I was asking about, and I’m still waiting on a salesperson to call me back. (Hint for Oracle — educate your sales staff about new products, just in case I decide to buy one the day after you announce it.)

You have to realize how “cheap” this is. It comes down to $25,000 per core for Oracle EE, RAC, and Partitioning! And extra “free” CPUs for decompressing, filtering and joining, and backups. That’s a good deal. Oh, did I mention you can interconnect several 42U racks?

Back to the main question, what problems does this product solve?

Configuration

That’s right, the number one problem this product solves is configuration. 90% of the problems I’ve seen are due to improperly configured systems. I am not talking init.ora settings here, or design, or indexing, or any of that. I am talking configuration mistakes all over the place. Starting from bottom up, these are the most common mistakes:

• buying large disks without accounting for I/O bandwidth delivery
• mis-configuring them in big meta-arrays (EMC style) with non-aligned stripe sizes. (See “turn-offs” in Christo Kutrovsky, Oracle Pinup)
• sharing the spindles for redo, datafiles, backup, and a bunch of databases (3par style), thus ensuring that I/O is never sequential
• getting single-channel Fibre Channel connectors to the database server
• not configuring directI/O or asyncI/O or the largest possible db_file_multi_read_count
• not using ASM. Of course, ASM reduces overhead, manages data in 1MB or larger (11g feature) extents (this is sequential data!)
• not using parallel query properly — using default values or considering all of the above, just not getting the bandwidth to perform

Using Exadata necessarily and immediately solves all of these issues. You don’t have a choice–you get more I/O bandwidth when you buy extra space, there’s no other way.

No expensive consultants to install your system. From the DBA perspective it’s heaven — no arguing with storage people for dedicated spindles, no arguing with CIOs about big vs. small disks, no arguing with system administrators for ASM. No hiring of expensive consultants to “tune” the system or apply best practices.

You may laugh at all of the above issues, but many shops are exactly like that. Especially the big ones (the target market for Exadata), where everyone is too afraid to change anything in case they get blamed if it doesn’t work. The “best practices” are the only practices with the Database Machine.

To maximize performance, you have to get all the pieces together. Then and only then will you get all the benefits. And this is quite difficult to achieve, especially in large shops where several entire departments are involved.

In all my experience at Pythian, there has been only one client–who, thanks to a combination of good managers, thrust, and desire for performance–would exactly follow my recommendations. And you know what? They are getting their 400 Mb/sec. The new server is reaching 800 with the dual 4gbit fibre channel.

Some interesting aspects of the Oracle Exadata Storage server.

Performance

The data sheet presents two options: 1 TB with SAS with 1000 MB/s bandwidth; or 3.3 TB with SATA and 750 MB/sec. Compression is “extra”, meaning in a typical data warehouse you get 2-3 times compression, meaning your actual bandwidth will be 2000-3000 MB/sec from a single Exadata server.

Redundancy

Mirroring is provided by ASM (either 2- or 3-way). It is also performed across Exadata storage servers (does that mean 2 minimum?)

Disk failure does not abort queries or transaction.

Exadata Storage server does abort queries or transactions, but with no data loss. This is important to know when calculating risk.

Manageability

There’s a plug-in available for 10g Enterprise Manager, a GUI to manage all that. Absolutely mandatory in my opinion.

Conclusion

Oracle has solved the communication issues for big shops, and the result is indeed extreme performance. Don’t get me wrong here, the Exadata is a brilliant idea that will solve some very difficult, specific problems for large data warehouse shops. But the Database Machine will do something much more real, that will help far more people: it will make it impossible for people to mis-configure their database systems.

Hats off to Oracle for releasing a product that solves a problem we are facing everyday: convincing clients to get the right hardware setup for their database workload.

Learn more about Pythian’s services for evaluation, migration to and operational support for Oracle Exadata.