Pythian Group

Automatic degree of parallelism, or Auto DOP, is a new feature in 11gR2 that promises to help manage systems where large subset of the workload runs with parallel processing. In this post I’ll introduce the feature and give very useful tips I got from Oracle’s Real World Performance expert Greg Rahn on how to use it. So this is worth reading even if you are familiar with the feature.

The problem is fairly well known – you system only has finite amount of resources. Only so many CPUs, only so many disks capable of delivering only so many IO/s and MB/s. A certain query may have amazing performance when running with 32 parallel processes all alone on your test system. When 5 people need to run it at once, and at the same time there are two scheduled jobs running each with its own parallel processes, there are two likely outcomes:

1. You will run more parallel processes than your system is capable of serving. Resulting in long queues on the CPU and storage, and overall performance degradation.
2. You limit the maximum number of parallel processes to protect the database resources, and some of the queries degrade. If you don’t detect it, the ETL process that should have finished in two hours takes 24, which means that the daily report sent to the CEO is missing some of the data. Ouch.

Read the rest of this entry . . .

With all the buzz in OOW about the big data machine, there was also a lot of non-sense flying around. I love it that the Oracle community is finally interested in Hadoop and NoSQL, but I hate it when people sound authoritative without having an actual clue. I’ve left a few presentations with smoke coming out of my ears.

Here are few things that people got all wrong:
Read the rest of this entry . . .

Oracle announced the Big Data Appliance on Monday morning keynote. Many people, me included, were long waiting for this to happen. Others didn’t think it will ever happen. So naturally, there is a lot of buzz and excitement around the new device in Open World. The keynote announcement was very short on details and certainly did not satisfy my technical curiosity. So I went to a few presentations to hear what exactly is included in the offering.
Read the rest of this entry . . .

I’ve attended the Strata conference in NYC last week. Its been many years since I’ve last attended a conference without presenting in it. On one hand, attending only makes for a far more relaxed experience. On the other hand, I missed having random people come up to me and talk about my presentation. I decided to attend the conference since it is considered the foremost data science conference. And I was very much interested in what those data scientists are up to.

Good data scientists  combine the abilities of business analysts, statisticians and software engineers. They have the skills, the tools and the mandate to mine and analyze all the data the organization collects to deliver valuable insights to the business and data-based features to the customers of the business. In addition, it is considered the hottest job around. Of course, it is data scientists who mined job postings and job moves to come up with this conclusion, so maybe take it with a grain of salt.

Data-scientists normally work with very large amounts of data, both structured (the enterprise data warehouse) and unstructured (web server logs, blog posts). Since I’m a big fan of big data, I was very curious to see what those data scientists care about.

So, in no particular order – stuff data scientists like:

Read the rest of this entry . . .

In early 2005 I worked for an SQL Server and Windows shop that wanted to transition to Linux and Oracle for the improved high availablity and scalability. We had exactly one SA who knew Linux, two SQL Server DBAs and one database developer.

The new manager of the DBA team also wanted to go for RAC.

We knew we didn’t have the necessary expertise, so after discussing with Oracle’s sales team, we decided to get a consultant to install it for us.

A week of consultant work later, countless of server re-provisions, and we still didn’t have a working RAC.
Not the consultant’s fault, but at the time Oracle’s RAC installation was both very senstivite to OS and HW configuration, and wasn’t very good at detecting problems and notifying the users. It also wasn’t easy to uninstall and re-install. Mix that with a team that has no experience with the underlying OS, and you have a guaranteed disaster.
Our SAs simply couldn’t understand and implement the installation requirements, and the DBA team couldn’t communicate this to them very well.

Fast forward six years:
Read the rest of this entry . . .

We don’t pay enough attention to Hadoop.

By “we” I mean DBAs, the rest of the world is paying plenty of attention to Hadoop. Recently, I started asking my customers and fellow DBAs about Hadoop adoption in their company. Turns out that many of them have Hadoop. Hadoop shows up in large companies and small ones, in established industries and in startups. Its everywhere.

The way Hadoop shows up in all companies, and the way DBAs don’t pay Hadoop much attention, reminds me a lot of how MySQL started showing up in the enterprise. It didn’t start by DBAs showing up one morning and telling their managers:
“There’s this new open source database. Its not as stable as Oracle and it doesn’t have all the features we need, but man – its going to save us tons of money, and its pretty simple to manage.”

Nope, this never happened. What happened instead is that developers learned about MySQL, and it seemed to them like an excellent way to go around this whole DBA thing. They could install it themselves, learn how to use it in a week and become happy and productive. Without ever having to discuss their schema, data model, requirements, capacity planning, availability, backups and all the other things that DBAs want to talk about.

By the time the application came out of developement and had to be deployed in production, MySQL was a done deal. No one is going to re-write the app just because the DBAs don’t know MySQL. Sometimes the Oracle DBAs were forced to learn and admin MySQL, but more often it was considered “not a database” and left for the sysadmins to manage, while the DBAs continued to pretend that the entire world is written by Oracle.

So thats what Hadoop adoption looks like now – Its usually introduced by the developers and administered by sysadmins, while DBAs continue to pretend it doesn’t exist or doesn’t matter. When pressed, some DBAs will even insist that all this “big data” thing can and should be done in a database, but the developers are too ignorant or lazy to work with a proper RDBMS.

I think the day arrived when, just like DBAs can no longer ignore MySQL, we can no longer ignore Hadoop either. So lets talk about it.

Read the rest of this entry . . .

Hotsos is a blast. Easily the best technical Oracle conference. The speakers are terrific, the topics are cutting edge and the audience is experienced, intelligent and engaged.

I’ve been to quiet a few conferences by now, and one of the things I noticed is that the best learning is rarely as organized as “I’ll go to this presentation about triggers and I’ll learn important things about triggers”. This works too, but often you learn more from chance comments, side conversations, something a presenter says that causes you to think more deeply about some topics.

I’m documenting the best lessons I’ve learned, so you can learn too and so I won’t forget them. Read the rest of this entry . . .

Yesterday was actually my second day at Hotsos, 2011. I arrived at Dallas late on Sunday night, and had a bit of time to catch up with friends and colleagues, but not much else.

Monday the presentations began, and early in the morning at that.

Kerry Osborne gave the keynote – some kind of “History of Computing” thing. The subject was slightly less than fascinating, but Kerry is a funny guy and it was a fun keynote anyways.

The next session was also Kerry, this time speaking about Exadata performance. I can’t say I learned anything new. This is usually an issue when attending a presentation about hot new topics – too much time is spent on the basics. In the past every presentation about streams performance included 30 minutes of “what are streams” slides which bored me to death.

Then came Cary Millsap with “Thinking Clearly about Performance”. It is really lovely to watch how his presentation evolved over the year or two that Cary was presenting on this topic. Cary is a master presenter – He has clear and minimalistic slides that support his talk, he has great stories and awesome delivery.

I appreciate the need to think clearly, so after lunch I attended Toon’s “Thinking Clearly about SQL”. This was a thought provoking presentation – I stayed up late last night trying to figure out what I learned there and what I think of it. Toon Koppelaars advocated formal methods of writing SQL statements. This involves two steps – converting business requirements to a format problem statement and then converting these statements to SQL code using formal transformations. The first step seems mandatory to me, however the formal conversion has high overhead and may result in very clunky SQL. I’d only try that if every other method of arriving at correct SQL will fail.

The next session was Andrew Zittelli’s “Four Things Every Developer Should Know about Oracle”. It was excellent. I want to send the slides to every developer I ever worked with. I learned quiet a bit about how redo works for parallel sessions and some about dbms_locks and driving tables.

My own session was up next, which caused me to miss “Contemporary Latch Internals” by Andrey Nikolaev – everyone says it was excellent and that I should have been there. My session went well – the attendees had great questions and for once they even laughed at my jokes. However, I believe that my TCP/IP presentation is cursed to always finish 10-15 minutes early. It happened yesterday, even though I’ve added a 10 minutes section about RAC troubleshooting to beef it up. For Collaborate, I’ll practice speaking slowly.

The last session for the day was Golden Gate performance by Stephen Haisley. It  was a good session and pretty in-depth. I learned some important things about monitoring GoldenGate, how to use the latest version (and when to avoid it) and why its important to use SQL batching.

It was a good and full day, and it continued with fun discussions over dinner and a surprisingly technical talk late into the night in the local bar. Hint to the wise – spending an hour or two in a bar with someone like Christo or Dr. Neil Gunther, and you learn more than you do in a full day of conference.

It started out innocently enough: Two node RAC cluster on two Linux RHEL5 with Netapp NFS used as shared filesystem for all shared files. My favorite OS and storage, so I felt confident that clusterware installation will be as smooth as it usually is. I told the customer that this can be done in 3 hours.

What I didn’t take into account is that this was my first 11gR2 installation, and that much have changed since 11gR1. As things turned out, it took over 20 hours of my time and a lot of help from colleagues and even former colleagues before we had a successful installation.

The time it takes you to read this blog post (and any other on this subject) is likely to be time well spent.

Read the rest of this entry . . .

The days are cold and wet. The nights are long and dark and full of beeping pagers. It is time to look back and reflect on the past year and plan for next year.

Accomplishment of the year: I still can’t believe I’m a member of the Oak Table.
Big change of the year: Started working for Pythian. Databases are the same everywhere, but the work environment couldn’t be more different.
Obsession of the year: NoSQL. Fortunately, the hype is slowing down and the field is slowly maturing. Making everything more interesting and less franctic.
Overblown concept of the year: CAP Theorem. Everyone talks about it, but most of it makes not sense and is absolutely useless.
Contribution of the year: Another successful and profitable training day for NoCOUG
Proud of this year: Several customers who were upgraded to 11g successfully. Two customers with new dataguard. Automated weekly restore script.
New Expertise: I’m now very good with RMAN and DataGuard, competent with ASM and Solaris and much better than I ever thought anyone could be with vi and ssh.
Worst moments: Getting paged on a serious issue for a customer you are not familiar with, at 4am, while you are working on another urgent issue, your backup is not reachable and both customers want progress reports every 5 minutes.
Best moments: Mountain biking in Downieville. I’ll be doing lots more of this in 2011.
Guilty pleasure of the year: Either Twitter or Oban 14.
Programming Language of the year: Perl. I knew it for a while, but this year I used it more than in the last 10 years combined.
Presentation of the year: Everything a DBA needs to know about TCP/IP networks. Its far more useful for more people than I ever dreamed.
Startup I wish I started in 2010: Database as a service. Its becoming the next big thing but no one is doing it well (yet). After you outsourced your DBAs and sysadmins, just letting them own the servers at their own datacenter and VPN there is the next logical step.
Next year I will: Upgrade my OCP to 11g, video blog, and do a webinar.
I want to but probably won’t: I miss having the time to dig deeply into OS problems. Sometimes spending few days collecting data, traces, reading source code.
Mission for 2011: Convince people who never presented before to give a presentation. Because its a great experience and everyone has something important to share.
Unexpected 2011 Plan: Presenting at MySQL Conference. I sent an abstract because I love the MySQL community and want to enjoy them more. I’m still shocked it was accepted despite my bio that says “Oracle DBA”. I’ll try hard not to disappoint.
2011 Prediction: I’m betting on 2011 being the year of the cloud integration services. SaaS solutions such as SalesForce  are gaining traction with enterprise IT, and in enterprise IT, integrating with existing systems is the name of the game. I foresee great future for connector solutions and the consultants who can make them useful.
Far fetched 2011 Prediction: Oracle will buy Netapp. After HP got 3Par, that just makes sense.

Have an excellent 2011!