I arrived at Omni Mandalay Hotel on Sunday evening with Dan Norris. I was flying through Chicago and it turned out that Dan was on the same flight and only few rows behind me. Small world.

Preparations for the conference were very chaotic on my part and, of course, I didn’t have either of my presentations ready. I was very stressed and getting sick as well — it looked like a complete disaster waiting to happen. I’d like to say that I was feeling like Doug Burns as he often managed to get sick just before a conference. Of course, I worked on my slides for the last few days as well as on the flight and presentation was slowly getting there but boy was I tired!

I quickly said hello to the crowd in the bar on the way to my room and rushed away to do some more damage to my slides. And then I had a brilliant idea — I could still see one of my best mates and do something good about my presentation! I asked Doug if he was interested in the preview (he probably wasn’t interested but he couldn’t say it to me) especially that my session wasn’t on his original agenda. Of course, that would mean that he had to leave a bunch of other good friends and spend some time tete-a-tete. Knowing Doug, this is some of the hardest thing to ask from him but it shows how good of a friend he is! (Plus, everyone thinks that he is anti-social anyway. Shhhh!)

Doug has made my day — while he provided lots of ideas and feedback on few things that I was lucking, he generally approved the idea and confirmed that it wasn’t totally crazy. I guess that was all I needed back then and Doug knew how nervous I was about it. (Thanks mate!)

So I called Sunday a day very early and went to bed before midnight. I really needed some sleep. Woken up by the alarm at 5AM (I woke up few times during the night looking at the clock — making sure I didn’t sleep through) and slides were ready just before lunch. I even managed to do a test run and it took 65 minutes — a wee bit too long for one hour session. But it was good test and I knew I had to be just a bit more concise in few parts.

Mi morning was very productive. Unfortunately, I missed the opening keynote from Tom Kyte. Such a pity! If what Doug wrote is true, Tom was talking about the mistakes we make *because* of our experience and our assumptions. This was exactly one of the points I was making in my Battle Against Any Guess — experience is danger. I wish I could see Tom’s example. Oh well, maybe another time.

I managed to attend half of the Richard Foote’s session on indexes but my mind was far away — with my own slides. Though, I did manage to focus on bitmap indexes part and the myth of bitmap indexes not working well for columns with high cardinality. Very interesting conclusions. I’m still wondering how much overhead updates will do to such bitmap index.

After lunch, it was my turn. I ordered few copies of the latest OakTable book — Expert Oracle Practices: Oracle Database Administration from the Oak Table — that I co-authored with the bunch of other Oakies. I contributed chapter 1 in the book titled just like my presentation — Battle Against Any Guess. The plan was to give a copy away during the presentation and do a draw for another one at the end of the session. I was so nervous that I forgot about it until the end of the session so I just did a draw for two copies. The lucky winners were Lynn-Georgia Tesch and Surendra Anchula. Congratulations! For the rest of you who left the contact details — please stay tuned and we’ll organize few things online.

Now the main topic of this post — my presentation. What’s unusual about this session is that it’s not some technical stuff that I usually do but a more conceptual and motivational talk. Could I pull it off? Well, I think it went fairly well in general even though I did identify few rough places and my lack of English language mastering. Might need to work a little bit more on the flow of the presentation.

We had quite a few good laughs. Later, people in the next hall were asking about it and Dan was making the jokes on the stage so it must have been loud. Anyway, I think nobody fell asleep and I managed to get people thinking about the topic. I received many “thank you” notes yesterday and compliments on a good session so by the end of the day I was more and more pleased. Thanks everyone for attending and especially big thanks to those of you who brought to my attention examples from their own battles. If you have more to discuss — contact me by email (my last name) {at} pythian.com.

Thanks to Marco Gralike for recording some fragments and sharing them. I think he has more to come.

This is the introductory couple minutes. You can definitely notice how nervous I am starting on the stage:

Solving the wrong problem example:

That’s all for now. Stay tuned — more to come.

Oracle 11g Release 2 Grid Infrastructure has been dramatically redesigned compare to 10g and 11gR1 Clusterware. Coming with impressive set of new features, Grid Infrastructure also uses much more memory. While RAM is rather inexpensive these days, it does pose an inconvenience in some scenarios. Particularly, for sand-box type installations that I use all the time for my own tests and demonstrations. For production upgrades, you need to be aware of and plan for increased memory usage.

I’ve been able to easily run a 2 node 10g RAC cluster on my MacBook with 4 GB of RAM allocating less than 1 GB of RAM to each virtual machine. That was even enough for a mini database instance with a very small memory footprint. Oracle 11g Release 1 was pretty much the same except maybe the database instance itself required a bit more memory but one node could still fit within 1 GB of RAM.

In 11gR2, bare-bone Grid Infrastructure processes alone consume 10+ times more memory (11.2.0.1 on 32 bit Linux to be precise):

[gorby@cheese1 ~]$ ps -eo pid,%mem,rss,user,cmd --sort=rsz --cols 100 | grep -e '^ *PID' -e grid -e ohasd | grep -v grep
  PID %MEM   RSS USER     CMD
 3614  0.0  1080 root     /bin/sh /etc/init.d/init.ohasd run
 4322  0.2  3368 oracle   /nfs/11.2.0/grid/opmn/bin/ons -d
 4323  0.4  5164 oracle   /nfs/11.2.0/grid/opmn/bin/ons -d
 4117  0.6  7860 root     /nfs/11.2.0/grid/bin/oclskd.bin
 3830  0.6  8788 oracle   /nfs/11.2.0/grid/bin/gipcd.bin
 5048  0.7  8992 oracle   /nfs/11.2.0/grid/bin/tnslsnr LISTENER -inherit
 4167  0.7 10052 oracle   /nfs/11.2.0/grid/bin/evmlogger.bin -o /nfs/11.2.0/grid/evm/log/evmlogger.i
 3969  0.9 12412 oracle   /nfs/11.2.0/grid/bin/diskmon.bin -d -f
 3860  0.9 12736 oracle   /nfs/11.2.0/grid/bin/mdnsd.bin
 4067  1.1 14648 root     /nfs/11.2.0/grid/bin/octssd.bin reboot
 5016  1.2 15860 root     /nfs/11.2.0/grid/bin/orarootagent.bin
 3956  1.3 16964 root     /nfs/11.2.0/grid/bin/orarootagent.bin
 4292  1.4 17984 oracle   /nfs/11.2.0/grid/bin/oraagent.bin
 3874  1.5 20112 oracle   /nfs/11.2.0/grid/bin/gpnpd.bin
 3817  1.5 20300 oracle   /nfs/11.2.0/grid/bin/oraagent.bin
 4083  1.8 23700 oracle   /nfs/11.2.0/grid/bin/evmd.bin
 4372  2.4 31548 oracle   /nfs/11.2.0/grid/jdk/jre//bin/java -Doracle.supercluster.cluster.server=eo
 3564  3.2 41532 root     /nfs/11.2.0/grid/bin/ohasd.bin reboot
 4081  3.5 44932 root     /nfs/11.2.0/grid/bin/crsd.bin reboot
 3906 18.6 239428 root    /nfs/11.2.0/grid/bin/cssdagent
 3887 18.6 239444 root    /nfs/11.2.0/grid/bin/cssdmonitor
 3924 20.1 258564 oracle  /nfs/11.2.0/grid/bin/ocssd.bin

The second column above gives you amount of resident memory in KB for processes related to Grid Infrastructure. As you can cleanly see, processes of CSS components consume well above 700MB! In total we can account for 1 GB. (those calculations are flawed — see below)

Compare that with 10g (10.2.0.3 on 32 bit Linux) — bare-bone Clusterware processes consume only 60MB:

[oracle@lh1 ~]$ ps -eo pid,%mem,rss,user,cmd --sort=rsz --cols 100 | grep -e '^ *PID' -e nfs -e crs -e css -e evm | grep -v grep
  PID %MEM  RSS USER     CMD
 6524  0.0  348 oracle   /nfs1/oracle/oracle/product/10.2.0/crs/opmn/bin/ons -d
 4892  0.1  992 oracle   /bin/sh -c cd /nfs1/oracle/oracle/product/10.2.0/crs/log/lh1/cssd/oclsomon;
 3262  0.1 1072 root     /bin/sh /etc/init.d/init.evmd run
 3506  0.1 1100 root     /bin/sh /etc/init.d/init.crsd run
 4575  0.1 1116 root     /bin/su -l oracle -c sh -c 'ulimit -c unlimited; cd /nfs1/oracle/oracle/pro
 4890  0.1 1120 root     /bin/su -l oracle -c /bin/sh -c 'cd /nfs1/oracle/oracle/product/10.2.0/crs/
 4664  0.1 1180 root     /bin/sh /etc/init.d/init.cssd oclsomon
 3263  0.1 1188 root     /bin/sh /etc/init.d/init.cssd fatal
 4677  0.1 1188 root     /bin/sh /etc/init.d/init.cssd daemon
 6525  0.5 4792 oracle   /nfs1/oracle/oracle/product/10.2.0/crs/opmn/bin/ons -d
 4922  0.6 5224 oracle   /nfs1/oracle/oracle/product/10.2.0/crs/bin/oclsomon.bin
 5915  0.7 6280 oracle   /nfs1/oracle/oracle/product/10.2.0/crs/bin/evmlogger.bin -o /nfs1/oracle/or
 4576  1.1 9312 oracle   /nfs1/oracle/oracle/product/10.2.0/crs/bin/evmd.bin
 5018  1.1 9428 oracle   /nfs1/oracle/oracle/product/10.2.0/crs/bin/ocssd.bin
 4606  2.0 16712 root    /nfs1/oracle/oracle/product/10.2.0/crs/bin/crsd.bin reboot

The memory usage above is a bit overstated. There are some shared memory accounted multiple times. I could use Smaps interface to get better per process statistics. For example, you could see that 3 of the “top offenders” (CSS binaries) have about 40MB of shared libraries each:

[root@cheese1 ~]# ./smaps.pl 3924 | head
VMSIZE:     258576 kb
RSS:       258564 kb total
            39164 kb shared
             5180 kb private clean
           214220 kb private dirty
PRIVATE MAPPINGS
     vmsize   rss clean   rss dirty  file
   15052 kb        0 kb    15052 kb
   12016 kb        0 kb    12016 kb
   11184 kb        0 kb    11184 kb
[root@cheese1 ~]# ./smaps.pl 3887 | head
VMSIZE:     239456 kb
RSS:       239444 kb total
            40096 kb shared
             6200 kb private clean
           193148 kb private dirty
PRIVATE MAPPINGS
     vmsize   rss clean   rss dirty  file
   14624 kb        0 kb    14624 kb
   10240 kb        0 kb    10240 kb
   10240 kb        0 kb    10240 kb
[root@cheese1 ~]# ./smaps.pl 3906 | head
VMSIZE:     239440 kb
RSS:       239428 kb total
            40096 kb shared
             6200 kb private clean
           193132 kb private dirty
PRIVATE MAPPINGS
     vmsize   rss clean   rss dirty  file
   14624 kb        0 kb    14624 kb
   10240 kb        0 kb    10240 kb
   10240 kb        0 kb    10240 kb
[root@cheese1 ~]#

One way to get a practical number is to check system memory usage with and without Grid Infrastructure running — the difference is about 750MB (see the “free” column of the second row).

[root@cheese1 ~]# free
             total       used       free     shared    buffers     cached
Mem:       1283040    1131584     151456          0      18504     295668
-/+ buffers/cache:     817412     465628
Swap:       655328         76     655252
[root@cheese1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cheese1'
...
...
CRS-4133: Oracle High Availability Services has been stopped.
[root@cheese1 ~]# free
             total       used       free     shared    buffers     cached
Mem:       1283040     397144     885896          0      18640     316632
-/+ buffers/cache:      61872    1221168
Swap:       655328         76     655252
[root@cheese1 ~]# ps -eo pid,%mem,rss,user,cmd --sort=rsz --cols 100 | grep -e '^ *PID' -e grid -e ohasd | grep -v grep
  PID %MEM   RSS USER     CMD
 3614  0.0  1084 root     /bin/sh /etc/init.d/init.ohasd run

I don’t have 11gR1 test cluster handy so I can’t check 100% but Oracle 11g Release 1 Clusterware is not much different from 10g so memory usage must be similar.

The lesson is that if you are upgrading your Oracle RAC Cluster to 11gR2 from 10g or 11gR1, then you have to account for additional 700MB memory for Grid Infrastructure alone on each node. Note that, this doesn’t take into account higher memory usage of the database instances themselves.

If you didn’t manage to attend my presentation, Oracle 11g ASM — The Evolution, during RMOUG or other conferences, you have a chance to see it online today. I’m doing it a web-cast at RAC SIG. It’s today, 4-Mar-10 at 12:00pm EST (9:00am PST).

I referenced a wake up call from a CTO of an Australian hosting company. Let me quote it again:

The outage, blamed on an IBM storage array, saw the company’s chief technology officer promise “significant changes to the way we deploy and manage our storage environment”.

Today, I stumbled across another article that demonstrates their solution of the storage reliability problem. From Melbourne IT on $18m Oracle revamp:

… to improve the reliability of its operational support systems at a cost of $7 million over three years, which has also seen it switch storage vendors from IBM to EMC. Data corruption that had occurred on its IBM storage systems were blamed for a several day outage experienced at the company’s WebCentral web-hosting business.

So we see that, instead of learning the right lesson, they conclude, “This IBM storage stuff isn’t reliable, EMC sales folks convinced me that they are better. Now my storage will not fail.” The “significant changes to the way we deploy and manage our storage environment” were mere vendor change.

Well, data recovery services will be flourishing!

These were great times except one day with the questionable results of an experiment to find a recipe for the perfect hangover that I and Mogens Nørgaard have conducted. Well, the science does require sacrifices…

Back to the conference…

I have really enjoyed few technical sessions that I managed to attend. I want to mention couple that I liked especially. I managed to attend the full session by Robyn Sands about adopting variance to performance troubleshooting and performance acceptance — lots of interesting goodies. I have also enjoyed High Performance Oracle on NFS from Jeff Needham. Jeff has been instrumental helping me in number of projects and I think anybody who is thinking to run Oracle on NFS should attend Jeff’s session and learn “The Formula” covering all the basics of a successful Oracle on NFS storage stack.

I have been honored to participate in the opening keynote amongst other Oracle ACEs and ACE Directors including Stanley. By the way, if you haven’t read yet Stanley’s report from Oracle ACEs Sharks Dive at Denver Aquarium, read it.

Actually, I think there were number of Oracle ACEs and ACE Directors on the floor that deserved to be on that stage far more than I did but I digress. One of the most interesting topics we touched was cloud computing and Oracle. I should say that I have already planned the whole line up of blogs on cloud computing so stay tuned — they are coming.

My own presentation was on the second day of the conference, closer to the end, and I estimated close to 50 attendees. It was good attendance I think but the hall did felt somewhat empty-ish as it was the largest hall at the conference (triple size). The presentation went very well — even experiences folks learned some lessons and generally folks showed good level of interest. I have uploaded the slides if you want to review them.

Since this blog post has so many photos, I want to finish with another one from Denver Aquarium — it’s me not freaking out of sharks around. They were very friendly.

Oracle ASM 11g Release 2 – The Evolution

Oracle Automatic Storage Management has proven to be one of the most widely adopted new features in Oracle Database 10g and it has been dramatically improved in the later 11g releases. This presentation will explain what changes are solved by ASM, how these challenges are solved, what barriers there are to ASM adoptions, and how 11g Release 2 addresses these barriers.

I shall say that the slides alone are not that helpful without my commentary but if you didn’t manage to attend it on one of the previous conferences, we will be releasing it as a webinar soon so stay tuned.

The flight Ottawa-Denver was quite enjoyable as it’s one of the rare direct flights from Ottawa. RMOUG volunteers pick up their guest speakers personally, which is a very nice touch, and with all this my travel to Denver feels just like a short drive to work. And that’s important if you know what we are up for tonight…

Few Oracle ACEs divers are going to jump into the pool with sharks at Denver Aquarium and show them who’s the boss. I’ll pretend to be a diving ACE as well (sh-h-h-h… don’t tell anybody I’m not). Unfortunately, it does mean that we will be late to the RMOUG speakers reception but we have to sacrifice something.

I’m sure you are dying to know what happens to us and if anybody gets eaten. If so, join us tomorrow, Wednesday 17-Feb, at the Denver Convention Center at the Oracle ACEs and ACE Directors Breakfast at 7:50am. This is your chance to ask any questions you want and get honest answers on anything from Oracle databases and Fusion Apps to diving with sharks and anything in between.

Also consider dropping by the OakTable / Apress booth to have a more private conversation with members of the OakTable Network. Traditional OakTable Challenge begins again tomorrow so if you think you have a tricky question around Oracle databases — by all means let us know. You can also tweet it using #oakrmoug. This is also a chance to get your book Expert Oracle Practices: Oracle Database Administration from the Oak Table signed by number of authors that attend the conference.

Today I’ve read the following email on the mailing list of Sydney Oracle Meetup. I thought that this question is asked many times in attempt to find a silver bullet to learning Oracle so I wanted to publish my reply here on the blog, especially, that I’m a firm believer in one silver bullet that exists — there are no silver bullets.

Hi everybody,

My name is Sebastian and I’m new in the group. Yesterday I arrived late because I had a problem at work at the very last minute. However I found the meeting super interesting.

I’d like to see if someone could give me some piece of advise… I have ZERO experience with Oracle. I’m a Software Engineer and I’ve been working for more than 7 years in development (open source technologies basically). When it comes to DBs, I have some good knowledge about MySQL, PostgreSQL and SQL Server (the only MS product I like). As OS, at the moment I’m working with Linux Ubuntu 9.04.
So, my question is what would be the best way to start working with / studying Oracle?? What can I install to actually have the DBMS running in my laptop and start writing PL/SQL?? Oracle XE??

Then, do you consider I can get any change of getting a internship position (I don’t have much free time) or what would be the best way to get started in this field? I see that ALL the positions require a lot of experience.

Sorry for writing to the whole list about a personal issue. Feel free to write me to my personal email address and thanks everybody.

I look forward to seeing you in the next meet up.

Kind Regards,
Sebastian

Sebastian,

If there is a silver bullet on how to get into new technology field than I don’t know about it. It all depends on personality and what works for you.

What I know is that I often prefer a quick learner to hire with no experience in the field but he or she must be a good thinker, analytical person and etc rather than someone with 20 years of experience in the field but not able to learn anything new in a reasonable amount of time.

On the technical side:
If you want to get into Oracle database development (and I understand it’s development you are interested in rather than administration) – Oracle XE (with APEX potentially) would do the trick. You might want also to use SQL Developer (a free tool from Oracle).

Now, installing and playing with things alone won’t make you any good so START READING good books. Also, start participating in online communities (but be careful to trust what people write there). Joining Sydney Oracle Meetup is one right step you’ve made already.

Good luck.
Alex

At the moment of this writing, one of our DBA teams has just completed failover to the standby database as a result of a database crash caused by a SAN issue. A few hours have passed, and parts of these databases are still not available on the formerly primary host, but traffic is being handled just fine on the standby. This customer provides SaaS type of services. Imagine what hours of downtime would do for them and their clients?

Unfortunately, people get bitten by this overestimated (god-like I’d say) SAN reliability. It must, however, be said: SANs do fail!

Do you want such a wake up call for your executives?

The outage, blamed on an IBM storage array, saw the company’s chief technology officer promise “significant changes to the way we deploy and manage our storage environment”.

Since I mentioned one Australian example, here is one more storage failure scenario described by our friends at Open Query. There are many cases from literally any industry, and some of them are rather complicated while others are just plain obvious.

Is there a silver bullet? Well, not as solution but as a concept, yes — simply admit that SANs do fail — this what should drive infrastructure design for business continuity. Actually, I should extrapolate it to another design principle — everything fails, but that’s another story.

As a special thanks to Pythian customers (you do know that Hotsos is a Pythian partner, don’t you?), there is a $100 discount so please get in touch with us to receive it.

What should you expect coming to the Hotsos Symposium 2010? It’s 3 days packed with sessions on all aspects of Oracle performance optimization whether it’s design, troubleshooting, development, methodologies and processes. Legendary Tom Kyte — who else can you expect for the keynote?!

If you take an optional training day with Tanel Poder then you are likely to learn at least as much about troubleshooting Oracle database performance as you do during the conference and probably even more. Every presentation by Tanel has been an eye opener for me. If you’ve seen his material, you’d know what I’m talking about. Now, imagine that it’s not a one-hour session but the whole day! It will fry your brains so this day is for the strongest! :)

By the way, from my past experience I recall that majority of attendees stay for the optional training day and as far as I know it’s the same year after year. They can’t all be wrong again and again.

Back to the general conference, the agenda is packed with top caliber speakers this year — no surprise. There is quite a bit of new blood — seven new Hotsos speakers in addition to twenty “old-timers”.

You might have noticed my name on the list as well. That’s right. I’ll do two presentations at the symposium – “Battle Against Any Guess” and “Run-Time Load Balancing in Oracle RAC”. You probably know what the first session is about if you read the latest book by the cool bunch of OakTable Network members or joined The BAAG Party. My second presentation is the expansion of my previous Hotsos presentation on RAC workload management but this time I’ll focus exclusively on run-time load balancing.

One special thing about Hotsos Symposium is the special atmosphere at the conference:

• new unique (and sometimes crazy) ideas start at Hotsos
• old ideas are run by your peers and validated (or not)
• toughest questions gets answered

Of course, one of my personal targets at the conference is to meet good ole’ friends and hopefully make some new ones. Please do come at say hello during the symposium — I’d be glad to shake the hands of the blog readers. You could even make a more formal meeting request if you feel like it.