What: Sydney MySQL User Group meetup #7 - The Reboot

When: July 16, 2009 5:30 PM (please don’t forget to RSVP yes/no/maybe)

Where: Sydney, CBD - join the meetup for exact location.

We are back! After 3 years of being silent, SMUG (can I call it so? I know there are conflicts with other acronyms) resurrects the meetings.

The logistic of the meetup is the following:

• 5:30pm — the gathering starts and we have pizza and beers and talking your peers
• 6:00pm — we start the presentation
• 7:00pm — break and peer networking consuming the leftovers
• 7:30pm — we have an open (but slightly moderated) discussion. Topic is TBC. Perhaps, we can talk about future of MySQL now that Oracle’s bought Sun?

The schedule is very approximate and we will be quite flexible. We should finish about 8:30pm.

Presentation: Migrating 1 Terabyte Database to MySQL — Case Study by Alex Gorbachev (well, that’s me).

This presentation is based on the real project migrating from RDB on OpenVMS to MySQL on Linux. We will review the migration of the data itself and what infrastructure has been setup — high availability cluster with Heartbeat, DR site replication, organizing backup for 1 TB database. The system has been successfully put in production few months ago.

The attendance is completely free but of course someone need to cover the venue and that great stuff we are going to eat and drink so…

My company, The Pythian Group, is sponsoring the SMUG for now. You probably heard of Pythian, the leading provider of database services for MySQL as well as Oracle and SQL Server.

Nice people from Navicat, kindly offered to join us sponsoring the pizza and drinks. If you happen to manage MySQL databases you probably heard for Navicat, an award-winning database administration tool. More details - http://www.navicat.com/en/products/navicat_mysql/mysql_overview.html

Those of you in Sydney — looking forward to see you all there.

Welcome to our meetup #7! This meetup will be focused on Oracle Advanced Queuing (AQ) feature and its usage in Oracle E-Business Suite.

For inexperienced SOM members, we are starting with the meet & greet and pizza+drinks at 5:30 pm and move to smart things at 6:00 pm. We will be there until about 8:30pm (some are sticking around a bit longer while some might take off a bit earlier) and will have a break in the middle. The second half is generally more open-ended as most of you already know.

So what are the goodies at this meetup?

We will start with an overview of the AQ technology in Oracle database - what it is and what it let you do. Then we will see how it’s used in Oracle E-Business Suite and how to manage it.

Our speakers for the evening are your fellow SOM members - Jurijs Velikanovs and Warwick Sands. TBC.

Warwick has been working with AQ quite extensively and doing quite a bit of development with AQ recently and will be glad to share his experience. Jurijs has been working with AQ a lot as part of Apps DBA administering OEBS and he will have a lot to say for both AQ in general and in relation to E-Business Suite. If we come to that, I would be glad to add few points from my own experience and what happened as we tried to scale on using AQ technology with transaction rate of thousands per second.

Every Oracle professional knows about Oracle Enterprise Manager’s extensive monitoring capability for Oracle products. However, only few realize that Oracle Enterprise Manager can be easily extended thanks to its Extensibility framework.

Session includes a demo of the plug-in development scenarios. After this presentation, attendees will be able to design and develop management plug-ins for their own applications.

Login to Oracle Mix and vote now. At the time of writing, my session is on the very top — help to keep it there! ;-)

I already have couple sessions to present selected by the user groups. One of them, via IOUG RAC SIG, will be on Sunday at the User Groups Forum. I really liked Sunday’s slot at the last OOW and the whole day itself was very nice — quiet before the storm. Another session was selected via APOUC. I will make sure the blog is updated with details as soon as I get the final confirmations.

Today, I’m posting the follow up session on VIP’s (Virtual IP’s) with Oracle RAC. What I demonstrate today is a typical configuration mistake for a RAC databases created with Database Configuration Assistant (DBCA) and what’s the result of such omission.

It’s actually logged as a bug 4338578 on Metalink (thanks to Marcin Przepiorowski for reference). However, I believe it’s not a database issue but (1) misunderstanding how remote listener registration works, (2) documentation bug and (3) DBCA bug.

The init.ora parameter remote_listener defines the list of remote listeners that an instance should register with. The parameter local_listener sets the address for the local listener registration and if not specified, by default it’s hostname and the default port 1521. However, it turns out that the local_listener parameter is used not only for local registration but for something else as well…

Whenever an instance registers with a remote listener, it provides a connection descriptor that is used by the listener to redirect the incoming connection requests from client programs (this process is known as server-side connection load balancing). This connection descriptor for the redirection is created based on the local listener configuration for that instance — using the address or alias provided in the local_listener parameter. So what’s many DBA’s (and obviously developers of DBCA) are missing is that local_listener controls how (to what address and port) connections from the remote listeners are redirected.

As the video demonstrates, an out-of-the box DBCA-created database causes some connections to use host’s main IP’s (use netstat to confirm that on the client or on the servers). As a result, some of the connections (and it might be the majority of connections if you use services distributed across many nodes of a cluster) don’t use virtual IP’s.

The video explains it all — demonstrates how to detect this issue and shows how to fix this misconfiguration. I had to use a smaller embedded size to fit it on our blog and it’s still a bit wider than our blog format here. However, the video is in HD format now (noticed the improvement from my first attempt?) uploaded as 720p. To see it in full size — go directly to YouTube but you won’t see anybody sexy there :) and because my glasses broke yesterday, I don’t even look intelligent anymore.

I have added some annotation to it and if you notice something — you can annotate it yourself as well. I don’t know how it works when guests annotating the videos on YouTube so try it and let me know — let’s see how cool is that.

The next videocast will be on troubleshooting ORA-12545 in RAC environment and this is the natural continuation of today’s topic. This will probably be the last videocast about virtual IP’s for Oracle RAC.

As usual, all feedback is welcome here — this is what will keep these videocasts going.

I should mention that the process actually involves the acceptance on my side and there are expectation from me such as participation in local, global and online events with Oracle and their product teams as well as participate in some OTN activities. Well, there is rather nothing I haven’t done before except maybe access to some special events where Oracle product teams share some pre-release information. Well, I surely won’t mind any additional info to satisfy my curiosity.

Finally, thanks to Francisco Munoz Alvarez for nominating me to become an Oracle ACE Director.

Now, I guess I will have to scout the internet and update all my online profiles. When is this magical Web 2.0 going to actually work?!

When: June 17, 2009 5:30 PM. Please RSVP Yes/No/Maybe.

Where: Our usual location at Sydney CBD. Level 3 this time!

Details:

We will start at 5:30PM with pizza and drinks and roll on from there as usual.
Note that we are meeting at the level 3 this time!

This meetup will be focused on storage technologies for Oracle database. It looks like a short presentation on Oracle Automatic Storage Management is in order - quite a few people are missing the concepts of the Oracle flagman storage storage solution and it’s useful to understand the approach whether you use it now or not.

So the presentation is - Oracle ASM 11g - the Evolution by Alex Gorbachev:

This is a short (30-40 minutes) presentation that starts by overview of non-ASM storage technologies for Oracle and what are their shortcomings that Oracle tried to address with ASM. Then we will introduce the principles of ASM and shortcomings of ASM 10g when it first was released. At the end we will touch 11g improvements (we already reviewed them during the past meetup) and see what’s still not addressed by enhancement of ASM 11g Release 1.

After the break, we will start an open ended discussion and I will keep us focused to avoid the havoc. :) I will also try to encourage few attendees to provide a short 10-15 introduction on some of the technologies you are using and their benefits or problems.

If we find it useful, I will introduce Oracle IO benchmarking tool - ORION and give you few real-world examples how we used in the past. However, it will need a separate presentation if we want more details so if there is interest - we will definitely schedule one of those.

Some of the typical areas we could touch (though, we won’t have time for all of them at this session):
- shared storage options if you decide to use cluster / RAC
- NFS (NetApp and etc) storage
- OCFS
- Advantages and disadvantages of different file systems
- raw devices - do we still use and want to use them?
- 3rd party filesystems (like Veritas)
- ZFS and why it’s so super cool

As usual, make sure you leave the comments on the meetup registration questionnaire — we will try to make sure it’s on the agenda.

Please don’t forget to RSVP — the earlier the better!

Well, yes we can!

And the answer is EXECUTE IMMEDIATE — it can be used to run anonymous PL/SQL blog and not just a SQL statement. However, you will want to think many many times before doing so… if you love your data.

Let’s create the test procedures:

SQL> create or replace procedure bingo as
  2  begin
  3  dbms_output.put_line('Bingo!');
  4  end;
  5  /

Procedure created.

SQL> create or replace procedure bambam as
  2  begin
  3  dbms_output.put_line('BAM BAM!');
  4  end;
  5  /

Procedure created.

Now let’s create a wrapper that we will call passing a procedure name:

SQL> create or replace procedure call_dynamic(proc_name IN varchar2) as
  2  begin
  3  execute immediate 'begin ' || proc_name || '; end;';
  4  end;
  5  /

Procedure created.

Now let’s enable server output and try to call our procedures dynamically

SQL> set serverout on
SQL> exec call_dynamic('bingo');
Bingo!

PL/SQL procedure successfully completed.

SQL> exec call_dynamic('bambam');
BAM BAM!

PL/SQL procedure successfully completed.

Now, what was that we wanted to think about? Ah here… Let’s create a table:

SQL> create table important_data (data varchar2(1000));

Table created.

SQL> insert into important_data values ('Oracle rocks!');

1 row created.

SQL> commit;

Commit complete.

SQL> select * from important_data;

DATA
-----------------------
Oracle rocks!

So what can someone do with such procedure… perhaps, something like this?

SQL> exec call_dynamic('delete from important_data; commit');

PL/SQL procedure successfully completed.

SQL> select * from important_data;

no rows selected

Oups… or maybe something like this?

SQL> exec call_dynamic('execute immediate ''drop table important_data''');

PL/SQL procedure successfully completed.

SQL> select * from important_data;
select * from important_data
              *
ERROR at line 1:
ORA-00942: table or view does not exist

You’ve got an idea. :)

OK. Let’s finally clean up:

SQL> exec call_dynamic('execute immediate ''drop procedure bingo''')

PL/SQL procedure successfully completed.

SQL> exec call_dynamic('execute immediate ''drop procedure bambam''')

PL/SQL procedure successfully completed.

And final clean up needs to be smarter as this will hang blocking itself:

SQL> exec call_dynamic('execute immediate ''drop procedure call_dynamic''')

^CBEGIN call_dynamic('execute immediate ''drop procedure call_dynamic'''); END;

*
ERROR at line 1:
ORA-01013: user requested cancel of current operation
ORA-06512: at line 1
ORA-06512: at "SYS.CALL_DYNAMIC", line 3
ORA-06512: at line 1

But doable:

SQL> exec call_dynamic('declare a number; begin dbms_job.submit(a,''begin execute immediate ''''drop procedure call_dynamic''''; end;''); commit; end')

PL/SQL procedure successfully completed.

SQL> exec call_dynamic('null');
BEGIN call_dynamic('null'); END;

      *
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00201: identifier 'CALL_DYNAMIC' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored

Anyway, the real requirements were more complicated. Granted I don’t know all the details but I strongly suspect that there would be a better solution than calling a PL/SQL procedure dynamically.

Turned out that there are very few people running 11g in production. Besides us at Pythian with number of clients on 11g, we’ve had only couple people I think including Carl Young from Metcash. Carl shared their experience of running a multi-terabyte data-warehouse on Oracle 11g and how the migration happened. Thanks a lot Carl for your insights!

If you haven’t seen the case study from Oracle about this migration — see what benefits Metcash had with 11g migration. I myself took note of few areas — Query Cache helped a lot on dimension tables lookups and some popular reports, Materialized Views invalidation problems reduced, CBO becomes smarter.

Some of the things to pay attention to in 11g — Oracle Warehouse Builder 11g (11.1.0.6) has actually less features than the latest OWB 10g release. If you look at 11.1.0.7’s list of bugs fixed (Metalink Note 601739.1), you would see a few dozen bugs on “Wrong Results” — those bugs are most worrying, of course. Many of them occur at the very specific circumstances. Particular attention for DW environment are bugs related to predicate push — can make your materialized views end up with wrong aggregates! Well, very few excuses to stick to 11.1.0.6 now.

Regarding the even in general, if I may borrow a response from one of the members to quote here:

Illuminating presentation by Alex, good questions and answers from participants. I was uncertain whether to go from 9i to 10g or 11g. This information and experience from other users has now helped me come to a decision to go straight to 11g.

Well, I should say mission accomplished!

For those of you who are not in Sydney or haven’t had a chance to attend, here are the slides from the meetup. I should note that the content is based on Christo Kutrovsky’s presentation that he delivered at the Oracle Open World 2007. Christo was scheduled to give a modified version at the IOUG Collaborate 2008 but couldn’t attend it being in Dubai at that time so I was doing it instead and took his slides as the basis. But I digress…

Enjoy! All feedback is appreciated as usual and see you at one of the next meetups here in Sydney!

One of the topics that beginners RAC DBA’s (along with network engineers supporting Oracle database infrastructure) are confused about is the Virtual IP usage in Oracle RAC starting from Oracle Clusterware 10g.

With this videocast, I will try to clarify those concerns once and for all. I have embedded the video here in a smaller window so you might want to go directly to YouTube for the full-size version of “Pythian Video: Oracle RAC - Why VIPs“.

This is an experiment for now so let’s see how it goes. Let me know if you find this format useful and don’t forger to rate it on YouTube. Note that if you don’t leave any comments here, I’d never know whether you liked it or not so don’t be shy and comment away…

Ric Van Dyke’s presentation on tuning SQL queries using 10046 trace is available on the SOM website in the Files section (you must be a member). However, there were no material from Tanel Poder’s session — it’s title started with “Zero Slides…” and Tanel demonstrated live some of his secrets of productivity in Oracle troubleshooting. Luckily, my colleague, Andrey Goryunov, managed to shoot some of it on the video and I’m publishing here a couple fragments on the visualization of Oracle performance troubleshooting.

Make sure you are watching them in HQ on YouTube to see more details and if you like these videos, make sure you rate them.

PerfSheet is a very handy solution based on Microsoft Excel scripting and let you automate extraction and charting of any data you can extract from an Oracle database (and generally speaking, any other database). The best thing is that Tanel has put great efforts in it and made it available to everyone for free. One demo is worth many words so here we go:

The second demo on Visual Performance Troubleshooting is about the application of 3D graphics in real-time Oracle troubleshooting. This is a truly revolutionary approach if you think about what you can do with it — nothing like this has been available until now.

I must mention that Hotsos helped to sponsor this meetup, so big thanks to them for pizza and beers that evening!

Now, if you are in Sydney and haven’t joined the Sydney Oracle Meetup, you know what you are missing. Our meetup #5 is next Monday, 1-Jun-09 see some of you there!