Let’s start by creating a simple table with a deferrable primary key:

SQL> create table def_bug(n number primary key deferrable initially deferred);

Table created

SQL> insert into def_bug values (1);

1 row inserted

SQL> insert into def_bug values (2);

1 row inserted

SQL> commit;

Commit complete

You can confirm that the primary key constraint is working fine by trying to insert a duplicate value:

SQL> insert into def_bug values (1);

1 row inserted

SQL> commit;

commit

ORA-02091: transaction rolled back
ORA-00001: unique constraint (SRC.SYS_C004070) violated

So far so good. Open a second session and execute the following update:

SQL> update def_bug set n=3 where n=2;

1 row updated

Do not commit yet and execute in your first session:

SQL> update def_bug set n=3 where n<=2;

The above update will block due to our second session holding a lock on the row where n=2. Now commit your second session…

SQL> update def_bug set n=3 where n=2;

1 row updated

SQL> commit;

Commit complete

…and then commit your first session:

SQL> update def_bug set n=3 where n<=2;

1 row updated

SQL> commit;

Commit complete

Take a look at the data now:

SQL> select * from def_bug;

         N
----------
         3
         3

Ouch! This was certainly unexpected. You can confirm that the primary key is still working by trying to insert a duplicate value again:

SQL> insert into def_bug values (3);

1 row inserted

SQL> commit;

commit

ORA-02091: transaction rolled back
ORA-00001: unique constraint (SRC.SYS_C004070) violated

It certainly looks like the update statement did not take into account deferrable constraint declared on the table during restart caused by the write consistency mechanism.

In this post, we’re going to take a closer look at various queries that the Extract process uses against the database. As before, we will start by examining the strace output:

nanosleep({1, 0}, NULL)                 = 0
...
read(20, "\1\"\0\0\255\1\0\0\217\0\0\0H\200\366\256\5\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024000) = 1024000
...
write(16, "\0$\0\0\6\0\0\0\0\0\3N'\7\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7\0011"..., 36) = 36
read(17, "\0\351\0\0\6\0\0\0\0\0\6\1\"\375\2\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 233
write(16, "\0 \0\0\6\0\0\0\0\0\3N(\10\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7\0011", 32) = 32
read(17, "\0\343\0\0\6\0\0\0\0\0\6\1\"\7\1\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 227
write(16, "\0K\0\0\6\0\0\0\0\0\3N)\t\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7,/"..., 75) = 75
read(17, "\0\341\0\0\6\0\0\0\0\0\6\1\"\375\1\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 225
write(16, "\0Q\0\0\6\0\0\0\0\0\3N*\n\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7,/"..., 81) = 81
read(17, "\0\254\0\0\6\0\0\0\0\0\4\1\0\0\0)\0\1\0\0\0\0{\5\0\0\0\0\n\0\0\0"..., 8208) = 172
lseek(20, 227328, SEEK_SET)             = 227328
nanosleep({1, 0}, NULL)                 = 0

File descriptors 16 and 17 are the pipes for one of the bequeath connections we have with the database. There are four queries being submitted each cycle (following the same order as those being sent to a database):

SELECT DECODE(archived, 'YES', 1, 0), status  FROM v$log WHERE thread# = :ora_thread AND sequence# = :ora_seq_no

SELECT MAX(sequence#)  FROM v$log WHERE thread# = :ora_thread AND status in ('INVALIDATED', 'CURRENT', 'ACTIVE')

SELECT DECODE(status, 'STALE', 1, 0) FROM v$logfile WHERE member = :log_name

SELECT 1  FROM V$LOGFILE WHERE(STATUS NOT IN ('STALE', 'INVALID') OR STATUS IS NULL) AND MEMBER <> :log_name AND EXISTS ( SELECT 1 FROM V$LOG WHERE GROUP#  = V$LOGFILE.GROUP# AND THREAD# = :ora_thread AND SEQUENCE# = :ora_seq_no ) AND ROWNUM = 1

The purpose of these statements is to constantly keep an eye on what’s happening inside the database by regularly polling the contents of the above views. What’s worth mentioning about the above queries is that all of them will cause extra I/O to the controlfile. On my test database, that equaled 640KB each cycle (40 I/O requests, 16KB each). In most cases, this is nothing to worry about–just keep the additional I/O in mind in case your controfile is already a hot spot.

The redo log stores object identifiers (a number), which means that when the Extract process encounters a supported operation, it needs a way to find out more details. This is achieved by a couple of statements against the data dictionary. The following statement will be issued first:

SELECT u.name, o.name, o.dataobj#, o.type#, (SELECT bitand(t.property, 1) FROM sys.tab$ t WHERE t.obj# = :ora_object_id) FROM sys.obj$ o, sys.user$ u WHERE o.obj# = :ora_object_id  AND decode(bitand(o.flags, 128), 128, 'YES', 'NO') = 'NO'  AND o.owner# = u.user# AND decode(bitand(o.flags, 16), 0, 'N', 16, 'Y', 'N') = 'N' AND (o.type# in (1, 19, 20, 34) OR EXISTS (SELECT 'x' FROM sys.tab$ t WHERE t.obj# = :ora_object_id))

In case the object turns out to be a table, it will be checked whether it is an overflow segment for an IOT:

SELECT nvl(iot_name, 'NULL')   FROM all_tables WHERE owner = :owner AND table_name = :object_name

This allows the Extract process to figure out whether it needs to process changes, in case the overflow segment belongs to an IOT from which we’re capturing the data. In case the object in question turns out to be an index, a corresponding check will be made to see whether it’s an underlying index for an IOT:

SELECT table_owner, table_name FROM all_indexes WHERE index_name = :object_name AND         owner = :owner AND index_type = 'IOT - TOP'

This is required so that the changes made to an IOT can be captured, in case it belongs to an interested tables list.

The above queries will be executed regardless of whether or not you’re interested in capturing changes from the particular object, because the queries are required before you can make that decision. In case this is something we’re interested in, additional information will be requested:

select object_type, object_name, subobject_name from dba_objects where object_id = :ora_object_id

The above statement is necessarily in case we’re dealing with the partitioned object and, depending on the result, one of the following two statements will be executed:

select ts.bigfile from dba_tablespaces ts,  all_tables t  where t.table_name = :ora_object_name and  t.tablespace_name = ts.tablespace_name and rownum = 1

select t.bigfile from dba_tablespaces t,  all_tab_partitions p  where p.partition_name = :ora_subobject_name and   p.tablespace_name = t.tablespace_name and rownum=1

There is obviously an issue with the above two statements. Neither of them specify the object owner and, in case you have two (or more) objects with the same name but in different schemas, the above statements may return incorrect information, if these objects are located in different tablespace types.

It’s interesting how the issue is shoved away using rownum = 1 condition. What could the potential impact be? One thought that immediately comes to mind is that the way ROWIDs are organized is different between small- and big-file tablespaces (the part being used for a relative data file number in a small file tablespace is used for a block number in case of a big file tablespace), so some functionality that potentially relies on that could be affected. I’ve got a couple of ideas, but I’ll hold these until I do some testing.

What else is interesting? The big thing is that none of the information on columns is being resolved. All I’m going to say right now is that column information will be fetched by the Replicat process using the destination system’s data dictionary. Combine this with the fact that an online data dictionary does not store historical information about an object’s metadata, and you have a perfect recipe to get yourself into various nasty situations (which is exactly the reason why Oracle Streams relies on MVDD instead of the online data dictionary). But more on that when we get to the Replicat process internals series.

One of the follow up questions to part I was about the way the Extract process reads from ASM storage. I’ve provided the answer, however, today we’re going get a detailed look at how the Extract process interacts with an ASM instance and what kind of implications may result.

Let’s take a look at what changes in the Extract process loop when it reads from ASM storage:

nanosleep({1, 0}, NULL)                 = 0
...
write(18, "\37\333\0\0\6\0\0\0\0\0\3^\v)\4\4\0\1\0\0\0\376\377\377\377\377\377\377\377C\0\0"..., 8155) = 8155
write(18, "\37\333\0\0\6\0\0\0\0\0\1\1\0Z\7\n\0\0\0\0\0\1\2\0\0W\7\n\0\237\0\0"..., 8155) = 8155
write(18, "\37\333\0\0\6\0\0\0\0\0\1\0\4\0\0\0\237\0\0\1u\7\n\0\0\0\0\0\1\2\235\336"..., 8155) = 8155
write(18, "\22\367\0\0\6\0\0\0\0\0\0.\1\300\0\204\0\n\0\0\200\0\0\217\7\n\0\237\0\0\1\232"..., 4855) = 4855
read(18, "\37\333\0\0\6\0\0\0\0\0\v\1\5\276\4\0\0\0\0\0\1\0\0\0\0\0\0\0\260[\211\312"..., 8208) = 8208
read(18, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8155) = 8155
read(18, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8155) = 8155
read(18, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8155) = 4540
...
nanosleep({1, 0}, NULL)                 = 0

As before, to make the loop a bit more understandable, I've removed a bunch of syscalls. Let's take a look at the file descriptor 18:

[root@gg1 fd]# ls -l 18
lrwx------ 1 oracle oinstall 64 Jan 24 20:56 18 -> socket:[24439]

Let’s find out more about this socket…

[root@gg1 fd]# lsof -p 7725 | grep 18u
extract 7725 oracle   18u  IPv6              24439                TCP [::1]:16428->[::1]:ncube-lm (ESTABLISHED)
[root@gg1 fd]# lsof -i :16428
COMMAND  PID   USER   FD   TYPE DEVICE SIZE NODE NAME
extract 7725 oracle   18u  IPv6  24439       TCP [::1]:16428->[::1]:ncube-lm (ESTABLISHED)
oracle  7753 oracle   16u  IPv6  24440       TCP [::1]:ncube-lm->[::1]:16428 (ESTABLISHED)
[root@gg1 fd]# ps -f -p 7753
UID        PID  PPID  C STIME TTY          TIME CMD
oracle    7753     1  0 20:56 ?        00:00:00 oracle+ASM (LOCAL=NO)

In place of the read syscalls there is now some sort of communication going on with ASM instance.

Alas, when it comes to tracing, an ASM instance is not the most friendly thing you can work with. But there are other ways. Enabling SQL*Net trace is enough to see what replaced the read syscall:

BEGIN dbms_diskgroup.read(:handle, :offset, :length, :buffer); END;

This is nothing other than one of the ASM internal packages which can be used to read directly from ASM disk group. How is the :handle being obtained? If we go a little bit up the trace, we can find the answer:

BEGIN dbms_diskgroup.getfileattr('+REDO/gg1/onlinelog/group_2.2547.708989191', :filetype, :filesize, :lblksize); END

The above code is used to get file attributes which are necessary to call the dbms_diskgroup.open procedure:

BEGIN dbms_diskgroup.open('+REDO/gg1/onlinelog/group_2.257.708989191', 'r', :filetype, :lblksize, :handle, :pblksize, :filesize); END;

This call will return a :handle which can later be used in dbms_diskgroup.read. The name of the online redo log file to read is being fetched by querying the RDBMS instance:

SELECT member,        DECODE(status, 'CURRENT', 1, 0),        DECODE(archived, 'YES', 1, 0)   FROM (select lf.member,        l.status,        1.archived        from v$logfile lf, v$log l        WHERE lf.group# = l.group# AND        l.thread# = :ora_thread AND        l.sequence# = :ora_seq_no AND        (lf.status NOT IN        ('INVALID','INCOMPLETE','STALE') OR        lf.status is null)        order by lf.member DESC )        where rownum = 1

Note that this query is a bit different form the one used when fetching the log file name on a file system. Both dbms_diskgroup.getfileattr and dbms_diskgroup.open calls happen when a new log file needs to be accessed (during startup or after a log file switch, for example), i.e. we do not execute these as part of a main loop listed above.

The next thing to find out is what happens in the ASM dedicated process as a result of dbms_diskgroup.read call:

read(16, "\37\333\0\0\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 8155
read(16, "\37\333\0\0\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 8208
read(16, "\1\0\0\0\0\0\0\0\0\0\0\v\1\33\5\0\0\31\2\2\r\0\0\0\0\0\0\372\0\300\0M"..., 8155) = 4803
pread(15, "\1\"\0\0\22 \0\0\21\0\0\0H\200\305\365\5\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 28672, 136324096) = 28672
write(16, "\37\333\0\0\6\0\0\0\0\0\v\1\5\326\4\0\0\0\0\0\1\0\0\0\0\0\0\0\240/V\232"..., 8155) = 8155
write(16, "\37\333\0\0\6\0\0\0\0\0\v\5\1\0\4\0\0\0\237\0\0\1g\251\v\0\0\0b\251\1\2"..., 8155) = 8155
write(16, "\37\333\0\0\6\0\0\0\0\0\0\2\0\3\0\4\0\5\0\6\0\7\0\10\0\t\0\n\0\v\0\f"..., 8155) = 8155

If you take a loot at the lsof output above, the file descriptor 16 is a socket back to the Extract process so these syscalls are related to SQL*Net messages. What we’re really interested in is a call on line# 4 with a pread syscall from the file descriptor 15:

[root@gg1 fd]# ls -l 15
lrwx------ 1 oracle oinstall 64 Jan 24 20:58 15 -> /dev/sdb1

Of course, /dev/sdb1 is a device under out REDO diskgroup:

SQL> select d.path, g.name
        from v$asm_disk d, v$asm_diskgroup g
        where d.group_number = g.group_number
                and d.path='/dev/sdb1';  2    3    4

PATH            NAME
--------------- ----------------
/dev/sdb1       REDO

What else can we say? First of all, the read size is significantly smaller, just 28672 bytes, compared to 1000K read size when the log has been located on a cooked file system. If you get such a small read size and the fact that the data needs to go through the network and SQL*Net stacks… I would say that I expect this to be much less efficient compared to how online redo logs are being read from a file system. One of the immediate things to realize is that, in case you’re running the Extract process on the same machine as your ASM instance, it probably makes total sense to configure the connection string to ASM instance (the one which is being specified in the Extract process parameters) using bequeath protocol so the traffic can go through a pipe instead of a socket which should provide better performance.

As many of you are probably aware, up until recently, GoldenGate had been a third-party product. Technology-wise this presents an interesting challenge for the GoldenGate development team as they have to rely on whatever Oracle makes available to the outside world. Let’s see what kind of techniques they were able to utilize in order to achieve their goals.

I did a simple replication setup between two different databases with the Extract, DataPump and Replicat processes. I’m planning to take a look at all of these but today is the Extract’s process turn.

Oracle GoldenGate Extract Process
The main duty of the Extract process is to read and process Oracle redo logs in order to extract relevant changes and write these to a trail.

Reading from the Redo Logs
This is probably the most interesting aspect of the Extract process as this is where various technology stacks are being bridged together. Unless you configure Extract process to read strictly from the archived logs, it will try to read from online redo logs whenever possible. Let’s take a look at an strace of a running Extract process:

lseek(18, 2659328, SEEK_SET)            = 2659328
nanosleep({1, 0}, NULL)                 = 0
...
read(18, "\1\"\0\0J\24\0\0U\0\0\0H\200\301\302\0\0\0\0\0\0\0\0\0\6\0\0\n\0\20\0"..., 1024000) = 1024000
...
lseek(18, 2659328, SEEK_SET)            = 2659328
nanosleep({1, 0}, NULL)                 = 0
...
read(18, "\1\"\0\0J\24\0\0U\0\0\0H\200\301\302\0\0\0\0\0\0\0\0\0\6\0\0\n\0\20\0"..., 1024000) = 1024000
...
lseek(18, 2659328, SEEK_SET)            = 2659328
nanosleep({1, 0}, NULL)

I’ve omitted a bunch of syscalls as I’ll get to these a bit later plus it makes it much easier to see and understand the loop. Judging by the nanosleep syscall you can figure out that this cycle repeats every second. I’ve left file descriptor 18 in place since it’s what pointing out to our current redo log file:

[oracle@gg1 fd]$ ls -l 18
lr-x------ 1 oracle oinstall 64 Jan 19 16:18 18 -> /u02/oradata/GG1/onlinelog/o1_mf_1_5oco39l6_.log

Let’s take a closer look at what happens inside that loop. The loop begins with a lseek syscall which sets offset for the specified file descriptor. After a one second delay the Extract process performs a 1000K read starting at the offset sat by the lseek. You can see multiple lseek calls to set the offset to exactly the same position, one for each time the loop gets executed. This is necessary since the read syscall advances the current position in a file, hence we need an lseek syscall to get us back to where we’ve started.

As you’ve probably already guessed, this is nothing but a constant polling of the redo log file contents to see whether the new data has arrived. You can confirm that 2659328 is the current position in the online redo log by using a query against x$kcccp:

SQL> select cpodr_bno, cpodr_bno*512, (cpodr_bno-1)*512
  2  	from x$kcccp where indx=0;

 CPODR_BNO CPODR_BNO*512 (CPODR_BNO-1)*512
---------- ------------- -----------------
      5195       2659840           2659328

The Extract process starts reading from (and including) the last written block. When the new data arrives, the Extract process advances the offset appropriately:

lseek(18, 2729984, SEEK_SET)            = 2729984
nanosleep({1, 0}, NULL)                 = 0
...
read(18, "\1\"\0\0\324\24\0\0U\0\0\0\20\200b\227\204\0\0\0\5h\0\0\200\321\10\0\1\0w5"..., 1024000) = 1024000
...
lseek(18, 2731008, SEEK_SET)            = 2731008
nanosleep({1, 0}, NULL)                 = 0

The above polling continues until the Extract process detects that there is a supported operation in the redo stream which potentially requires capturing. Remember that redo records contain object ids, not object names. When the object identifier is encountered for the first time, the Extract process will issue a series of statements against the data dictionary to find out what it is. These queries are relatively lightweight, however, if you have a large number of objects inside your database then your data dictionary may became bombarded with these queries upon the Extract process start. The corresponding object name and type will be determined. In case of an index it will be checked whether it’s an IOT in order to resolve the table name properly.

Challenges
There are a number of interesting scenarios which may occur during online redo log reading. To begin with, you’re dealing with a file to which lgwr is actively writing. Processes inside an Oracle database use a vast amount of coordination to make sure they do not step on each other’s toes, however, when you’re a third-party process operating outside of the database then what’s left available to you is to constantly observe the situation and react accordingly while making sure you won’t get in anybody’s way. For example, it would be nice to ask lgwr to notify us when the new data had been written to the online redo log file, but we can’t. So we have to resort to a constant polling to see if something new has appeared. This is one of the areas to keep an eye on as GoldenGate integration with Oracle progresses over time.

Oracle redo logs are being written sequentially in the circular fashion which makes the task of reading these a bit less challenging, however, you may still find yourself in various nasty situations when you think about different possibilities. For example, while you’re reading the next 1000K, the lgwr might be writing to the same area so some (or all) of the data which you read back might be garbage. My guess would be that the Extract process just keeps reading forward until it encounters data that is no longer valid (due to invalid redo record or RBA not from the expected sequence) and then resumes from the last valid record encountered.

Here is something more interesting to think about. What if, while you’re reading the redo log file, lgwr manages to go the complete round (i.e. switch through all redo groups and get to the one you’re currently reading from) and overwrite the data you’re currently working with? This race condition is relatively unlikely to happen in a real life but still a possibility so I became curious to test this out.

I’ve blocked the Extract process, switched the redo logs around and made sure that the current online redo log gets overwritten with the new data past the current extract point. Here is an excerpt of the relevant data form the strace output for the Extract process:

lseek(18, 3027456, SEEK_SET)            = 3027456
nanosleep({1, 0}, NULL)                 = 0
fsync(19)                               = 0
lseek(15, 0, SEEK_SET)                  = 0
write(15, "H\0\0\10R\0\n\0\2\0\4\0A\0\1\0\1\0C\0\4\0\3\0\1\0S\0\30\0Sf"..., 2048) = 2048
fsync(15)                               = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
select(8, [7], NULL, NULL, {0, 0})      = 0 (Timeout)
read(18, "\1\"\0\0\31\27\0\0{\0\0\0\340\201\205N****************"..., 1024000) = 1024000
write(16, "\0$\0\0\6\0\0\0\0\0\3N3\7\0\0\0\2\0\0\0`\0\0\0\0\0\0\0\7\0011"..., 36) = 36
read(17, "\0\254\0\0\6\0\0\0\0\0\4\1\0\0\0002\24\1\0\0\0\0{\5\0\0\0\0\7\0\0\0"..., 8208) = 172
write(4, "2010-01-20 12:49:02.084  Redo th"..., 146) = 146
write(16, "\0.\0\0\6\0\0\0\0\0\3N4\16\0\0\0\1\0\0\0`\0\0\0\0\0\0\0\7\0031"..., 46) = 46
read(17, "\1\2\0\0\6\0\0\0\0\0\6\1\"\0\1\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 8208) = 258
stat("/u02/fra/GG1/archivelog/2010_01_20/o1_mf_1_120_5ogjbvbj_.arc", {st_mode=S_IFREG|0640, st_size=3034112, ...}) = 0
open("/u02/fra/GG1/archivelog/2010_01_20/o1_mf_1_120_5ogjbvbj_.arc", O_RDONLY|O_DIRECT) = 20

Let’s go through what happened and how this situation has been handled. The loop starts with positioning the file descriptor at where we left off which is 3027456. After a one second delay, a 1000K redo log read follows. Upon startup, the Extract process establishes two connections with a database using whatever userid credentials you specified in the parameter file. File descriptors 16 and 17 are pointing out to one of these sessions (a dedicated Oracle processes in my case) which the Extract can use to query various information about the database. The query on line 12 checks whether the sequence and thread from which we’re currently reading got archived:

SELECT DECODE(archived, 'YES', 1, 0),       status  FROM v$log WHERE thread# = :ora_thread AND       sequence# = :ora_seq_no

In our case, that statement will return zero rows since this particular sequence (we’ve been reading seq# 120 when the wrap has occurred) is no longer there. My guess would be that that is one of the clues the Extract process might be using to deal with the situation. Another clue is by looking at what was read from the redo log file itself. Remember that the Extract process read includes the last written redo block which can potentially be compared with the same block from a previous read in order to detect if the wrapping has occurred. After realizing that the log has wrapped, the Extract process will write a warning into it’s report file (line 14):

2010-01-20 12:49:02.084  Redo thread 1: Online log /u02/oradata/GG1/onlinelog/o1_mf_2_5og4zot6_.log on sequence# 120 has missing trailing blocks.

The next step is to do a fall back and figure out where to pick up. This task is being accomplished by another query sent on line 15:

SELECT  name    FROM gv$archived_log   WHERE sequence# = :ora_seq_no AND         thread# = :ora_thread AND         resetlogs_id = :ora_resetlog_id AND         archived = 'YES' AND         deleted = 'NO' AND         name not like '+%'         AND standby_dest = 'NO'

The bind values are:

BINDS #14:
 Bind#0
  oacdty=01 mxl=32(03) mxlc=00 mal=00 scl=00 pre=00
  oacflg=21 fl2=1000000 frm=01 csi=178 siz=96 off=0
  kxsbbbfp=2ba8c340f618  bln=32  avl=03  flg=05
  value="120"
 Bind#1
  oacdty=01 mxl=32(01) mxlc=00 mal=00 scl=00 pre=00
  oacflg=21 fl2=1000000 frm=01 csi=178 siz=0 off=32
  kxsbbbfp=2ba8c340f638  bln=32  avl=01  flg=01
  value="1"
 Bind#2
  oacdty=01 mxl=32(09) mxlc=00 mal=00 scl=00 pre=00
  oacflg=21 fl2=1000000 frm=01 csi=178 siz=0 off=64
  kxsbbbfp=2ba8c340f658  bln=32  avl=09  flg=01
  value="708599728"

In other words, we’re looking into gv$archived_log to find out where the required sequence went to (note how ASM files will be bypassed by the means of name not like ‘+%’ condition). Once the name of the archived log file has been returned, the Extract process will proceed with opening and reading from it. After that, the Extract process will check whether the next sequence is available to read from the online log file:

SELECT lf.member,        DECODE(l.status, 'CURRENT', 1, 0),        DECODE(l.archived, 'YES', 1, 0)   FROM v$logfile lf, v$log l  WHERE lf.group# = l.group# AND        l.thread# = :ora_thread AND        l.sequence# = :ora_seq_no AND        lf.member not like '+%' AND        rownum = 1 AND        (lf.status NOT IN        ('INVALID','INCOMPLETE') OR        lf.status is null)

If the sequence is available, the online redo log file will be opened, otherwise the query against v$archived_log will be issued to find out the required archived log file to open. As I’ve already mentioned, the Extract process prefers to read from the online log file whether possible.

It was indeed a pleasure to see that that situation has been handled correctly. The warning emitted into a report file was nice to see as well (now you know one of the situations when this warning may appear). The GoldenGate development team has earned some credit, at least on this particular situation.

Stay tuned for the next part.

It is somewhat timely that we’ve been doing some Oracle Golden Gate testing which in turn made me curious to take a closer look at Oracle Streams in 11GR2 and see where all the performance is coming from.

I’ve setup a simple replication for table t1 from schema src to schema dst, changed Apply Server parallelism to 1 and did a simple test with inserting 100 rows while performing a sql trace:

SQL>desc src.t1;
Name Type          Nullable Default Comments
---- ------------- -------- ------- --------
N    NUMBER
V    VARCHAR2(100) Y                       

SQL> select count(*) from src.t1;

COUNT(*)
----------
       0

SQL> select count(*) from dst.t1;

COUNT(*)
----------
       0

SQL> select sid from v$streams_apply_server;

     SID
----------
      22

SQL> exec dbms_monitor.session_trace_enable(22, waits => true);

PL/SQL procedure successfully completed

SQL> insert into src.t1
2  	select level, to_char(level)
3  		from dual
4  		connect by level <= 100;

100 rows inserted   SQL> commit;

Commit complete

SQL> select count(*) from dst.t1;

COUNT(*)
----------
     100

SQL> exec dbms_monitor.session_trace_disable(22);

PL/SQL procedure successfully completed

To my surprise, when I went to take a look at the trace file, I couldn’t find anything related to Apply process inserting rows into dst.t1, only a handful of internal housekeeping statements. This made me curious as to where all the stuff really went to so I’ve decided to take a look at v$sql and see if there will be any clues:

SQL> select sql_text
2  	from v$sql
3  	where lower(sql_text) like '%insert%dst%t1%';

SQL_TEXT
--------------------------------------------------------------------------------
select sql_text  from v$sql  where lower(sql_text) like '%insert%dst%t1%'

The only thing I was able to see there was . . .  my own statement. Things were starting to look serious.

In order to finally figure out who did what I’ve launched a logminer:

SQL> begin
2  	dbms_logmnr.start_logmnr(
3  		startTime => sysdate-30/1440,
4  		endTime => sysdate,
5  		Options => dbms_logmnr.DICT_FROM_ONLINE_CATALOG+dbms_logmnr.CONTINUOUS_MINE
6  	);
7  end;
8  /

PL/SQL procedure successfully completed

Let’s see who actually inserted all these rows:

SQL> select * from (
2  select session#, sql_redo
3  	from v$logmnr_contents
4  	where operation='INSERT'
5  		and seg_owner='DST'
6  		and table_name='T1'
7  	order by timestamp desc
8  ) where rownum <= 5;

 SESSION# SQL_REDO
---------- --------------------------------------------------------------------------------
       22 insert into "DST"."T1"("N","V") values ('1','1');
       22 insert into "DST"."T1"("N","V") values ('5','5');
       22 insert into "DST"."T1"("N","V") values ('4','4');
       22 insert into "DST"."T1"("N","V") values ('3','3');
       22 insert into "DST"."T1"("N","V") values ('2','2');

Session with SID 22 is nothing else but our Apply Server . . . 

The next step was to try and figure out whether we’re really dealing with some new codepath responsible for such spectacular performance (apparently, due to the complete lack of instrumentation :) or this is just another weird Oracle bug.

I’ve blocked the Apply Server on updating a row and looked at the Apply Server’s stack:

[oracle@ora11gr2 trace]$ pstack 17036
#0  0x0000003b83cd450a in semtimedop () from /lib64/libc.so.6
#1  0x00000000085ef3f3 in sskgpwwait ()
#2  0x00000000085ee5c6 in skgpwwait ()
#3  0x000000000829ee31 in ksliwat ()
#4  0x000000000829e422 in kslwaitctx ()
#5  0x0000000000af92f5 in ksqcmi ()
#6  0x00000000082ac019 in ksqgtlctx ()
#7  0x00000000082aa77a in ksqgelctx ()
#8  0x0000000000c4d566 in ktcwit1 ()
#9  0x00000000082d5d99 in kdddgb ()
#10 0x00000000082c7530 in kdusru ()
#11 0x00000000082c0902 in kauupd ()
#12 0x0000000001f57c14 in kddiruUpdate ()
#13 0x000000000179eeab in knasdaExecDML ()
#14 0x000000000179d928 in knasdaProcDML ()
#15 0x000000000178c6fd in knaspd ()
#16 0x0000000001787d2f in knasplcr ()
#17 0x00000000017866d7 in knaspx ()
#18 0x0000000001770fd5 in knalsProc1Txn ()
#19 0x000000000177022d in knalsptxn ()
#20 0x00000000017424a6 in knasm2 ()
#21 0x0000000001776d8d in knalsma ()
#22 0x0000000000c25a7d in knlkcbkma ()
#23 0x0000000000b93ba7 in ksvrdp ()
#24 0x00000000020d2dd7 in opirip ()
#25 0x00000000016fe729 in opidrv ()
#26 0x0000000001b7183f in sou2o ()
#27 0x00000000009d3f8a in opimai_real ()
#28 0x0000000001b76ace in ssthrdmain ()
#29 0x00000000009d3e71 in main ()

Before we move on, here is a stack dump from a blocked Apply Server in 10.2.0.4:

[oracle@ora10gr2 trace]$ pstack 23787
#0  0x0000003b83cd450a in semtimedop () from /lib64/libc.so.6
#1  0x00000000085ef3f3 in sskgpwwait ()
#2  0x00000000085ee5c6 in skgpwwait ()
#3  0x000000000829ee31 in ksliwat ()
#4  0x000000000829e422 in kslwaitctx ()
#5  0x0000000000af92f5 in ksqcmi ()
#6  0x00000000082ac019 in ksqgtlctx ()
#7  0x00000000082aa77a in ksqgelctx ()
#8  0x0000000000c4d566 in ktcwit1 ()
#9  0x00000000082d5d99 in kdddgb ()
#10 0x00000000082c7530 in kdusru ()
#11 0x00000000082c0902 in kauupd ()
#12 0x00000000084588c9 in updrow ()
#13 0x00000000084f2580 in qerupFetch ()
#14 0x0000000008453cdd in updaul ()
#15 0x0000000008451bca in updThreePhaseExe ()
#16 0x00000000084509f5 in updexe ()
#17 0x00000000083fe18f in opiexe ()
#18 0x00000000083f5c0d in opiall0 ()
#19 0x0000000008403d25 in opikpr ()
#20 0x00000000083f78b9 in opiodr ()
#21 0x00000000084892af in __PGOSF141_rpidrus ()
#22 0x00000000085ee820 in skgmstack ()
#23 0x000000000848a759 in rpiswu2 ()
#24 0x000000000848fdf4 in kprball ()
#25 0x0000000001c7c4d7 in knipxup ()
#26 0x0000000001c72651 in knipdis ()
#27 0x000000000178cacc in knaspd ()
#28 0x0000000001787d2f in knasplcr ()
#29 0x00000000017866d7 in knaspx ()
#30 0x0000000001770fd5 in knalsProc1Txn ()
#31 0x000000000177022d in knalsptxn ()
#32 0x00000000017424a6 in knasm2 ()
#33 0x0000000001776d8d in knalsma ()
#34 0x0000000000c25a7d in knlkcbkma ()
#35 0x0000000000b93ba7 in ksvrdp ()
#36 0x00000000020d2dd7 in opirip ()
#37 0x00000000016fe729 in opidrv ()
#38 0x0000000001b7183f in sou2o ()
#39 0x00000000009d3f8a in opimai_real ()
#40 0x0000000001b76ace in ssthrdmain ()
#41 0x00000000009d3e71 in main ()

The stack is only 30 functions deep in 11.2.0.1 compared to 42 in 10.2.0.4! Given that whatever goes up the stack from ktcwit1 () function is due to both sessions waiting on the enqueue, the relative codepath change is even bigger.

All the difference comes from the one key thing: a recursive call. If you take a look at line #25 (highlighted), you’ll notice rpiswu2 () function (for these of you unfamiliar with Oracle Kernel Layers, RPI stands for Recursive Program Interface). Whatever happens further up the stack is essentially the same codepath any user session would use while executing an UPDATE statement. The Apply Servers in 10.2.0.4 generally behave like any other user session would and whatever diagnostic techniques you have learned while troubleshooting user issues could be, to a large extent, applied to the Apply Servers as well. Every LCR execution leads to at least one recursive call (so if you got, say, a transaction with 1000 LCRs that would be at least 1000 recursive calls by the Apply Server). In 11.2.0.1 the recursive call is missing and the codepath is different up to kauupd () (KA, Access Layer) function.

Indeed, by looking at the Apply Server statistics in 11.2.0.1 you will notice that executing an LCR no longer results in a recursive call so the entire change seems to be around a shortcut which allows the Apply Server to proceed directly into KD (Data) layer, bypass the “regular” codepath and avoid a recursive call.

On a side note it appears the this new codepath was first introduced in 11.1.0.7.

What’s up with the instrumentation?

While performance improvement is certainly most welcome, there is a big downside—all these new functions seems to be poorly, or not-at-all, instrumented. This makes it hard to evaluate the gains, as some stuff is simply not there.

How are you supposed to figure out what’s going on then? The good news is that all regular dynamic performance views (like v$session_wait, v$session_event, etc.) seems to be populated correctly, but sql trace took a big hit (plus you can no longer see sql_id in v$session). Whatever falls out of the “old” stuff looks like a black box . . .  pretty much.

Puzzled by this problem, I’ve tried to see whether there is any easy way to enable the old codepath so you can get all the instrumentation facilities back in place. After some trial and error, it turned out that a simple row level trigger . . . 

SQL> create or replace trigger dst.buid_t1 before delete or insert or update on dst.t1
 2  for each row
 3  begin
 4  	null;
 5  end;
 6  /

Trigger created

SQL> begin
 2  	dbms_ddl.set_trigger_firing_property('DST', 'BUID_T1', false);
 3  end;
 4  /

PL/SQL procedure successfully completed

 . . . is enough to get the old codepath back. Here is the stack of Apply Server process in 11.2.0.1 with such a trigger in place:

[oracle@ora11gr2 trace]$ pstack 30640
#0  0x0000003b83cd450a in semtimedop () from /lib64/libc.so.6
#1  0x00000000085ef3f3 in sskgpwwait ()
#2  0x00000000085ee5c6 in skgpwwait ()
#3  0x000000000829ee31 in ksliwat ()
#4  0x000000000829e422 in kslwaitctx ()
#5  0x0000000000af92f5 in ksqcmi ()
#6  0x00000000082ac019 in ksqgtlctx ()
#7  0x00000000082aa77a in ksqgelctx ()
#8  0x0000000000c4d566 in ktcwit1 ()
#9  0x00000000082d5d99 in kdddgb ()
#10 0x00000000082c7530 in kdusru ()
#11 0x00000000082c0902 in kauupd ()
#12 0x00000000084588c9 in updrow ()
#13 0x00000000084f2580 in qerupFetch ()
#14 0x00000000046a363f in qerstFetch ()
#15 0x0000000008453cdd in updaul ()
#16 0x0000000008451bca in updThreePhaseExe ()
#17 0x00000000084509f5 in updexe ()
#18 0x00000000083fe18f in opiexe ()
#19 0x00000000083f5c0d in opiall0 ()
#20 0x0000000008403d25 in opikpr ()
#21 0x00000000083f78b9 in opiodr ()
#22 0x00000000084892af in __PGOSF141_rpidrus ()
#23 0x00000000085ee820 in skgmstack ()
#24 0x000000000848a759 in rpiswu2 ()
#25 0x000000000848fdf4 in kprball ()
#26 0x0000000001c7c4d7 in knipxup ()
#27 0x0000000001c72651 in knipdis ()
#28 0x000000000178cacc in knaspd ()
#29 0x0000000001787d2f in knasplcr ()
#30 0x00000000017866d7 in knaspx ()
#31 0x0000000001770fd5 in knalsProc1Txn ()
#32 0x000000000177022d in knalsptxn ()
#33 0x00000000017424a6 in knasm2 ()
#34 0x0000000001776d8d in knalsma ()
#35 0x0000000000c25a7d in knlkcbkma ()
#36 0x0000000000b93ba7 in ksvrdp ()
#37 0x00000000020d2dd7 in opirip ()
#38 0x00000000016fe729 in opidrv ()
#39 0x0000000001b7183f in sou2o ()
#40 0x00000000009d3f8a in opimai_real ()
#41 0x0000000001b76ace in ssthrdmain ()
#42 0x00000000009d3e71 in main ()

Now that looks much more familiar! All the instrumentation appeared to be back in place as well.

I’ve also discovered that:

1. DELETE seems to be always handled through the old codepath.
2. In case you have a unique constraint or a primary key supported by a non-unique index, INSERT will fall back to the old codepath.
3. UPDATE needs a primary key or a key column(-s) supported by an index in order to use the new codepath.

It remains to be seen whether this new codepath has been implemented as a shortcut for most frequently used scenarios, or whether there are some implementation restrictions as it progresses with the future releases . . .  or maybe not, due to Golden Gate taking over as a strategic direction.

How Can Range-Based Predicates Cause an Unstable Plan?

Quite easy, take the following example:

SQL> create table t as
  2  	select level n, rpad('x', 200, 'x') v
  3  		from dual
  4  		connect by level <= 100000;

Table created

SQL> create index i_t_n on t (n);

Index created

SQL> exec dbms_stats.gather_table_stats(user, 't');

PL/SQL procedure successfully completed

Now, I’ll query the table using two different conditions:

SQL> set autot traceonly explain
SQL> select * from t where n <= 100;

Execution Plan
-----------------------------------------------------
Plan hash value: 2912310446

----------------------------------------------------
| Id  | Operation                   | Name  | Rows |
----------------------------------------------------
|   0 | SELECT STATEMENT            |       |   95 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T     |   95 |
|*  2 |   INDEX RANGE SCAN          | I_T_N |   95 |
----------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("N"<=100)

SQL> select * from t where n <= 25000;

Execution Plan
------------------------------------------
Plan hash value: 1601196873

------------------------------------------
| Id  | Operation         | Name | Rows  |
------------------------------------------
|   0 | SELECT STATEMENT  |      | 24998 |
|*  1 |  TABLE ACCESS FULL| T    | 24998 |
------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("N"<=25000)

(Note that I’ve trimmed the output for the sake of readability.)

The first query was executed using INDEX RANGE SCAN, the second one using TABLE ACCESS FULL. This makes perfect sense, since the first query is going to return only a small fraction of data in a table, while the second one is going to fetch substantially more data. As you probably already guessed, if you substitute a literal value for a bind variable, your plan will depend on what value was passed during a hard parse:

SQL> variable n number;
SQL> exec :n:=25000;

PL/SQL procedure successfully completed.

SQL> set autot traceonly stat
SQL> set arraysize 100
SQL> select * from t where n<=:n;

25000 rows selected.

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
       3198  consistent gets
       2650  physical reads
          0  redo size
     271952  bytes sent via SQL*Net to client
       3135  bytes received via SQL*Net from client
        251  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      25000  rows processed

SQL> exec :n:=100;

PL/SQL procedure successfully completed.

SQL> select * from t where n<=:n;

100 rows selected.

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
       2949  consistent gets
       2647  physical reads
          0  redo size
       1479  bytes sent via SQL*Net to client
        396  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
        100  rows processed

You can already spot from the stats that both queries have used full table scans:

SQL> select sql_id, executions, child_number
  2   from v$sql
  3   where sql_text='select * from t where n<=:n';

SQL_ID        EXECUTIONS CHILD_NUMBER
------------- ---------- ------------
avj8fuq6s3j2q          2            0

SQL> select * from table(dbms_xplan.display_cursor('avj8fuq6s3j2q'));

PLAN_TABLE_OUTPUT
------------------------------------------
SQL_ID  avj8fuq6s3j2q, child number 0
-------------------------------------
select * from t where n<=:n

Plan hash value: 1601196873

------------------------------------------
| Id  | Operation         | Name | Rows  |
------------------------------------------
|   0 | SELECT STATEMENT  |      |       |
|*  1 |  TABLE ACCESS FULL| T    | 24998 |
------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("N"<=:N)

18 rows selected.

(Note that I’ve trimmed the output again.)

If I had started my example with n=100, both executions would have used index range scans instead. What if both plans are equally important to you and you can’t sacrifice one in favor of the other? There are a number of solutions to that; I’m going to describe my favorite one.

Tell the Database About It

You have to cooperate with the database, because if you do not, then the database will not cooperate with you. It is a fair game and the choice is yours.

After all, your are the only one who has the intimate knowledge of your data and how it is used. Let’s say your application is using the following function which returns a ref-cursor:

SQL> create or replace function get_data(
  2  	p_n	in number
  3  ) return sys_refcursor is
  4  	l_rf sys_refcursor;
  5  begin
  6  	open l_rf for 'select * from t where n<=:n' using p_n;
  7  	return l_rf;
  8  end;
  9  /

Function created

The above function will be subject to the same problem as described above: the plan will be developed on the first hard parse, and all subsequent executions will use the same plan regardless of the fact that it might be inefficient. The optimal execution plan is driven by how much data you request, which in turn depends on what bind variable value was supplied.

There is a point where Oracle should switch to an FTS instead of an IRS (and vise versa). Ultimately, we would like to make our function smart enough to be able to provide us different execution plans, and at the same time, leave the final decision to the optimizer. If we expect potentially different execution plans when p_n changes to the power of 2, then all we have to do is tell the database about it:

SQL> create or replace function get_data(
  2  	p_n	in number
  3  ) return sys_refcursor is
  4  	l_rf sys_refcursor;
  5  begin
  6  	open l_rf for 'select /* '||to_char(power(2, floor(log(2, p_n))))||' */ * from t where n<=:n' using p_n;
  7  	return l_rf;
  8  end;
  9  /

Function created

The above function will automatically add a comment into each SQL statement, embedding as many rows (to the power of 2) as we are expect:

SQL> variable rf refcursor;
SQL> exec :rf:=get_data(100);

PL/SQL procedure successfully completed.

SQL> print rf;

100 rows selected.

SQL> exec :rf:=get_data(25000);

PL/SQL procedure successfully completed.

SQL> print rf;

25000 rows selected.

SQL> select	s.sql_text,
  2  		max(p.operation||' '||p.options) keep (dense_rank last order by id) access_path
  3    	from v$sql s, v$sql_plan p
  4    	where s.SQL_ID=p.SQL_ID
  5  		and s.sql_text like 'select /* % */ * from t where n<=:n'
  6  	group by s.sql_id, s.sql_text;

SQL_TEXT                                 ACCESS_PATH
---------------------------------------- --------------------
select /* 16384 */ * from t where n<=:n  TABLE ACCESS FULL
select /* 64 */ * from t where n<=:n     INDEX RANGE SCAN

Since different comments will essentially lead to a different SQL statements, Oracle was able to develop appropriate plans.

This solution is very simple, and it provides the desired results. Its advantage is that, compared to replacing bind variables with literals, you still share most of your SQL while not sacrificing the quality of execution plans. Our function will result in only 17 different SQL statements for all possible values:

SQL> declare
  2   l_rf sys_refcursor;
  3  begin
  4   for i in 1 .. 100000
  5   loop
  6    l_rf:=get_data(i);
  7    close l_rf;
  8   end loop;
  9  end;
 10  /

PL/SQL procedure successfully completed.

SQL> select s.sql_text,
  2      max(p.operation||' '||p.options)
  3     keep (dense_rank last order by id) access_path,
  4    max(executions) execs
  5     from v$sql s, v$sql_plan p
  6     where s.SQL_ID=p.SQL_ID
  7      and s.sql_text like 'select /* % */ * from t where n<=:n'
  8     group by s.sql_text
  9   order by execs;

SQL_TEXT                                 ACCESS_PATH           EXECS
---------------------------------------- -------------------- ------
select /* 1 */ * from t where n<=:n      INDEX RANGE SCAN          1
select /* 2 */ * from t where n<=:n      INDEX RANGE SCAN          3
select /* 4 */ * from t where n<=:n      INDEX RANGE SCAN          4
select /* 8 */ * from t where n<=:n      INDEX RANGE SCAN          8
select /* 16 */ * from t where n<=:n     INDEX RANGE SCAN         16
select /* 32 */ * from t where n<=:n     INDEX RANGE SCAN         32
select /* 64 */ * from t where n<=:n     INDEX RANGE SCAN         64
select /* 128 */ * from t where n<=:n    INDEX RANGE SCAN        128
select /* 256 */ * from t where n<=:n    INDEX RANGE SCAN        256
select /* 512 */ * from t where n<=:n    INDEX RANGE SCAN        512
select /* 1024 */ * from t where n<=:n   INDEX RANGE SCAN       1024
select /* 2048 */ * from t where n<=:n   INDEX RANGE SCAN       2048
select /* 4096 */ * from t where n<=:n   INDEX RANGE SCAN       4096
select /* 8192 */ * from t where n<=:n   INDEX RANGE SCAN       8192
select /* 16384 */ * from t where n<=:n  INDEX RANGE SCAN      16384
select /* 32768 */ * from t where n<=:n  TABLE ACCESS FULL     32768
select /* 65536 */ * from t where n<=:n  TABLE ACCESS FULL     34464

17 rows selected.

Because the decision is still made by the optimizer, the above function will be able to automatically adjust to changing data volumes as well. If tomorrow our table contains 100M rows instead of 100K, our function will still be able to provide us good execution plans — which, by the way, might be different from what we see with 100K rows. This is a key difference from explicitly hinting the query.

Note that we still can have different execution plans inside a “transient” window. That should not be a problem, since this is where a plan switch should occur anyway, and doing the entire window one way or the other should not produce bad results. You can even go a bit further and make the base adjustable; this will allow us to properly balance SQL statements reusability while providing optimal execution plans:

SQL> create or replace function get_data(
  2  	p_n	in number,
  3  	p_f	in number default 2
  4  ) return sys_refcursor is
  5  	l_rf sys_refcursor;
  6  begin
  7  	open l_rf for 'select /* '||to_char(power(p_f, floor(log(p_f, p_n))))||' */ * from t where n<=:n' using p_n;
  8  	return l_rf;
  9  end;
 10  /

Function created

The above technique can be used virtually everywhere you have to have bind variables, and at the same time absolutely need different execution plans depending on bind variable values. You will only have to come up with a proper way to tell Oracle when you expect different plans, based on your domain knowledge and how your data is being used.

What About 11G?

There is a new feature introduced in 11G to help overcome the above problem — the ability to produce a different child cursor if the database detects a potentially suboptimal execution plan due to bind peeking. If you are interested how well it performs, then stay tuned for a next blog post.

What is “real world?”

Of course, the important question is whether I tested a feature in a way it was never designed to be used, or whether someone is just trying to make an excuse for poor scalability results by defining “real world” in a way that makes my tests inappropriate.

A new feature

What do you do, then, you first see a new feature? You read about it in the documentation, and then you test it in order to compare what you have read with what you have in reality.

What the documentation tells us

Open the Performance Tuning Guide and go to 7.3.1.4 Result Cache Concepts:

When these queries and functions are executed repeatedly, the results are retrieved directly from the cache memory. This results in a faster response time. The cached results stored become invalid when data in the dependent database objects is modified. The use of the result cache is a database-wide decision.

All it says is that you have to have repeatedly-executed functions and queries to get faster response time. It says nothing about what kind of queries or functions. It also suggests that the result cache should be used database-wide or shouldn’t be used at all (which is perfectly sound according to Jonathan Lewis’s Rules for Hinting).

Now skip up to 7.3.2.7 Use of Result Cache:

OLTP applications can benefit significantly from the use of the result cache. The benefits highly depend on the application. Consider the use of the PL/SQL function result cache and the SQL query result cache when evaluating whether your application can benefit from the result cache.

It clearly says that result cache is perfectly appropriate for OLTP applications. They leave a backdoor with the words, “depend on the application” but, yet again, they say nothing about what kind of OLTP applications.

Am I reading something wrong there?

Now, I may well be misunderstanding something, but I got the following impressions about the result cache after reading the documentation:

• Its use is recommended for frequently-executed functions and queries.
• It is to be used database-wide or not used at all.
• it is highly recommended for OLTP applications.

The above doesn’t leave the impression that the result cache is not appropriate for caching small queries and functions being executed at a very high rate with high degree of concurrency (yes, I am spelling out the basic definition of an OLTP system right now). Moreover, the documentation actually suggests that it is recommended to use result cache in such circumstances.

What others are doing?

Let’s say there is still a chance that I misread documentation. Others must be doing something else, then. So I started to search around to see other examples.

I saw that Steven Feuerstein wrote On the PL/SQL Function Result Cache. You can spot the words “best practices” and “the real world” there. He even explains which kind of application he is simulating. This article was published in Oracle Magazine which means someone actually reviewed it before publishing and it shows good result cache usage.

What is the test about? It caches a function that does a single-row lookups into a table. I was doing exactly the same in my scalability tests with the exception that I was caching the query itself rather than a PL/SQL function that calls it. The results, though, are quite different. Of course, you can shave off more by caching a function call, and this article is able to demonstrate quite a bit of improvement, which is probably why no one has complained about this test being less than “real world” — despite the fact that it is doing essentially the same thing as mine did.

What about scalability?

The tests published in Oracle Magazine article lacks an important part: they didn’t test concurrency. If you read my blog posts regarding result cache, you already know that it is not the pure speed of getting something out which is a problem — it is concurrency that kills the result cache.

If I am right about the result cache not being scalable, I should see the same results. So I took the same test case and tested it in a concurrent environment (we’re after “real world” here). Here is how I created an employees table:

SQL> create table employees as
  2  	select level employee_id, rpad('x', 100, 'x') v
  3  		from dual
  4  		connect by level <= 10000;

Table created

SQL> alter table employees add constraint pk_employees primary key (employee_id);

Table altered

Note that I am already giving the result cache an artificial advantage by not placing this table into a hash cluster (and a hash cluster is perfectly acceptable, considering test case description).

After the employees table, I created two functions in exactly as in the Oracle Magazine article — one that does result caching, and another that does not. I ran my tests in exactly the same way as I did before — parallel job processes executed the function using a random employee_id. I measured how many executions per second we were able to archive across 1 million executions. I repeated every test twice, and picked up the best result. I also made sure both runs provided consistent results. As before, all tests were performed on an 8-way Itanium server. Here is what I got:

And a nice graph:

I hope you guessed which line is red.

And if we draw a graph out of “% linear”:

I wonder if anyone still says that the result cache scales. Because it simply doesn’t.

Of course, you may invalidate my tests by saying that, because everyone lives in a single-user environment, concurrent tests are not “real world”.

So what now?

I understand Tom’s point about my using result cache in a way he doesn’t envision it. Let’s not forget, however, that I wrote my blog post back in November 2007, and back then, all we had was a standard Oracle Documentation. I see my tests as perfectly in accord with what the documentation says (and you can actually see others using result cache as I did). And instead of discarding all these tests by simply “adjusting” what “real world” means, I thought I could better spent my time by clarifying these issues in the documentation.

What are the causes of unstable plans due to bind variable peeking?

There are three things that might put you at risk of unstable plans due to bind variable peeking. Those are histograms, partitions, and range-based predicates. I’ll cover last two in upcoming blog posts.

Histograms

Let me share with you my déja  vu. When I see this:

SQL> select value
  2  	from v$parameter
  3  	where name='_optim_peek_user_binds';

VALUE
--------------------
FALSE

I immediately expect this:

SQL> select	sum(case when max_cnt > 2 then 1 else 0 end) histograms,
  2  		sum(case when max_cnt <= 2 then 1 else 0 end) no_histograms
  3  from (
  4  	select table_name, max(cnt) max_cnt
  5  		from (
  6  			select table_name, column_name, count(*) cnt
  7  				from dba_tab_histograms
  8  				group by table_name, column_name
  9  		) group by table_name
 10  );

HISTOGRAMS NO_HISTOGRAMS
---------- -------------
      1169          2494

The above is an example from a real-world OLTP system running with bind peeking disabled. It is no surprise to me. An exception, you say? Here’s another one . . .

SQL> select value
  2  	from v$parameter
  3  	where name='_optim_peek_user_binds';

VALUE
--------------------
FALSE

SQL>
SQL> select	sum(case when max_cnt > 2 then 1 else 0 end) histograms,
  2  		sum(case when max_cnt <= 2 then 1 else 0 end) no_histograms
  3  from (
  4  	select table_name, max(cnt) max_cnt
  5  		from (
  6  			select table_name, column_name, count(*) cnt
  7  				from dba_tab_histograms
  8  				group by table_name, column_name
  9  		) group by table_name
 10  );

HISTOGRAMS NO_HISTOGRAMS
---------- -------------
       304           521

Here comes my déja vu. If you tell me that you disabled bind peeking, my immediate response will be “do you have a lot of unnecessary histograms?” On an OLTP system, there is no way that you need histograms on a third of your tables (and I can hardly think of any DSS system where this amount of histograms can be justified).

How histograms and bind variable peeking can cause an unstable plan

I’ll give you a simple example:

SQL> create table t as
  2  	select case when level <= 9900 then 0 else level-9900 end n,
  3  			rpad('*', 100, '*') v
  4  		from dual
  5  		connect by level <= 10000;

Table created

SQL> create index i_t_n on t (n);

Index created

SQL> exec dbms_stats.gather_table_stats(user, 't', method_opt => 'for columns n size 254', cascade => true);

PL/SQL procedure successfully completed

In other words, we have a table T with 10000 rows, where 9900 rows have N=0 and 100 rows have N from 1 to 100. The histogram tells Oracle about this data distribution:

SQL> select column_name, endpoint_number, endpoint_value
  2  	from user_tab_histograms
  3  	where table_name='T';

COLUMN_NAM ENDPOINT_NUMBER ENDPOINT_VALUE
---------- --------------- --------------
N                     9900              0
N                     9901              1
N                     9902              2
N                     9903              3
N                     9904              4
N                     9905              5
...skipped for clarity...
N                     9997             97
N                     9998             98
N                     9999             99
N                    10000            100

101 rows selected

The above is known as a skewed data distribution, and a histogram can help Oracle to choose the right plan, depending on the value:

SQL> set autot traceonly explain
SQL> select * from t where n=0;

Execution Plan
----------------------------------------------------------
Plan hash value: 1601196873

--------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |  9900 |   531K|    42   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| T    |  9900 |   531K|    42   (0)| 00:00:01 |
--------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("N"=0)

SQL> select * from t where n=1;

Execution Plan
----------------------------------------------------------
Plan hash value: 2912310446

--------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |     1 |    55 |     2   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T     |     1 |    55 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_T_N |     1 |       |     1   (0)| 00:00:01 |
--------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("N"=1)

That is, Oracle is able to choose an index range scan when N=1 (returns only one row) and do an FTS when N=0 (returns 9900 rows) — perfect execution plans given the conditions. What does this have to do with bind peeking you ask? Well, imagine that you’ve submitted the following query:

select * from t where n=:n;

We used a bind variable in place of a literal. On a hard parse, Oracle will peek at the value you’ve used for :n, and will optimize the query as if you’ve submitted the same query with this literal instead. The problem is that, in 10G, bind variable peeking happens only on a hard parse, which means that all following executions will use the same plan, regardless of the bind variable value. This is easy enough to demonstrate:

SQL> variable n number;
SQL> exec :n:=0;

PL/SQL procedure successfully completed.

SQL> set autot traceonly stat

SQL> select * from t where n=:n;

9900 rows selected.

Statistics
----------------------------------------------------------
        982  recursive calls
          0  db block gets
        951  consistent gets
          0  physical reads
          0  redo size
     106664  bytes sent via SQL*Net to client
       7599  bytes received via SQL*Net from client
        661  SQL*Net roundtrips to/from client
         12  sorts (memory)
          0  sorts (disk)
       9900  rows processed

SQL> exec :n:=1;

PL/SQL procedure successfully completed.

SQL> select * from t where n=:n;

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
        155  consistent gets
          0  physical reads
          0  redo size
        476  bytes sent via SQL*Net to client
        350  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

You can see that both executions used the same cursor with a full table scan:

SQL> select sql_id, executions, child_number
  2   from v$sql
  3   where sql_text = 'select * from t where n=:n';

SQL_ID        EXECUTIONS CHILD_NUMBER
------------- ---------- ------------
g2n32un6t1c55          2            0

SQL> select * from table(dbms_xplan.display_cursor('g2n32un6t1c55'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------
SQL_ID  g2n32un6t1c55, child number 0
-------------------------------------
select * from t where n=:n

Plan hash value: 1601196873

--------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |    42 (100)|          |
|*  1 |  TABLE ACCESS FULL| T    |  9900 |   531K|    42   (0)| 00:00:01 |
--------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("N"=:N)

18 rows selected.

Both queries would use an index range scan if I had started my example with exec :n:=1 instead of exec :n:=0.

Now, if 99% of your queries are not interested in querying the above table with :N:=0, then you want the plan with an index range scan because it will provide optimal performance most of the time, while resulting in suboptimal performance for only 1% of executions. With the histogram in place, one day you will be unlucky enough to have Oracle hard parse the query with a bind variable value of 0, which will force everyone else to use an FTS (as was demonstrated above), which in turn will result an abysmal performances for 99% of executions (until the next hard parse when you might get lucky again). And if you have a system where a third of the tables have histograms on them then — I think you probably get the idea now.

What to do?

Well, just get rid of any histogram that does nothing but messes up your execution plans. That’s easy enough:

SQL> exec dbms_stats.gather_table_stats(user, 't', method_opt => 'for columns n size 1', cascade => true);

PL/SQL procedure successfully completed

Oracle will no longer have distribution information, and you’ll get the same plan regardless of the value:

SQL> select * from t where n=0;

Execution Plan
----------------------------------------------------------
Plan hash value: 2912310446

--------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |    99 |  5445 |     3   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T     |    99 |  5445 |     3   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_T_N |    99 |       |     1   (0)| 00:00:01 |
--------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("N"=0)

SQL> select * from t where n=1;

Execution Plan
----------------------------------------------------------
Plan hash value: 2912310446

--------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |    99 |  5445 |     3   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T     |    99 |  5445 |     3   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_T_N |    99 |       |     1   (0)| 00:00:01 |
--------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("N"=1)

No more surprises! This also means that you’ll get the same plan regardless of the bind variable value during a hard parse. However, the real question is . . .

What is making all those histograms I didn’t ask for?

It is a default behavior. With the declaration of RBO’s obsolescence in 10G, we were also presented with a default gather stats job in every 10G database. This jobs runs with a whole bunch of AUTO parameters, but one parameter is of particular interest to us:

SQL> select dbms_stats.get_param('method_opt') method_opt from dual;

METHOD_OPT
--------------------------------------------------------------------
FOR ALL COLUMNS SIZE AUTO

The SIZE ... part controls histograms collection. You can get the definition of AUTO in the Oracle documentation:

AUTO: Oracle determines the columns to collect histograms based on data distribution and the workload of the columns.

Well, as it says — Oracle will decide on which columns to collect histograms. Not you. As a result, every day when this jobs runs, you might be presented with something new to make your life more interesting.

Why SIZE AUTO is doing such a bad job?

Because coming up with optimal parameters for statistic gathering involves many more variables than the DBMS_STATS package can ever have. As a result, AUTO tends to be a “lowest common denominator”.

For example, if you have an OLTP system, chances are you don’t need histograms at all (apart from on a couple of tables, maybe). But how can DBMS_STATS know which type of a system do you run? OLTP, DWH, or mixed? Or maybe you’re doing OLTP five days per week and DWH on a weekend? DBMS_STATS tries to use some heuristics to come up with an answer, but that’s why two days of DWH on a weekend can completely screw up your OLTP activity for the rest of the week. DBMS_STATS just doesn’t have enough information. We humans do.

What could we use instead of AUTO?

Because to the above, there is no single answer since it involves “know your data” and “apply your domain knowledge”. However, there is one option that works particularly well for most environments. I’m talking about REPEAT.

REPEAT: Collects histograms only on the columns that already have histograms.

That is, Oracle will no longer make histograms you didn’t ask for. This will be your first step in stabilizing your bind peeking behaviour:

SQL> exec dbms_stats.set_param('method_opt', 'FOR ALL COLUMNS SIZE REPEAT');

PL/SQL procedure successfully completed

What to do with existing histograms?

Dealing with them depends on the situation. Chances are, however, you have many more histograms than you’ll ever need. That means that starting from scratch to figure out when do you need histograms is usually a much simpler task compared to the clean-up of existing onces. If this is your case, then it might be a good idea to wipe out all histograms in a database (gather your stats with FOR ALL COLUMNS SIZE 1 clause), and manually add them when you decide that you really need one.

The number of times I have had to go back to add histogram after a complete wipe out is surprisingly low — much lower than number of surprises histograms were causing on these systems, and those systems never had any bind peeking surprises due to the excessive amount of unnecessary histograms.

Are there any other sources of unnecessary histograms in my system?

Absolutely. People are the next source of unnecessarily histograms. It constantly surprises me how many people treat histograms as a kind of a silver bullet. For example, someone recently communicated to me that select count(*) from table was running slow, and added that, “maybe we should collect histograms on that table.” How on earth will a histogram help you to run this query faster? Many histograms on your system might be a result of a complete misunderstanding how histograms work, what they do, and, more important, what they do not.

I have a misbehaving query due to incorrect peeking caused by histogram, and I need to fix it right now. What do I do ?

Don’t hurry to flush your shared pool. First, as a result of a complete brain cleaning, your instance will have to hard parse everything it had in the library cache, causing tons of redundant work. Second, and this is much more important, these hard parses might well result in an incorrect peeking for some other queries. So you might end up in a worse situation than you were in before.

The right way is to get rid of a histogram by collecting stats on required table, with METHOD_OPT => 'FOR COLUMNS X SIZE 1' and NO_INVALIDATE => FALSE. This will cause all dependent cursors to be invalidated immediately after stats have been gathered.

Sometimes, however, you don’t have enough time to understand what caused a problem (or you simply don’t have time to regather the stats) and, if probability theory is on your side (chances for a good peeking are much higher), all you have to do to invalidate dependent cursors is to create comment on a table:

SQL> select n from t where n=:n;

         N
----------
         1

SQL> select executions, invalidations
  2   from v$sql
  3   where sql_text = 'select n from t where n=:n';

EXECUTIONS INVALIDATIONS
---------- -------------
         2             0

SQL> comment on table t is '';

Comment created.

SQL> select n from t where n=:n;

         N
----------
         1

SQL> select executions, invalidations
  2   from v$sql
  3   where sql_text = 'select n from t where n=:n';

EXECUTIONS INVALIDATIONS
---------- -------------
         1             1

The above will invalidate cursors that depend only on a specific table, thus significantly decreasing the risk of side effects.

Isn’t that much more work compared to simply turning off bind variable peeking?

Actually it is not. If you are starting a new system, all you have to do is to modify the default parameter from AUTO to REPEAT and you are done. You’ll have to create all required histograms manually but that’s our intentional goal: to do histogram creation in a controllable and predictable manner.

For existing systems that are plagued by gazillions of histograms, you’ll have to figure out what to do. Wiping out histograms for entire database will do for queries with bind variables pretty much the same as turning off bind peeking. I still think, however, that disabling bind peeking in this situation is the wrong choice, since (a) you still run the risk of getting unpredictable results from queries with literals; and (b) you will be doing tons more work during statistics collection, since histogram computation is expensive.

Is there anything else wrong with disabling bind peeking?

Running Oracle Database with an underscore parameter makes you different from the rest of the world, and this is not how Oracle Database was tested and intended to be run in a first place (think bugs). While disabling bind peeking seems to be relatively safe, it also very easy to avoid doing so.

Get histograms under control

Making sure new histograms appear in a controllable and predictable manner will be your first step in building a predictable environment.

Please note that today’s numbers are different since I’m using an entirely different hardware platform. While the four-way tests were done on a 2.4GHz Core 2 Quad box, today’s eight-way tests were done using four dual core Itanium 2 CPUs running at 1.1GHz.

Let’s take a look at the results:

I made a nice-looking graph from this:

The performance drops are quite dramatic. While going from one to two processes can bring us an additional 13430 RC lookups per second, going from seven to eight processes gives us only 2560.

And stats regarding Result Cache: Latch:

Here’s another pretty graph, this time of Result Cache: Latch wait time:

As you can see from the above figures, it only takes six concurrently-running processes before we start observing major issues regarding the Result Set: Latch contention.

SQL> alter database recover managed standby database finish skip standby logfile;
alter database recover managed standby database finish skip standby logfile
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-01110: data file 1: '/oradata/stage/datafile/system_01.dbf'
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '/oradata/stage/datafile/system_01.dbf'
ORA-01130: database file version 9.2.0.0.0 incompatible with ORACLE
version 9.2.0.0.0

Database file version 9.2.0.0.0 is incompatible with ORACLE version 9.2.0.0.0, is it?