I am not sure how much sense all this will make without my comments. We may do it in a webinar if there is sufficient interest. Regardless I will probably be doing it again at some point in the future.

Here’s a serious performance bug that’s been in Oracle for a while, and finally there are hints of it been fixed, but only partially.

The bug happens when performing sorting operations in parallel, and the source data is already well sorted. The “ranger” doesn’t do a good job of properly assigning row ranges to sorter processes, and ~90% of the rows end up being sent to the same parallel process, regardless of the level of parallelism. So even if you have 256 CPUs, only about 10% performance improvement is achieve, instead of a factor of your parallelism when running the query in parallel.

For example, if the non parallel sort/index creation took 45 minutes, running with parallel 32 will take 41 minutes, instead of the possible 1.4 minutes (assuming you have sufficient horsepower).

When running a “Sort” operation in parallel, there are 2 sets of parallel processes. Producers and consumers. The number of producer/consumer pairs depends on your parallelism settings. So in the case of parallel 32 there are 32 producers and 32 consumers. This is well illustrated in the Oracle documentation HERE (Oracle web based documentation down, will update later). As each ‘producer’ is reading data it is sending it to the appropriate consumer for that “range”. For example, consumer 1 takes A-B, consumer 2 takes C-F, consumer 3 G-L and etc. The exact split is dynamically calculated by the “Ranger” process. Unfortunately it doesn’t work so good with sorted data.

The same applies to index creation. Index creation is basically a big sort, followed by writing out the result set into a B-Tree structure. Index creation suffers from the exact same ranging issues, at least until 11.2.

Here’s an example:

-- Create mini-sample table
create table mytest_s as select rownum r from dual connect by level <=400000;

-- Fetch only 1 row, no need to fetch all
begin
  for c in ( select /*+PARALLEL(t,4)*/ * from mytest_s t order by 1) loop
    exit;
  end loop;
end;
/
select dfo_number "d", tq_id as "t", server_type, num_rows,rpad('x',round(num_rows*10/nullif(max(num_rows) over (partition by dfo_number, tq_id, server_type),0)),'x') as "pr", round(bytes/1024/1024) mb,  process, instance i,round(ratio_to_report (num_rows) over (partition by dfo_number, tq_id, server_type)*100) as "%", open_time, avg_latency, waits, timeouts,round(bytes/nullif(num_rows,0)) as "b/r"
from v$pq_tqstat order by dfo_number, tq_id, server_type desc, process;

d t SERVER_TYPE   NUM_ROWS pr           MB PROCESS I    %  OPEN_TIME AVG_LATENCY   WAITS   TIMEOUTS  b/r
- - ----------- ---------- ----------- --- ------- - ---- ---------- ----------- ------- ---------- ----
1 0 Ranger             372 xxxxxxxxxx    0 QC      1  100          0           0       0          0   11

1 0 Producer        126144 xxxxxxxxxx    1 P012    1   32          0           0      19          2    6
1 0 Producer         47304 xxxx          0 P013    1   12          0           0       8          1    6
1 0 Producer        110376 xxxxxxxxx     1 P014    1   28          0           0      17          2    6
1 0 Producer        116176 xxxxxxxxx     1 P015    1   29          0           0      16          0    6

1 0 Consumer          7885               0 P008    1    2          0           0       6          1    5
1 0 Consumer          7884               0 P009    1    2          0           0       6          1    6
1 0 Consumer          7884               0 P010    1    2          0           0       7          2    6
1 0 Consumer        376347 xxxxxxxxxx    2 P011    1   94          0           0      16          4    6

1 1 Producer          5536 xxxxxxxxxx    0 P008    1   55          0           0     577        568    3
1 1 Producer          4508 xxxxxxxx      0 P011    1   45          0           0     587        571    4

1 1 Consumer           100 xxxxxxxxxx    0 QC      1  100          0           0       1          0  161

As you can see from this test case, the sorter processes (4 consumers) had a very uneven split, with 94% of the rows been sent to only one consumer. I tested this case with parallel 64, and in that case 90% gets sent to 1 consumer, with the other 10% evenly distributed on the remaining ones.

This essentially reduces your execution time by at most 10%.

A very similar thing happens if you create an index. I did, however, catch an anomaly in the test case. With parallel 4, the distribution is 66/34/0/0, while with parallel 8, it’s 100/0/0/0/0/0/0/0. I.e., terrible as all the work will be performed by 1 process, absolutely the same as Serial, only a little worse as there will be inter-process communication.

create index on mytest_s (r) parallel 4;
select dfo_number "d", tq_id as "t", server_type, num_rows,rpad('x',round(num_rows*10/nullif(max(num_rows) over (partition by dfo_number, tq_id, server_type),0)),'x') as "pr", round(bytes/1024/1024) mb,  process, instance i,round(ratio_to_report (num_rows) over (partition by dfo_number, tq_id, server_type)*100) as "%", open_time, avg_latency, waits, timeouts,round(bytes/nullif(num_rows,0)) as "b/r"
from v$pq_tqstat order by dfo_number, tq_id, server_type desc, process;

SERVER_TYPE   NUM_ROWS pr          MB PROCESS  I   %  OPEN_TIME AVG_LATENCY      WAITS   TIMEOUTS        b/r
----------- ---------- ----------- -- -------- - --- ---------- ----------- ---------- ---------- ----------
Ranger              12 xxxxxxxxxx   0 QC       1 100          0           0          1          0       3974
Producer        132601 xxxxxxxxxx   2 P004     1  33          0           0         19          1         18
Producer         95265 xxxxxxx      2 P005     1  24          0           0         14          2         18
Producer         95265 xxxxxxx      2 P006     1  24          0           0         14          1         18
Producer         79497 xxxxxx       1 P007     1  20          0           0         11          0         18

Consumer        262308 xxxxxxxxxx   4 P000     1  66          0           0         76         73         18
Consumer           164              0 P001     1   0          0           0         77         74         19
Consumer           164              0 P002     1   0          0           0         76         73         19
Consumer        137364 xxxxx        2 P003     1  34          0           0         76         73         18

Producer             1 xxxxxxxxxx   0 P000     1  25          0           0          0          0        322
Producer             1 xxxxxxxxxx   0 P001     1  25          0           0          0          0        322
Producer             1 xxxxxxxxxx   0 P002     1  25          0           0          0          0        322
Producer             1 xxxxxxxxxx   0 P003     1  25          0           0          0          0        322
Consumer             4 xxxxxxxxxx   0 QC       1 100          0           0          1          0        322

create index on mytest_s (r) parallel 8;

SERVER_TYPE   NUM_ROWS pr            MB PROCESS  I   %  OPEN_TIME AVG_LATENCY      WAITS   TIMEOUTS        b/r
----------- ---------- ------------- -- -------- - --- ---------- ----------- ---------- ---------- ----------
Ranger               0                0 QC       1              0           0          6          2 

Producer         47304 xxxxxxxx       1 P008     1  12          0           0         14          2         18
Producer         51246 xxxxxxxxx      1 P009     1  13          0           0         15          1         18
Producer         47304 xxxxxxxx       1 P010     1  12          0           0         15          2         18
Producer         45220 xxxxxxxx       1 P011     1  11          0           0         15          2         18
Producer         59130 xxxxxxxxxx     1 P012     1  15          0           0         17          3         18
Producer         35478 xxxxxx         1 P013     1   9          0           0         10          3         18
Producer         55188 xxxxxxxxx      1 P014     1  14          0           0         16          1         18
Producer         59130 xxxxxxxxxx     1 P015     1  15          0           0         17          1         18

Consumer        400000 xxxxxxxxxx     7 P000     1 100          0           0         52         49         18
Consumer             0                0 P001     1   0          0           0         52         49
Consumer             0                0 P002     1   0          0           0         52         49
Consumer             0                0 P003     1   0          0           0         52         49
Consumer             0                0 P004     1   0          0           0         52         49
Consumer             0                0 P005     1   0          0           0         52         49
Consumer             0                0 P006     1   0          0           0         52         49
Consumer             0                0 P007     1   0          0           0         52         49 

Producer             1 xxxxxxxxxx     0 P000     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P001     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P002     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P003     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P004     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P005     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P006     1  13          0           0          0          0        322
Producer             1 xxxxxxxxxx     0 P007     1  13          0           0          0          0        322

Consumer             8 xxxxxxxxxx     0 QC       1 100          0           0          2          1        322

To further explore the implications of this bug, I created a more elaborate test case. I created several types of data, and tested ordering against each “class.”

create table mytest as
select rownum pk, trunc(dbms_random.value(0,400000)) rnd,
floor(rownum/1000) type_1000, floor(rownum/5) type_5,
mod(rownum,1000) type_1000mod,mod(rownum,5) type_5mod,mod(rownum,10009) type_10000mod,
sysdate-rownum/100 as dt,
rpad('x',10,'x') pad from dual connect by level <=400000;

To spare you some of the testing, here are my results. Query ran:
select /*+PARALLEL(t,[DEGREE])*/* from mytest order by [ORDER BY];

A few quick conclusions:

• The first column of order matters for distribution.
• If the Order by column has repeated values, PQ sort will be limited by number of distinct values, but only if they are not grouped together.
• Index creation on time series (log table, stock table) is slow, “type” indexes are slow.
• If you reorder the keys of an index, you may affect the time it takes to create it.
• Follow up on the preceding, especially true if you try to put low cardinality columns first to improve compression.

One could dispute in which cases data in the table is ordered, and it’s amazing how many cases this is:

• Time series data — ever-growing data. The PK is ordered; the “insert date” is somewhat ordered.
• Data warehouses — bulk load files are often ordered via some conditions.
• Sometimes it is good to reorder a table, to improve data locality and compression in data warehouses. This can, however, have negative effects on index build time.
• Sometimes one of the intermediate steps will return an ordered set for the final processing.

One example of the last type, is analytics. But that’s for a separate blog post.

And finally, to end on an optimistic note, it appears that 11.2 has the index creation issue resolved, but the order by in queries is still bad.

Oracle 11g has improved this behaviour by storing the default value in the metadata and making the column addition instantaneous. An example of this feature at work can be seen in 11g Enhancement for ALTER TABLE .. ADD COLUMN Functionality and some bugs regarding sysdate, as pointed out in the comments.

Although this is a welcomed enhancement, there are some unexpected aspects beyond the basic operations.

First, we know default values for new columns are stored in the metadata, but what happens when you change the default?

create table mytest (a number); -- instant
insert into mytest (a) select rownum from dba_objects; -- 10 seconds
alter table mytest add b varchar2(2000) default rpad('x',1000,'x') not null; --instant

And now the test:

alter table mytest modify b  default rpad('Z',1000,'Z') ;-- instant !

Instant! If we query the table what are we going to see? The correct, ‘xxxxx’ value or the new ‘ZZZZZ’ value? How does it work?

Obviously Oracle has to behave properly. So querying the table shows correctly the already-initialized “xxxx” value. Through experimentation, I have determined that this enhancement works as follows:

There are two default values for each column:

• one default expression for “not null with default value” columns that do not have a value at the block level
• another default value for all new inserted records

I’ll spare you all the test cases, but based on the above mechanics, we can observe the following not-so-obvious behaviours:

• Rebuilding the table with “alter table move” can (and probably will) make it bigger, as all the defaults are initialized at the block level.
• Updating other columns will not initialize the default value, thus does not grow the table.
• Changing the column constraint to “null” (from not null) will initialize the default values, causing a massive update and locking of the table
• Some “magic” can be applied to large fact tables by adding, instead of creating them with default values for very popular values, but that can be a nightmare to maintain.

Virtual columns may present an innovative space-saving approach by using nvl for default values. Of course, it works differently, but for very large tables, it may be of significant advantage. For example:

create table mytest ( colA_data varchar2(30), colA as (nvl(colA_data, 'DEFAULT VALUE')));

The default value will not consume any space (except 1 byte for ‘null’ value), ever.

Battle of the Nodes: RAC Performance Myths — Riyaj Shamsudeen
A great presentation on popular RAC myths, with some great examples. Excellent visuals that made complex processes look simple. I really liked this one.

Getting the Most Out of AWR — Tim Gorman
A first-rate session attended by a lot of the conference. It went into detail on what scripts are available to extract AWR information without needing Grid Control or Database Control. For command-line lovers, it’s great.

The SAN is guilty… until proven otherwise — Gaja Krishna Vaidyanatha
A very important session for all DBAs, showing the end-to-end components involved in database I/O. There are so many more components that can cause problems between the database and the physical spindles. Concepts, case studies, plenty of information.

Understanding Oracle Execution Plans: How SQL is Really Executed – Tanel Poder
One of those eye-opening sessions, starting with how to read SQL Execution Plans, and moving to showing stack traces and mapping function calls, to Execution Plan steps. A must-see for everyone tuning SQL.

And that is it. As exhausting conferences are, I always wish for them to have been longer.

There were quite a few interesting presentations. Unfortunately, the very first I went to was canceled due to car trouble. I also found that several sessions of similar interest to me overlapped, so I had to choose my spots.

Advanced Oracle Troubleshooting
This presentation was particularly good. Tanel goes into detail on how to quickly asses a situation without going through a number of “health checks” and still be nowhere near solving the problem. His approach is to look directly at what a “hanging” session is waiting on, and to systematically determine the cause of the problem, with no time wasted.

Putting your database on a Diet: Oracle’s Data compression
A short overview of table compression. I found that that even though the presenter obviously had some experience with compression, there were hardly any examples nor anything mentioned about how to determine proper re-ordering to improve compression.

All About Oracle’s In-Memory Undo
An unusual topic—something that works so well that no one really talks about it. The presentation, however, was very short, and provided little new information. There was only one demonstrated test case. Although it went into detail about the difference between in-memory and standard undo, the other-than-obvious effects were omitted.

During lunch I took a picture that shows the entire RMOUG crowd:

Tomorrow is Day 2, and I will be posting about it here.

I am quite excited, as the RMOUG schedule (PDF) looks quite promising, especially these presentations:

• Further RMAN Optimizations in 11g — Stephan Haisley
• Advanced Oracle Troubleshooting: No Magic is Needed — Tanel Poder
• Understanding Oracle Execution Plans: How SQL is Really Executed — Tanel Poder
• The SAN is Guilty until proven otherwise — Gaja Krishna Vaidyanatha

Some of these overlap, so I guess I will have to make a difficult choice.

I hope to see you all in Denver.

The audio quality is a little spotty here and there, so you might like to follow the transcription below.

Paul gets the interview started.

Paul Vallée (PV): Christo Kutrovsky and myself, Paul Vallée. We’re on the line with Kevin Closson of Oracle (and prior to that with Hewlett-Packard, and prior to that with Polyserve, and prior to that with Sequent). A giant of our industry, and I’m honoured to be speaking to him. Kevin, hello.

Kevin Closson (KC): Well, they always say that flattery gets you nowhere, but apparently it’ll get you on the phone.

PV: [laughs] Very nice!

KC: No seriously, it’s more than a pleasure to be here. I like what you guys do, so this is good.

PV: Thank you, Kevin. So, we are here to talk about the work that Larry Ellison announced yesterday, specifically the work around the Oracle Database Machine and the Exadata Storage Server. Kevin, can you just quickly introduce yourself and how you came to be involved in the project?

KC: Right. So, I’m a performance architect with Oracle, and the project that I’m stationed on, if you will, is the development team for Oracle Exadata Storage Server. And the way I came to Oracle is, quite a few of the folks who are involved with the very genesis of Exadata are people that I’ve known and worked with closely dating back to the early ’90s. And after a fruitful endeavour as the chief software architect for Oracle solutions at Polyserve, it became an opportunity to latch onto Oracle, because we sold our company to them. So there we are.

PV: How exciting! Congratulations! So I noticed that there’s still a little, I guess a diversion in terms of the branding. Larry definitely introduced it as the Exadata Programmable Storage Server, and I double-checked the video. But in your blog, you’re calling it, for sure, just the Exadata Storage Server. Just how recently was the marketing/messaging developed for this?

KC: You know, I’m not a part of the Go-To-Market (GTM) efforts, but, you know, honestly, the way these things are brought to market . . .  They’re developed under a project name, and the project name remains the same for years. It was over the last few months that Marketing began cooking the name and what-have-you. Now, if you’re referring to something that Larry said in his keynotes, I have to admit I didn’t commit to photographic memory all the slides. And certainly, if he used the term “programmable”, I’m not going to correct Larry Ellison.

PV: [laughs] That would be risky.

KC: Having said that, I’m here to tell you that if somehow the word “programmable” sticks and becomes pervasive, it’ll be misleading. Because the connotation of “programmable” is akin to what’s possible on products like Netezza where there’s real programmable data rates that on-site people can fiddle with. And that’s not the nature of the Exadata Storage Server. The Exadata Storage Server is programmable by us, the folks on the development team, and it comes to you in the form of software, and occasional lower-level firmware fixes from HP. I hope that answers the question—safely for me—and for you as well.

PV: Sounds good. So, I must say—just to go on the record—that this is not at all what I thought you might be cooking up, but by the same token — and obviously the evidence for that is all over the Pythian blog, where we had a speculative blog entry about what might be in-store. But that being said, I am tremendously impressed and really excited, and already getting to work on identifying which Pythian customers have a use-case for the technology, and trying to get a good early-adopter case to co-sell with Oracle. So I’m that jazzed about it, and definitely, congratulations are in order.

KC: Well, I’m glad to hear that, and I appreciate congratulations. You know, when it comes down to it, this is a new product that solves problems that cannot be solved with any other storage technology. Of course, everybody’s already crafted their position papers and will argue the fact that perhaps the speeds and feeds are different than them. But what I just said, I stand by, full-stop. You cannot solve this problem with any other technology. And what I mean by that is, in order to have Oracle as your commercial store for all of EP or ERP (which, you know — nobody’s going to argue [about] our footprint in that space), in order to do EWBI on that with someone else’s offering — you now have two vendors, you have two different types of technology. Exadata is a unifying storage platform for Oracle. And now Oracle is all.

PV: I would like to introduce Christo Kutrovsky, who is one of Pythian’s leading experts especially on how Oracle talks to storage — certainly not our only such expert given than Alex G. [Gorbachev] presented on “Under the Hood of the Oracle Clusterware”, just at Open World earlier this week. But that being said, this is a definite subject matter of interest for Christo, and he has prepared a couple questions for you Kevin.

KC: Oh good!

Christo Kutrovsky (CK): Hi Kevin, how are you?

KC: Hi Christo. Just fine. You and I have met in passing, so this is not a first meeting by any means.

CK: Oh yeah, absolutely. I admire your presentations — they are always to-the-point. I hope you keep up with that.

KC: Well thank you.

CK: So back to the questions and to the nitty-gritty stuff. I want to start with a couple of questions [about] the Infiniband implementation. Each Exadata instance is linked via 40Gb Infiniband link to the switch. Correct?

KC: Partially. So the numbers are off. Each Exadata cell—as well as all of the RDBMS hosts in the HP Oracle Database Machine—all of them are connected with two 20Gb paths to the switch.

CK: Okay. So does that mean that each database server can only accept 2*20 Gb of data?

KC: No. We have two paths of 20Gb of bandwidth. They are joined together in a bonded relationship, therefore only one path is active at one time. We don’t need more than one path, because arithmetically, over 2 gigabytes-per-second is not only more than a single two-socket, eight-core Xeon server can ingest on the RDBMS host, it’s most certainly more than a single Exadata storage cell can produce. So from a produce-and-consume perspective, we’re not starving anybody. So what’s failover if, for instance, the HCA happens to fail. It’s a redundancy purpose for the dual paths, but as the numbers work out, it’s over 2 gigabytes-per-second. With 20Gb paths, we’re not feeding something like a 128-core Superdome or something like that. In that particular case, the plumbing would be slightly different.

But look at what we’ve built. The HP Oracle Database Machine consists of VL360s. Each of those has the traditional Xeon processors in them, and they don’t have unlimited bandwidth, so they can ingest as much as even one single 20Gb path can provide.

CK: Okay. And, just to clarify on the [subject of] the paths internals — it’s like a switch, basically. Meaning that any two pairs can talk at full bandwidth without affecting other talking pairs. Is this correct?

KC: Yes. There’s no bottlenecks in the point-to-point meter. So if a cell is delivering I/Os to three or four of the RDBMS hosts, that’s happening concurrently in-flight with, let’s say, I/O requests from RDBMS hosts 6 and 7 talking [to yet] other cells. So there’s no queuing.

CK: Basically what I was trying to clarify is that — imagine you have one database server running one query, and you have two Exadata storage systems, that will pretty much saturate your bandwidth, assuming no filtering is happening on the Exadata. That means, if you have two database servers, each database server can talk at 20Gigiabits to two separate Exadata servers.

KC: Yes. Every point is 20Gb.

CK: Alright, I just wanted to clarify this. Those were my questions [about] Infiniband.

KC: Oh good, and along those lines, I should hope that that puts to rest any concerns about—as I always say—”plumbing.” We didn’t build any bottlenecks into this system.

PV: Just a comment for out listeners who aren’t familiar with Infiniband as a networking protocol—not only as a networking protocol, but also as a disk-access protocol—Infiniband has not only very high bandwidth characteristics, but also ultra-low latency characteristics, and that’s why it’s such a good choice for cluster solutions like this.

KC: And Infiniband support multiple communications protocols, and to that end, we’ve developed and brought to market the lightest—at least in our assessment—the lightest and most adaptable of all of them, which is Reliable Datagram Sockets [RDS]. Sure, you can do IP over Infiniband, and that starts to chew into some of the value propositions involving Infiniband, but we’ve done none of the sort. We are fully Remote Direct Memory Access (RDMA) from point to point over RDS.

CK: Alright. So continuing onto more details of how the system works, a question on OLTP. For OLTP systems, and considering Database Machines and Exadata, does OLTP benefit in any way other than a best-practices-built system.

KC: That’s such a fair question, but let me answer it this way. The design center for Exadata is providing uninhibited utilization of all of the bandwidth that all of the disks are capable of delivering. Workloads that require that generally look a lot more like Business Intelligence Data Warehousing. It’s not that common to see, strictly speaking, OLTP-style workloads suffering bottlenecks like traditional storage arrays—fibre-channel SANs, iSCSI, or even high-performance NAS.

That doesn’t mean, however, that there are no benefits for OLTP using Exadata, because indeed there are. Probably the most substantial benefit that OLTP-style workloads will derive from being serviced by Exadata is the fact that I/Os from the Oracle kernel during OLTP will no longer have to interface through standard C libraries and making system calls to instantiate and report completions of I/Os. Doing that has always been entirely too costly, in processor-cycles terms. All of the requests for I/O from all Oracle processes to Exadata are done from userland using Remote Direct Memory Access (RDMA) directly into the processes of the Oracle Storage Server. So there’s no queuing up and de-queuing as far as the sending and the delivery of I/O requests.

Now, what does all that mean to us? It means that if you’re doing thousands-upon-thousands of I/Os-per-second of OLTP size—let’s say 8 kilobytes—you’ll see the reduction in processor cycles lost just doing I/O, substantial. You can do well over 8,000 [or] 10,000 random I/Os per second with much less than five percent of all processor cycles spent in kernel-mode. So we start freeing-up substan— Oh, and doing so with traditional SAN host bus adapters can often cost as much as twenty percent of all processor cycles spent just doing I/O — not doing anything with the results of the I/O, which is why somebody bought the computer in the first place. [13:44]

We’re talking about relief on processor cycles, lost I/O—which I think should be substantial for a lot of people—and, as well as, we have a balanced system. You don’t have to worry about collusion of applications, for instance, on the same fibre channel arbitrated group, where your disks for OLTP reside. And those sorts of balance aspects. They do pay off.

CK: Excellent, very cool. Okay—moving on now, this brings me to my next question. What’s the 8 gigabyte memory on the Exadata cells? Is this useful [for] caching, or is it just working memory to manage the software running on the Exadata?

KC: Well, that’s a great question. And people keep saying “cache cache cache”, although unless you have as much cache as you have dataset, cache actually gets in the way. I’m talking specifically—in that case—about, the types of work, the types of I/O profiles you see with DW/BI. To that end, we do configure 8 gigabytes of RAM per Exadata cell, and your question was, what is that for. Well, each Exadata cell has an operating system kernel, and that is Oracle Enterprise Linux. And that shouldn’t surprise anybody, because every storage device that’s out there—all SAN arrays, all NAS filers—they all have operating systems in them as well. And that takes up a portion—let’s say for instance 1GB—and that leaves us something on the order of 7 gigabytes of working memory. The primary value proposition of Exadata is to service, to scans, the type of I/O profiles you see with DW/BIs, and we scan this using 1 Megabyte frees.

So, let’s say for instance, we’re trying to push through a gigabyte per second. Depending on how long the I/Os take, we need to be able to buffer on the order of 5,000 of those I/Os per second. So 5,000 1 Meg buffers is 5 gigabytes. That leaves us about 2 gigabytes of cushioning. I think you can see where I’m going with this. In order to handle thousands of I/Os per second at that size requires just buffering, and that different from cache, right? Buffering is the memory that you pin down so that the disks, through the drivers, can DMA to memory. And then from there, of course, Exadata will RDNA that result over Infiniband with RDS directly into the address space of the database server process. But you have to have some holding space. Buffers are reused immediately, so in that case, it’s not cache. Does that answer your question?

CK: Absolutely. I was kind of suspecting this is the case, but I just wanted to hear it from you. And I assume this is also used to facilitate filtering and joining and extrapolation of query data?

KC: We all love the fact that Exadata is brute-force, but it’s also brainy. So, if we’re pushing through a gigabyte of I/O per second, from disk through the Exadata server out onto Infiniband, in the meantime, we’re also applying our intelligence. And our intelligence is, we perform predicate filtration. So if you’re querying for rows of HR records where “salary is greater than a million dollars”—and I’m sure there’s a lot of folks [garbled] like that. In order to filter through that data, that buffer has to be held down by the Exadata server for the amount of time it takes to rip through there looking for the rows that match. And then after that, whatever columns are sighted in that query, we have to walk through each of the rows and pick out the columns and send them back. [17:43]

So the buffer stays pinned-down while we’re doing intelligent processing of the contents of the buffer. And then we wipe the buffer out and hammer it with another I/O. Did that make any sense? [17:53]

CK: Absolutely. Now that we’ve touched on the subject of filtering — do only parallel queries benefit from this filtering push down to the Exadata cells?

KC: Right. So I think what you’re asking is whether or not, somehow, OLTP operations will benefit from what we currently offload to storage. The answer is no, but if that wasn’t your question, go ahead and state it.

CK: Yeah. No, the question would be, can you run Standard Edition on the database servers?

KC: No. That’s a topic regarding licensing, but I happen to know that this is Enterprise Edition only.

CK: Okay. Maybe you know the answer to this: does the cost that Larry included on one of his slides include the RAC and partitioning options?

KC: I do recall that he put up a slide that compared on a per-terabyte basis the HP Oracle Database Machine to Teradata and Netezza.

KC: And built into that cost-per-terabyte was the cost for software. Because this is a pre-packaged and ready-to-go deal, and because you have to have Real Application Clusters and you have to have partitioning, the answer to that is, yeah it’s built-in to that cost. What that number is, I wouldn’t quote. I don’t deal with the money.

CK: Yeah, absolutely. And talking further about the cells. In one of the datasheets it’s mentioned that backups benefit from this, and that sheet mentions specifically incremental backups being processed and filtered by the cells. Which is pretty cool. Now the question is, is the cell CPU used for compression for the backups — when you do full backups, for example?

KC: That’s an excellent question. And there’s a lot of other functionality that’s offloaded to the cells—and I’m sure we’ll talk about that—specific to backup. The answer to your [question about the] compression aspect of backup is, no — cells do currently do the compression. Could they? Sure. We’ll talk about that in the future. So you still use host CPU to do the compression. Will that mean that, when it comes times to access compressed data, Exadata is helpless? The answer is no, because Exadata is able to do filtering and projection on data that is compressed. It understand enough about compressed data to be able to do that.

But back to the backup issue. The value proposition for backup is, if you’re doing something like, let’s say, an incremental backup, and you have to go through terabytes of data to find several blocks that have been changed before the beginning of the backup, they have to be backed up. Instead of troubling the RDBMS hosts to go looking for those blocks, what they do is send off a smart operation to Exadata cells, who then go and look for the blocks that are old enough to need to be backed up. So it’s offloading finding the blocks that need to be backed up. Did that make any sense?

CK: Absolutely. So, what other operations do we have? Incremental backups, tablespace creation, joining . . .  Does sorting also count? Can you offload sorting, grouping by? These are data warehouse operations that usually consume a lot of CPU. Can those be assisted by the CPU power from the Exadata cells?

KC: The answer to sorting and joining and grouping and aggregation . . .  The only joining that we do is very good technology. It’s very beneficial on so many queries (I intend to blog exactly about that). It’s a bloom filter join, and we can discuss that some other time. We don’t do hash joins in storage cells because cells would have to see all of the other cells’ data, and filter idempotent storage—pods, if you will—they don’t know about each other. But as far as storage sorting and aggregation — that’s too far upstream in the Oracle kernel. It’s too far up above where Exadata adapts. Exadata is a storage that happens to be able to do a few things that are intelligent. In future, perhaps — but at this point the answer to that is, no. [22:44]

CK: It seems like there is a lot of extra things that can be added to the cells — software, et cetera. Is the roadmap for cell upgrade similar to database? Like every two years or something like that? Has it been discussed at all?

KC: Although I know a little bit about the product, what I don’t know is what the release schedules are. That would be committed by someone clear outside of my group, for certain. Do I think that the software will rapidly evolve? I would say most certainly. This is the initial release of a product that has been in development for three years. What are the odds that we knew everything over three years that we’ll find in just a short period of time [garbled]? Pretty slim. We intend to be very aggressive—not chaotic, but aggressive—in enhancing Exadata.

CK: Disk failures. How are those handled? I know that disk failures will not affect data or running queries or anything like that. But I’m curious — when you lose a disk, a single disk in a single cell, where is this handled? Is this still something that is happening on the ASM level, is it something that is happening on the cell/disk level? Where does this happen? Who notices this and who cleans up?

KC: That’s an excellent question. Disks in storage cells are treated by ASM really no differently than disks out in a fibre channel SAN. So ASM will respond to failure the same way it does to a single-disk failure in a fibre channel SAN. So that’s a two-part answer. ASM will shield the database processes from knowing anything about that disk failure, as we expect it to because it runs on a fibre channel SAN.

Now, we haven’t [garbled] about the fact that you’ve got a physical disk failure, and a human being has to get involved at some point. Exadata is physical storage management. Gone are the days when you’d have to do fdisk and all of that sort of stuff. So from soup to nuts, we manage the physical disks. And that means, when a physical disk fails, you can get an alert via email (I suppose that could be an email to an SMS or what-have-you). You’ll know the disk has failed, and there will be enough information regarding the disk failure so that you can attach to the correct cell and interface with the cell command-line interface to execute a command that will vacate that disk from Exadata ownership. You put another physical disk in there, and there’s a very short command to bring that disk online. And, just like you would in a fibre channel SAN environment, you go and ask ASM to re-balance. [25:45]

CK: So ASM actually sees each individual disk on the Exadata cells?

KC: Yes. Every disk in Exadata is an ASM disk. In fact, and nobody’s been talking about this yet because we have to roll this out — we can’t just turn on the garden hose and hose everybody down.

The way this goes is, physical disks become, conceptually, to Exadata a cell disk, and so that’s a single physical disk and a single logical management-level disk. So a single physical disk is a single cell disk. You can take cell disks and carve them up into what we refer to as grid disks. If you took at 300GB SAS from our SAS option, if you took a 300GB SAS drive and created a groupdisk on it that was 100GB, you’ll get the outermost 100GB of space from the platter. Now we have a lot of shorthand for doing this. Don’t believe for a moment that it’s a bunch of laborious scripting commands. We’re very good about that. If, for instance you wanted to—just in two commands—create cell disks on all disks, and then create a set of grid disks for something called “data”, you would just simply say, create celldisk all [garbled] initialize cell (and I’m blogging about that syntax, you’ll see that soon) . . .  initialize cell, followed by create celldisk all, followed by create griddisk --prefix=data --size=100G, and at that point in time you’d be able to go over on the database hosts and ASM would be able to see those disks.

CK: It’ll be like a candidate on the ASM level?

KC: It would be a candidate. It would be ready to use. So at that point in time, you would have twelve ASM disks, each of 100GB.

CK: Oh, okay! So, basically, a cell grid is like a group command, not a virtualization. In the end it will present it as twelve different ones, but it will present it as a group?

KC: Actually, more to the point, a griddisk is an ASM-usable chunk of a celldisk. And if you create. let’s say, at one blast on a cell, you create twelve griddisks called “data” you would have griddisks named “data01″ through “data12″, and they would each be 100GB. And when you go and add them to a diskgroup, you wouldn’t have to list out all those happenings, you could use wildcard characters to say /0/*data* — and now you have a diskgroup that consists of twelve 100GB slices of celldisk. [28:55]

CK: In a way, you’re exporting the slices?

KC: Yes.

CK: Making the slices visible to ASM?

KC: Yes. griddisk to a cell is a logical object. griddisk to ASM treats it just as if it was a physical disk.

CK: And what about failure groups? Is this created automatically?

KC: No, you still use the age-old ASM incantations to create your failure groups. If you just create . . .  let’s say, for instance, you have the world’s smallest Exadata configuration, which is two cells. If you create a diskgroup of that, you would have the same normal redundancy, and it would create a failgroup out of that. It would all be mirrored. No disks are mirrored within a cell — it’s smart enough between the cells.

CK: The minimum number of cells is two?

KC: Yes. Because it you don’t have two cells, you have no redundancy.

CK: Absolutely.

KC: Okay?

CK: Perfect! Very cool! Now we can envision how this works. Well, those are my questions for now. I’m sure there will be a lot more once you start hosting again and giving us more details on how all this works. Some part of me wants to ask you, do you have all these [blogs] prepared in advance and you’re just pushing the “Publish” button on a pre-determined day. I’m hoping you’re answering some of them as people ask them.

KC: Are you referring to my blog thread on Exadata Questions and Answers?

CK: Yes.

KC: Listeners will know where my blog is because when you post this, of course you’ll post up a URL to my blog. The way I’m approaching this is, you know, look — I’ve got a day-job. My night-job, at this point in time, is also very exciting because I’m taking it on myself to disseminate some information about Exadata. So what I do throughout the day, I’m just like all of you folks — connected and getting your RSS updates and what-have-you. I’ll see when people are speaking about Exadata, and if I see something that looks like it’s fodder for a blog Q&A, I just cut the question and paste it into my working sheet, and when I get around to it I type in an answer ,and once I’ve got something cooked, I post it up to the blog.

CK: Filling up the gaps.

KC: Filling in the gaps. You know, honestly — this is the twenty-first century and it’s Web 2.0. I think we would be way behind the times to not be handling some of our information dissemination the way we are. Because if you wanted to collect even the information that we’ve disclosed in this conversation by going out and trolling through white papers and what-have-you, the odds that you would actually get all of the information is pretty slim. So how many books and white papers do people want to troll through? I think blogging is a very effective way to get some timely information out, and I’m hoping I’ll be able to continue that.

CK: Very cool. Alright, thank you so much for this interview, Kevin. We really appreciate your finding the time to talk to us. We’ll see you in the blogs, I guess.

KC: Indeed. And I appreciate the opportunity to do this. Gee whiz, wouldn’t it be good if we could do this again some time!

CK: Oh, absolutely! Let’s wait until the gap gets a bit bigger, and then we’ll get it in.

KC: Okay, very good!

It’s fast, and much faster than anything out there right now. But how many shops will actually need this? How many shops can spend 2.2 million dollars on hardware and equipment?

What are the products, in a nutshell? The Oracle Exadata Storage Server (Data Sheet, PDF):

• 2U Storage “unit” with either 1 TB SAS or 3.3 TB SATA redundant capacity. There is a query processor in the box that can “offload” tasks from the main database server. Primary filtering, decompression, joins, backups.
• Storage units linked to database servers via dual Infiniband offering 20 Gbit/s (2.5 GBytes/sec) bandwidth

The Database Machine (Data Sheet, PDF):

• A standard 42U rack with 8 database servers and 12 Exadata storage servers.
• Pre-installed Linux and Oracle. Pre-configured.
• In 8 servers — a total of 256GB RAM, 64 Intel cores @ 2.66 Ghz, InfiniBand-ed and gigabit-switched.

The cost for one Database Machine: $2.33M ($650,000 + $1,680,000 in software) as grabbed from Larry’s keynote (thank chet) I called the “call us now” phone mentioned on the Oracle Exadata website to ask them for pricing. They had no idea what I was asking about, and I’m still waiting on a salesperson to call me back. (Hint for Oracle — educate your sales staff about new products, just in case I decide to buy one the day after you announce it.)

You have to realize how “cheap” this is. It comes down to $25,000 per core for Oracle EE, RAC, and Partitioning! And extra “free” CPUs for decompressing, filtering and joining, and backups. That’s a good deal. Oh, did I mention you can interconnect several 42U racks?

Back to the main question, what problems does this product solve?

Configuration

That’s right, the number one problem this product solves is configuration. 90% of the problems I’ve seen are due to improperly configured systems. I am not talking init.ora settings here, or design, or indexing, or any of that. I am talking configuration mistakes all over the place. Starting from bottom up, these are the most common mistakes:

• buying large disks without accounting for I/O bandwidth delivery
• mis-configuring them in big meta-arrays (EMC style) with non-aligned stripe sizes. (See “turn-offs” in Christo Kutrovsky, Oracle Pinup)
• sharing the spindles for redo, datafiles, backup, and a bunch of databases (3par style), thus ensuring that I/O is never sequential
• getting single-channel Fibre Channel connectors to the database server
• not configuring directI/O or asyncI/O or the largest possible db_file_multi_read_count
• not using ASM. Of course, ASM reduces overhead, manages data in 1MB or larger (11g feature) extents (this is sequential data!)
• not using parallel query properly — using default values or considering all of the above, just not getting the bandwidth to perform

Using Exadata necessarily and immediately solves all of these issues. You don’t have a choice–you get more I/O bandwidth when you buy extra space, there’s no other way.

No expensive consultants to install your system. From the DBA perspective it’s heaven — no arguing with storage people for dedicated spindles, no arguing with CIOs about big vs. small disks, no arguing with system administrators for ASM. No hiring of expensive consultants to “tune” the system or apply best practices.

You may laugh at all of the above issues, but many shops are exactly like that. Especially the big ones (the target market for Exadata), where everyone is too afraid to change anything in case they get blamed if it doesn’t work. The “best practices” are the only practices with the Database Machine.

To maximize performance, you have to get all the pieces together. Then and only then will you get all the benefits. And this is quite difficult to achieve, especially in large shops where several entire departments are involved.

In all my experience at Pythian, there has been only one client–who, thanks to a combination of good managers, thrust, and desire for performance–would exactly follow my recommendations. And you know what? They are getting their 400 Mb/sec. The new server is reaching 800 with the dual 4gbit fibre channel.

Some interesting aspects of the Oracle Exadata Storage server.

Performance

The data sheet presents two options: 1 TB with SAS with 1000 MB/s bandwidth; or 3.3 TB with SATA and 750 MB/sec. Compression is “extra”, meaning in a typical data warehouse you get 2-3 times compression, meaning your actual bandwidth will be 2000-3000 MB/sec from a single Exadata server.

Redundancy

Mirroring is provided by ASM (either 2- or 3-way). It is also performed across Exadata storage servers (does that mean 2 minimum?)

Disk failure does not abort queries or transaction.

Exadata Storage server does abort queries or transactions, but with no data loss. This is important to know when calculating risk.

Manageability

There’s a plug-in available for 10g Enterprise Manager, a GUI to manage all that. Absolutely mandatory in my opinion.

Conclusion

Oracle has solved the communication issues for big shops, and the result is indeed extreme performance. Don’t get me wrong here, the Exadata is a brilliant idea that will solve some very difficult, specific problems for large data warehouse shops. But the Database Machine will do something much more real, that will help far more people: it will make it impossible for people to mis-configure their database systems.

Hats off to Oracle for releasing a product that solves a problem we are facing everyday: convincing clients to get the right hardware setup for their database workload.

Oracle has been keeping the new feature a secret. Even the 11gR2 beta program had very few participants to prevent information leaking out. It’s, “Something’s coming, but I am not telling what.”

Okay, it worked on me, I’m excited about it. Let’s think what it could be. What single database feature is so major, that Larry himself will announce it during OpenWorld?

What do we know so far?

• Starting with the obvious, Larry’s keynote is “Extreme Performance”, so it’s related to performance.
• We know Kevin Closson has worked on it – he had a blog entry saying “I am working on something big” that got pulled off the web. (Here’s Google’s cache.)

Given these two point, let’s further think about it. What do we know about Kevin?

• He worked for PolyServe — a company whose main product is a cluster file system.
• He worked for Sequent on NUMA systems, which in today’s world is pretty close to cluster software with a very fast, low latency interconnect.
• He is an expert in storage systems and disk performance.
• He joined Oracle recently, possibly to work on this secret project.
• He must be really excited about it, to post anything on his blog under radio silence.

I think it’s something related to storage, something new and revolutionary about storage. But what?

We already know, from leaks on certain websites, that ASM will become a cluster filesystem which will allow storing OCR files, as well as user files, on the ASM disks.

But is this big enough? It’s definitely significant. Now you get a “free” reliable, cluster file system with Oracle. I don’t think it’s big enough though. Oracle already had OCFS and OCFS2. So it’s not something new to release a filesystem. And even if ASM becomes a true filesystem, that would not provide such a significant performance boost to warrant a keynote called “Extreme Performance”. An ASM filesystem would be a major manageability feature, not so much a performance feature.

That being ruled out, what could it be?

Recently, when setting up a new 11g database on a server with 128gb of RAM, I was setting up hugepages as usual, and thinking about how big my cache would be. It struck me that the cache will be bigger than the database for quite a while. Why do we even need the SAN/Datafiles?!

Then it hit me.

We don’t! We don’t need them at all!

What are the main storage components of a production database?

• the redo logs — to guarantee crash recovery
• the datafiles — primary storage
• the backups — mandatory for a production system
• the SGA — why is this part of storage? Well, because you can’t have a database without some fast in-memory storage, right?

If you have sufficient SGA (RAM) to load your entire database (datafiles), why do you need the datafiles?
I am sure you are immediately thinking what if the database crashes?

Remember, what’s the recent push in Oracle: grid computing.

Picture a RAC database – 8 nodes, 128 GB of RAM each, totaling 1 TB of storage. Add 2- or 3-way mirroring and you get 300 GB of highly redundant, extremely fast storage. A true, native “in-memory” cluster database. A true “shared nothing” cluster database.

Even if you do not consider the performance increase, the redundancy level goes up. You no longer have a “central” SAN to rely on. Maybe you have two mirrored SANs in your enterprise to protect you against such failures. How about none?

Let’s keep moving with that idea. How can Oracle achieve it? What technologies would be needed?

I think Oracle already has all the required technologies to achieve this “extreme performance”. It’s just a matter of connecting them.

And the answer is Cache Fusion. But how? Imagine this scenario. During database startup you would “restore” your database from your backups (compressed or not) directly into memory. Remember that’s 8 nodes that are doing the uncompression/reading. So starting up won’t really take much time.

Once the database is up, cache fusion will take care of the rest: sending blocks over the interconnect, keeping past images, keeping and managing multiple copies. Oracle already does this, just not for redundancy reasons. Look at my Cluster Efficiency query.

If a node (or two) go down, who cares? All the data is already replicated 2- or 3-way. In the event all nodes go down, Oracle would still keep the online redo logs for archival purposes. Or maybe not? Replicated in memory REDO? Why not?

In fact, the only real changes are:

• backup will be restored into memory
• no dbwriter — no datafiles to write to
• cache fusion block replication for redundancy

The result? “Extreme Performance.” Now that’s definitely worthy for a keynote by Larry himself.

A major innovation indeed. For Oracle, at least. MySQL cluster databases are already all in memory. Actually, it’s the only way it can be, and this is seen as a limitation by the community, simply because it’s the only way.

Oracle doesn’t need to make the feature exclusive for the entire database. This may be a tablespace level feature, or even a table/partition level one. Then you would really be in control of which areas of your database get “extreme performance”. Think of the possibilities.

We were brainstorming with Paul Vallée on what the new feature could be. Paul’s idea was slightly different than mine. He envisioned ASM to be the driving technology behind an all-in-memory database. ASM already has 2- and 3-way mirroring. The change would be minor — instead of creating disks out of LUNs, they would be created out of RAM. ASM would take care of the inter-node replication.

If Oracle had an all-in-memory database done with ASM, you would still have to “read” the data into the buffer cache, introducing double-buffering. This would be a step back, actually. In the PC world, Windows NT/2000 revolutionized caching from DOS/Windows95. The merging of the file system cache with the execution memory was a significant step forward to avoid double buffering. And this would limit the granularity of what is “all-in-memory”.

This is how Paul’s idea looks:

We have our bets. What’s yours? Please throw in some wild guesses. The winner (the earliest correct guess) gets a Pythian Maestro shell shipped to him or her. (NOTE: I was going to write “does not apply to Oracle employees”, but I decided to give them a chance too. As long as you don’t know and you are guessing, you can try).

Here’s Darrin Leboeuf, Pythian’s V.P. Client Services, modeling the Pythian Maestro shell.

The script is fully RAC-aware, and although it’s not 100% perfect, I use this imperfection to see if any particular node is doing something stupid. Although it is primarily targeted for OLTP systems, it can be useful for data warehouses as well, especially if they use the parallel option.

The query is to the database what “load” (uptime) for a Linux/Unix machine is, except it has much more detail. It is basically a summarization query of the v$active_session_history table. NOTE: you need to have the performance pack license to use it. It is not designed to be aligned, or read. The best is to leave it on just a few lines and concentrate on the results.

It has two “variables” that you can adjust: how far back to look (I use two hours), and how aggressively to look for problems (having count(*) >= 2).

An explanation of how to make sense of the results follows the query.

select round(avg(max(cnt_tot)) over (order by sample_time RANGE BETWEEN INTERVAL '5' minute PRECEDING AND current row)) as avg,max(cnt_tot) as tot,
SAMPLE_time,max(cnt) as cnt, event,substr(sq.sql_text,1), ash.sql_id, ash.sql_child_number chd,/* plsql_entry_object_id, plsql_entry_subprogram_id, plsql_object_id, plsql_subprogram_id, session_state, qc_session_id, qc_instance_id, blocking_session, blocking_session_status, blocking_session_serial#, event_id, event#, seq#, p1text, p1, p2text, p2, p3text, p3, wait_class, wait_class_id, wait_time, time_waited, xid, current_obj#, current_file#, current_block#, program, module, action, client_id*/to_char(round(sum(elapsed_time) / nullif(sum(executions), 0) / 1000000, 6), '9,999,999,990.999999') as "sec p",round(sum(disk_reads) / nullif(sum(executions), 0), 0) as "disk p", round(sum(buffer_gets) / nullif(sum(executions), 0), 0) as "gets p",round(sum(rows_processed) / nullif(sum(executions), 0), 0) as "rows p",round(sum(cpu_time) / 1000000 / nullif(sum(executions), 0), 3) as "cpu p", sum(executions) as exec, sum(users_opening) as open,sum(users_executing) as e
from ( select sum(count(*)) over (partition by sample_time) as cnt_tot, count(*) as cnt,SAMPLE_time, event,sql_id,sql_child_number from gv$active_session_history where 1=1
and sample_time > sysdate - interval '2' hour
group by event, sql_id,sql_child_number, SAMPLE_time having count(*) >= 2) ash, gv$sql sq
where  ash.sql_id = sq.sql_id(+) and ash.sql_child_number=sq.child_number (+)
group by event,sql_text,ash.sql_id,ash.sql_child_number, ash.SAMPLE_time order by sample_time desc

The query concentrates on the idea, “you shouldn’t have too many sessions doing the same thing at the same moment”. Basically, anytime you have multiple sessions running the same query and waiting on the same event, they are grouped together.

Here’s the explanations of each column:

AVG — the average “load” (active sessions) over a 5-minute interval. This should help you spot a problem when you scroll through the results.
TOT — total “load” (active sessions) for that sample time. RAC users: each RAC node will have its own sample time, within 1 second of each other, but not exactly spot-on. So, even if you have sessions waiting on the same event, they will not be grouped together. I kind of like it this way, for now.
SAMPLE_TIME — self-explanatory
CNT — the number of active sessions waiting on the same event and query
EVENT — the event been waited on
SQL_TEXT — self-explanatory, except when empty which means either not found in shared pool or not available in ASH
SQL_ID — if you need to find the SQL
CHD — the child number being executed

NOTE: the following columns are as of “now” and not as of the SAMPLE_TIME.

sec p, dsk p, gets p, rows p, cpu p — these are average statistics for the query being executed. This should give you a quick overview of whether the query is a big query, a small query, a CPU-intensive or a disk I/O-intensive query. Be careful: since the query was introduced to the shared pool, those columns are averages, and therefore could be misleading. These are all per execution stats, so sec p represents the number of seconds on average it took to execute the query.

OPEN — comes from v$sql: the number of sessions that have the query open
E — the number of sessions actively executing the query

I generally use having count(*) >= 2 — the lowest reasonable setting to get an overview of what’s happening on the server. This usually doesn’t show many rows on my servers. It really depends on how busy your server is. You should play with this “filter” to see only what you are interested in.

If you are looking for a problem, you can raise that to, say, 10 and remove the time restraint in order to find times when the database was particularly busy.

Feel free to ask any questions if you have any.