My First Experience Running SLOB – Status Update 2 (first results)
May 15, 2012 / By Yury Velikanov
NOTE: Others SLOB related posts list avaiable from “My SLOB IO testing index”
I think the results we got so far may surprise you. At lease those doesn’t seems to be the results +Alex Gorbachev and +Kevin Closson expected to see. You can find the first related blog post over here. It will give you the necessary context for further reading. Just to recap: +Kevin Closson says “Orion may give It’s VERY easy to get huge Orion nums but reasonable SLOB” and +Alex Gorbachev “lots of the system IO bound below the CPU level so you should see similar number with Orion or SLOB.” Let see what first testing results revealed.
I would say that Orion results we received seems to be expected numbers :
ran (small): my 12 oth 0 iops 1378 size 8 K lat 8.71 ms bw = 10.77 MBps dur 59.97 s READ ... ran (small): my 24 oth 0 iops 2114 size 8 K lat 11.35 ms bw = 16.52 MBps dur 59.96 s READ ... ran (small): my 240 oth 0 iops 4305 size 8 K lat 55.74 ms bw = 33.63 MBps dur 59.81 s READ
We ran OLTP 8k Orion test in read only mode providing 12 x 500GB disks. It executed 20 test increasing number of parallel threads from 12 to 240. The IOPS numbers grew slowly along with increased response time. The best response time we got was 8.71 ms with 12 processes harming storage and the best IOPS 4305 with correspondent latency 55.74ms running 240 processes. It gave us from 115 to 359 IOPS per IO spindle.
The SLOB results surprised us a bit:
We ran “runit.sh 0 24″ test. Where there were 24 Oracle sessions executing single table’s block reads via index access (db file sequential read). We made sure that most of IO issues by foreground processes would be PIOs. AWR data confirmed that we succeeded: Logical reads Per Second 4,541.6, Physical reads Per Second 4,524.4. It looks like most of block reads 99.6% have been processed via storage requests. However hold on a second 4,524.4 is nothing else but IOPS. If we compare it with the Orion numbers we see that SLOB shows better IO performance than Orion does. Plus on top of that the SLOB latency is significantly less than Orion’s – 5ms vs 55.74ms. I think nobody expected that type of result. Just to give you a bit more input there the most relative parts of the SLOB AWR report:
Snap Id Snap Time Sessions Curs/Sess
--------- ------------------- -------- ---------
Begin Snap: 104 15-May-12 01:00:13 59 1.2
End Snap: 105 15-May-12 01:43:04 36 .7
Elapsed: 42.86 (mins)
DB Time: 990.84 (mins)
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 23.1 5,404.6 1.25 499.58
DB CPU(s): 0.3 68.0 0.02 6.28
Logical reads: 4,541.6 1,061,707.7
Physical reads: 4,524.4 1,057,677.6
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
db file sequential read 11,630,768 58,928 5 99.1 User I/O
DB CPU 748 1.3
resmgr:cpu quantum 1,854,672 79 0 .1 Scheduler
resmgr:internal state change 1 0 101 .0 Concurrenc
undo segment extension 2 0 5 .0 Configurat
The status is still work in progress. We are running some other test now. There is “runit.sh 0 128″ SLOB test on the way. Surprise, surprise it didn’t melt down the server with 128 processes running on the same time :). Talking about the results we have got so far I have suspicions that SLOB doesn’t have big enough objects for hit over the HDDs caches and some of the SLOB request are served from HDDs’ cache. If this is correct than in this round Orion beat SLOB in terms of providing a “cleaner” IO test results.
+Kevin Closson said “Orion may give It’s VERY easy to get huge Orion nums but reasonable SLOB” – kind of FALSE on this occasion
+Alex Gorbachev said “lots of the system IO bound below the CPU level so you should see similar number with Orion or SLOB.” – FALSE again on this occasion :)
Stay tuned folks,
More results on the way …
Yury & Co
Well right after I published this blog post while on the way to English Pronunciation course I thought about other most probable reason for SLOB giving the better results than Orion does. It still is related to the size of the SLOB data set. As we are using 12 x 500GB big HDDs for testing and the SLOB data set is relatively small all the data is located close to the fastest (outer) edge of the disk. Orion on the other side uses all devices’ address space to read from. This may be a good explanation discrepancy in the SLOB & Orion results. We will test it and update you on the results.
Great post, could You please post Orion command line related parameters as well ?
Regards
GregG
Sure I can :)
nohup time ./run_orion_oltp.sh 2>&1 1>./run_orion_oltp.01.log &
[grid@host2 orion]$ cat ./run_orion_oltp.sh
./orion_linux_x86-64 -run oltp -testname data_dg -num_disks 12 -write 0
[grid@host2 orion]$
[grid@host2 orion]$ cat data_dg.lun
/dev/mapper/disk_0_y00_532421940p1
/dev/mapper/disk_0_y01_532341712p1
/dev/mapper/disk_0_y04_532337776p1
/dev/mapper/disk_0_y05_532378236p1
/dev/mapper/disk_0_y08_532384844p1
/dev/mapper/disk_0_y09_532341692p1
/dev/mapper/disk_1_y02_532253988p1
/dev/mapper/disk_1_y03_532384572p1
/dev/mapper/disk_1_y06_532385008p1
/dev/mapper/disk_1_y07_532341764p1
/dev/mapper/disk_1_y10_532385560p1
/dev/mapper/disk_1_y11_531870356p1
[grid@host2 orion]$
Love the testing, Yury..your work will help others so thanks.
A couple of comments:
1. Regarding the HDD cache hypothesis. You have 12 disks atached to your server so I’ll just presume you have an LSI controller. Depending on the vintage that will provide either 256 or 512MB cache on the PCI card. If you run more than 8 reader.sql users you will blow out that cache. The cache on an LSI card is generally a <200us service time PIO. You are seeing 5ms which is physically acceptable for fast drives with short seeks. You cite the capacity and count of your disks but not the performance attributes (e.g., RPM, track buffer size). Your 5ms I/O is quite likely due to short stroke seeks because you are in the neighborhood of 370 IOPS/drive which is quite possible with short seeks.
2. I'll be gentle on this comment. Your testing is at the extreme low end. Modern Xeons (e.g., WSM, SNB) can handle on the order of 20,000 IOPS/core. Since Orion uses no user-mode CPU it has the propensity to get to high-performance numbers easier than SLOB. SLOB has Oracle latches to contend with. Modern QPI servers handle that better, sure, but at your PIO rate there simply is no contention.
All that aside, nice work. Thanks for posting for us too.
Thanks for the comments and attention Kevin,
I think I do not understand your’s 2d comment:
2. I’ll be gentle on this comment. Your testing is at the extreme low end. Modern Xeons (e.g., WSM, SNB) can handle on the order of 20,000 IOPS/core. Since Orion uses no user-mode CPU it has the propensity to get to high-performance numbers easier than SLOB. SLOB has Oracle latches to contend with. Modern QPI servers handle that better, sure, but at your PIO rate there simply is no contention.
I thought IO performance is limited by number of HDDs handles. If I have 12 HDDs in my test. Each HDD have average latency 5ms than each spindle (HDD) can handle 200 (- CPU overhead time). 12 x 200 => 2400 IOPS is the maximum theoretical IOPS I can reach here. Any higher number is ether HW or SW “cheating”.
To reach mentioned by you 20,000 theoretical IOPS number we should have 100 HDDs in the system. IMHO: 100 HDDs is normal for High end system. However I would bet that 80% of Oracle RDBMS clients doesn’t have 50 HDDs in their systems (I am not talking about $ licences %, I am talking about systems and DBAs :)
Yury
Yury,
Orion gets raw disks. How does SLOB access disk? Is it through a file system with direct I/O?
ASM
Sorry, Yury, one more comment. Do you intend to analyze the behavior difference between SLOB and Orion for writes? I’ve been pointing out the difference in CPU profile for the two kits. DBWR is different than Orion. Perhaps some runit.sh N 0 ?
IMHO: Writes is the area where SLOB is superior vs Orion in most of the cases. As to test writes with Orion we need a dedicated device for testing only. This is luxury that happens only during the initial build of a system (and not always even than e.g. ODA). I may put some efforts in place to deallocate a device from ASM and test writes but most luckily I will not have enough time for doing it.
Pingback: SLOB Is Not An Unrealistic Platform Performance Measurement Tool – Part I. Let’s See If That Matters…To Anyone. « Kevin Closson's Blog: Platforms, Databases and Storage
Hi Yury,
Are you using resource manager on your testing DB ? As far as I remember event “resmgr:cpu quantum” is related to that and it can impact your tests as well.
If you are looking for disk cache impact on your test check “db file sequential read” histogram. I have been tested SLOB on ext3 file system with filesystemio_options set to all and here are my results (time vs % of event)
under 1 ms 5.1 %
under 2 ms 12.7 %
under 4 ms 35.8 %
under 8 ms 35.9 %
under 16 ms 9.7 %
regards,
Marcin
>> Are you using resource manager on your testing DB ?
Let me check. I didn’t switch it. It may be that in 11.2.0.3 it is a default option.
This is a very good point. Thank you for the hint!
Than you for sharing your results. I am still working on getting consistent results. I hope to publish those as soon as ready.
Yury
>getting consistent results
I’ve never seen SLOB return inconsistent results with read-only. Are you doing write-only (REDO model) or mixed? I’d like to understand your scenario.
Pingback: My SLOB IO testing index | The Pythian Blog