Are There Performance Penalties Running Oracle ASM on NFS?

In this blog post I am publishing my SLOB tests results and conclusions testing Oracle database IO performance placing data files on NFS directly and on ASM disk group located on NFS.

You may ask “Why anyone would consider placing ASM files on NFS and if it is possible at all?” It appears that there are legitimate reasons for doing that and Oracle supports it. You could find additional information under the following blog post Reasons for using ASM on NFS.

Some additional details on how I have executed the tests provided below. Please do not hesitate to ask for more details. At this stage I just want to mention that I have used my favorite testing tool’s (SLOB from Kevin Closson) modified version SOS – SLOB on steroids.

You may be interested to have a look on my DNFS presentation as it is related to kNFS vs dNFS comparison. It is available here.

There are no visible penalties

Placing data file on NFS directly and running SLOB’s 22 readers test gave me 0.53 ms average response time in DNFS configuration and 1.73 ms in kernelized NFS test.
ASM setup using the same NFS configuration gave 0.49 ms and 1.74 ms correspondingly.
Clearly based on the tests results there are no performance related penalties running Oracle ASM on NFS comparing with placing data file directly to NFS. I think less than 10% discrepancy in DNFS tests is neglectable.

Testing results

The following are bits of AWR reports that I found relative. The full reports are available here.

Just kNFS

 Elapsed:                1.09 (mins)
      DB Time(s):               21.4              116.9       0.36       4.94
       DB CPU(s):                0.8                4.4       0.01       0.19
   Logical reads:           12,140.1           66,458.8
   Block changes:               41.8              228.8
  Physical reads:           12,042.2           65,923.3
db file sequential read             791,093       1,370      2   97.6 User I/O
DB CPU                                               53           3.8
awr_0w_22r.20121023_201639.txt
Tue Oct 23 20:16:40 EDT 2012

Just dNFS

real    1m13.117s
user    0m0.576s
sys     0m1.281s
   Elapsed:                1.04 (mins)
      DB Time(s):               21.3              110.7       0.13       4.68
       DB CPU(s):                5.0               26.0       0.03       1.10
   Logical reads:           37,408.2          194,450.9
   Block changes:               33.3              173.0
  Physical reads:           37,298.0          193,878.0
db file sequential read           2,326,535       1,229      1   92.5 User I/O
DB CPU                                              312          23.5
awr_0w_22r.20121023_203540.txt
Tue Oct 23 20:35:40 EDT 2012

ASM on kNFS

real    1m13.052s
user    0m0.606s
sys     0m1.259s
   Elapsed:                1.04 (mins)
      DB Time(s):               21.3              111.1       0.36       4.69
       DB CPU(s):                1.2                6.2       0.02       0.26
   Logical reads:           11,883.9           61,944.8
   Block changes:               43.7              227.8
  Physical reads:           11,786.3           61,435.9
db file sequential read             737,114       1,283      2   96.3 User I/O
DB CPU                                               74           5.6
awr_0w_22r.20121104_030233.txt
Sun Nov  4 03:02:34 EST 2012

ASM on dNFS

real    1m13.743s
user    0m0.633s
sys     0m1.241s
   Elapsed:                1.05 (mins)
      DB Time(s):               21.2              111.8       0.13       4.71
       DB CPU(s):                6.5               34.2       0.04       1.44
   Logical reads:           37,754.5          198,865.5
   Block changes:               43.9              231.3
  Physical reads:           37,639.5          198,259.8
db file sequential read           2,379,107       1,162      0   86.7 User I/O
DB CPU                                              411          30.6
awr_0w_22r.20121104_025602.txt
[oracle@ycluster1a SLOB]$

Details on how did I test

• Both NFS server and Oracle database have been located on the same host
• Volume where a data file have been located for all tests have been created on Linux RAM disk to exclude any slow HDDs impact
• NFS mount has been mounted via loopback device (127.0.0.1) to exclude any slow network components impact on the tests results
• Tests used “db file sequential read” aka random reads
• The host was located on Oracle VM

After reading above points you most probably would say: Hey Yury, your tests are far from real world workload. And I totally agree. But I didn’t want to test the real life load. In fact it is quite difficult to reach close to real life workload in any testing. The only goal I had was to see if an additional IO layer (ASM) would make any difference in terms of IO performance. From my perspective this setup may be one of the best for the purpose as it eliminates a lot of components that may have an impact on the testing results. The only concern on my side as of now is the fact that I ran Oracle VM aware kernel and it may have an impact on kNFS vs dNFS comparison. However I think we are good comparing NFS vs ASM on NFS.

For those of you who have been used SLOB & SOS I have used the following command to run the tests and SOS scripts I have published here – SLOB on steroids.

v_r=22; v_w=0; v_c=60
date; time bash -x runit.sh ${v_w} ${v_r} ${v_c} 2>&1 > ./runit.sh_${v_w}_${v_r}c1t.`date +%Y%m%d_%H%M%S`.log ; egrep "Elapsed:  |Logical reads:   |Reddo size:   |Block changes:    |Physical reads:|DB Time\(s\):|DB CPU\(s\):"  `ls -trp awr_* | tail -1` ; egrep "db file sequential read|DB CPU" `ls -trp awr_* | tail -1` | head -3 | tail -2 ; ls -trp awr_* | tail -1 ; date