Average time spent by an IO request in the queue is await-svctm to the best of my knowledge. What exactly do you extract reading the kernel source?

you say:
“You already have 144 IOs in the device queue”

avgqu-sz ,is not average queue lenth,it represents the waiting time(ms) that all requests in the queue .

you can deep into kernel code.

In your case, it might very well be that IO requests are processed by the application mostly synchronously – issuing and IO and waiting on it an then another one and waiting and so on.

In many cases, you can’t say whether IO is a bottleneck using iostat – say doesn’t tell you how much time IO contributed to you response time. What you see in iostat is what’s *average* IO workload and how you IO subsystem delivers. You need to dig into application instrumentations to understand the bottleneck. In case of an Oracle database as an application using IO as a service – you can user 10046 trace.

dm-0 and dm-1 is how your multipathed devices are represented.

Hi, what does this mean:

avgqu-sz await svctm %util
1.17 3.66 3.11 99.72

1. Why is the disk hitting almost 100% utilization? Is it alright and can we say there is no IO bottlneck as await is almost equal to svctm and avgqu-sz is close to 1.

This is happening on a RAID 5 SAN partition.

2. Iostat shows the SAN disks as dm-0 and dm-1 so what does that mean in terms of IOstat output? Should it be read just as any normal disk?

how do we run iostat to get information on something like
/dev/mapper/mpath0p1 mounted on /somepartition

You already have 144 IOs in the device queue and your device simply can’t process them faster than that. If IOs are done sequentially and each is 6.75ms long and you are doing 148.2 IOs per second then in one second you get 1,000 ms of IOs. Providing they are all serialized to a single IO thread – there is no way unless you can split those IOs.

I don’t see how increasing queue size would help (unless you are talking about different queue which is set to 1 right now somewhere in the layer below normal linux device queue – didn’t look into iSCSI software much). You could try to look into why you average actual IO size is half of max 1MB that you are requesting. Maybe limitation of your iSCSI device or some Linux config – this could be a simple way to increase throughput.

Another place to look is your Linux IO scheduler.

I don’t know if running two dd’s in parallel would make any difference.

Anyway, dd test is pretty artificial. If you need to simulate Oracle workload – do yourself a favor and have a look at ORION.

# dd if=/dev/zero of=/vdbench/test bs=1024k

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdj 0.00 17236.60 0.00 148.20 0.00 67.40 931.42 144.41 915.11 6.75 100.00
sdj1 0.00 17236.60 0.00 148.20 0.00 67.40 931.42 144.41 915.11 6.75 100.00

So here, the virtual qutim value is very very high, hence the average time a request is spending in the queue is quite abnormal. Do you think that increasing the iscsi queue depth on client side (linux) would improve something ?
Well, I may be far away from the root cause of this.
Many thanks.
fred

We are currently having some performance issues, our currently linux installation is on Red Hat 4.1.2-48. It has an Oracle 11g installed and based on the Oracle dba’s we are having I/O contention. After reading your post and analyzing the iostat results we received, the stats does not seem to point to a contention.

My question is, how does the avgqu-sz relate to await, as based on the stats i get, await is greater than svctm, but avgqu-sz is not relevant. Can you help me interpret the numbers below? Btw, disk is on a SAN setup, and the stats below are based on some of the most questionable numbers.

avgqu-sz await svctm
0.70 107.03 0.77
0.79 15.35 0.45
0.70 12.11 0.10
0.28 10.96 0.97
0.56 9.95 0.60