Comments on: Aligning ASM Disks on Linux

By: Ben M.

Ben M. — Wed, 12 Nov 2008 17:04:20 +0000

Great post indeed! Very hard to find information about this topic.

An other great page is the following:
http://insights.oetiker.ch/linux/raidoptimization.html

What would be great is a complete post on how to optimize the whole thing…

HW RAID -> Partitioning -> LVM -> File system option (Ext3 like stride, fs-type largefile4, etc)

About the following:

> Essentially, I “waste” 16 MBs, but gain aligned
> I/O for stripe width of up to 16 MB.”

Based on the following page, I suspect “stripe width” isn’t rightly used here…

“Stripe width refers to the number of parallel stripes that can be written to or read from simultaneously. This is of course equal to the number of disks in the array.”

http://www.storagereview.com/guide2000/ref/hdd/perf/raid/concepts/perfStripe.html

By: Hai Wu

Hai Wu — Wed, 09 May 2007 23:11:04 +0000

Ok. thanks for the confirmation. I will go with 2048 (1mb) then.

By: David Edwards

David Edwards — Wed, 09 May 2007 18:43:02 +0000

[Editorial Note: Christo pointed out that the sidebar on this page was getting pushed to the very bottom in IE browsers. The culprit was the long raw URL in the first comment (by LSC). To fix it, I edited the comment, condensing the URL into the words “this paper from Oracle”. The rest of that comment is untouched.]

By: Christo Kutrovsky

Christo Kutrovsky — Wed, 09 May 2007 17:36:29 +0000

It’s all good. However, consider this. ASM does it’s allocations in 1mb “blocks”. I wouldn’t go under 1mb.

However, given what I know so far, 256k will be fine. Remembe we’re talking about saving ~760 kb here …

By: Hai Wu

Hai Wu — Wed, 09 May 2007 16:00:38 +0000

Thanks Christo. Since we aren’t going to get any stripe size that would be different from 128KB in this case, I guess I could configure the first sector to be 256 (128K) instead of 32768 (16M), so that it would do alignment for 128K stripe size. Do you see any issues in doing this?
Thanks,
Hai

By: Christo Kutrovsky

Christo Kutrovsky — Wed, 09 May 2007 14:25:54 +0000

Yes it is. Because the default is a not a multiple of your stripe size. As long as it is a multiple of your stripe size you are good.

So if you offset at the 16th Mb as I do, then you will be alligned for all stripe sizes that are a power of 2, up to 16 Mb stripe size.

Now EMC has these hypervolumes that have the weird stripe sizes of 960 kb, that completelly destroys any allignement attempts :)

By: Hai Wu

Hai Wu — Wed, 09 May 2007 03:09:33 +0000

Hi Christo,
I was told by SAN administrators here that there’s no way to change the stripe size for EMC LUN to be 1MB here, since it has already been set to 128KB for a big chunk of RAID5, every LUN will come from that one.

In this case, there’s no choice but to accept that as a fact. Is it necessary to do alignment as above for 128KB stripe size?

Thanks,
Hai

By: Christo Kutrovsky

Christo Kutrovsky — Wed, 11 Apr 2007 13:26:22 +0000

Hi Gustavo,

The aligning has to do with the RAID level stripe size, not with the ASM stripe size.

There’s always a “penalty” on miss-aligned IO. You are using 2 devices, when you could have used only 1. Note, that using multiple devices is not bad by itself. What’s bad is reading small amounts of data from multiple devices, as opposed to just 1.

You generally want to be doing at least 512kb io sizes per IO device, to achieve 90% of it’s sequencial read speed (in my testing).

By: Gustavo Tamaki

Gustavo Tamaki — Wed, 11 Apr 2007 09:52:20 +0000

Hi Christo,

In your example, with a stripe size of 64KB, the number of unaligned I/Os are about 12.5%. If I consider the ASM stripe size of 1MB, can I assume that this number would drop to less than 1%?

For a storage configured in RAID-10, the effect would be negligible, since the unalignment occurs only in the stripe boundaries and there’s no right penalty, correct?

Gustavo

By: Christo Kutrovsky

Christo Kutrovsky — Mon, 02 Apr 2007 00:22:35 +0000

Yes it does mention it for solaris.

The aligment applies to any OS. I must point out, that the default partitions in linux are worst case for alignment.