Aligning ASM Disks on Linux

Linux is a wonderful operating system. However there are a number of things that one needs to do to make sure it runs as efficiently as possible. Today, I would like to share one of them. It has to do with using ASM (Automatic Storage Manager) disks.

In Linux, there are 2 major ways to create ASM disks.

1. you can use ASMlib kernel driver
2. you can use devmapper devices

You could also use /dev/raw devices, but I don’t recommend this at all. I will write another blog explaining why.

Regardless of which approach you take, you have to create partitions on your LUNs. Starting with version 2, ASMlib won’t let you use the entire disk. You have to create a partition.

The reason to force the creation of this partition is to make explicit that something exists on that device, and that it’s not empty. Otherwise, some OS tools assume the disk is unused and could mark it, or just begin using it, and override your precious Oracle data.

(Read more after the jump.)

Most people would use the “fdisk” command provided by Linux distributions. This command is quite old, and so has some old-fashioned DOS-style behaviours built in.

When you create your partition, by default, the unit of measure is based on cylinders. Here’s a typical print command from fdisk on a 35 GB disk:

Command (m for help): p

Disk /dev/sde: 35.8 GB, 35864050176 bytes
64 heads, 32 sectors/track, 34202 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1       34202    35022832   83  Linux

Notice where it says Units. 2048 cylinders, 512 bytes each = 1 MB. So your units are 1 MB. only when the disk is relatively small.

When your disk is larger — which is far more typical in the database world, especially with raid arrays — the Units change:

Disk /dev/sdc: 513.5 GB, 513556348928 bytes
255 heads, 63 sectors/track, 62436 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       62436   501517138+  83  Linux

Take a look at the Units number. It’s 8 MB minus 159.5 kB. This is a very weird number, totally misaligned with any possible stripe size or stripe with (stride).

This, by itself, is not a big deal, since the best practice is to have 1 partition per LUN, which represents the entire device. However, this is not the end of it. If you switch to sector mode, you will see what the true start offset is:

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): p

Disk /dev/sdc: 513.5 GB, 513556348928 bytes
255 heads, 63 sectors/track, 62436 cylinders, total 1003039744 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1003034339   501517138+  83  Linux

Notice the start sector, 63rd sector. That’s the 31.5 kB boundary. This value does not align with any stripe size. Stripe sizes are usually multiples of 2 and above 64 kB.

The result is, every so often, a block will be split between 2 separate hard disks, and the data will be returned at the speed of the slower (busier) device.

Assuming the typical 64 kB stripe (way too low, as I will discuss in another blog), and 8 kB database block size, every 8th block will be split between 2 devices. If you do the math, that’s about 12% of all your I/O. Not a significant number, but when you consider how discs are arranged in RAID 5, instead of a logical write being 2 reads and 2 writes (data+checksum, update, then write them back), each logical write could be 4 reads and 4 writes, significantly increasing your disk activity.

The solution?

Before you create your partitions, switch to “sector” mode, and create your partitions at an offset that is a power of 2.

I typically create my partitions at the 16th megabyte (32768th sector). Essentially, I “waste” 16 MBs, but gain aligned I/O for stripe width of up to 16 MB.

The procedure to create an aligned disk is:

[~]# fdisk /dev/sdg
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

When building a new DOS disklabel, the changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

Command (m for help): u
Changing display/entry units to sectors
Command (m for help): p
Disk /dev/sdg: 143.4 GB, 143457779712 bytes
255 heads, 63 sectors/track, 17441 cylinders, total 280190976 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot      Start         End      Blocks   Id  System
Command (m for help): n
Command action
e   extended
p   primary partition (1-4) p
Partition number (1-4): 1
First sector (63-280190975, default 63): 32768
Last sector or +size or +sizeM or +sizeK (32768-280190975, default 280190975):
Using default value 280190975
Command (m for help): p
Disk /dev/sdg: 143.4 GB, 143457779712 bytes
255 heads, 63 sectors/track, 17441 cylinders, total 280190976 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdg1           32768   280190975   140079104   83  Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

This way, the disk is aligned. When it is aligned at least on 1MB, then the ASM files will be also aligned at the 1MB boundary.

This alignment also applies to ext3 file systems. This file system takes it a step further, allowing you to provide the array stride as a parameter during creation, optimizing writing performance (I have not tested this). Look in the man pages for more information.