Troubleshooting a Multipath Issue

Multipathing allows for the configuration of multiple paths from servers to storage arrays, providing I/O failover and load balancing. Linux uses the device mapper kernel framework to support this functionality.

Troubleshooting a Missing Multipath Device in RHEL6

In this post, I will explain the steps taken to troubleshoot a multipath issue. This should provide a glimpse into the tools and technology involved. The problem was reported on a RHEL6 system where backup software complained that the device from which /boot is mounted does not exist.

Following is the device. You can see the device name is a WWID (World Wide Identifier):

# df  Filesystem 1K-blocks Used Available Use% Mounted on  [..] /dev/mapper/3600508b1001c725ab3a5a49b0ad9848ep1  198337 61002 127095 33% /boot 

The file /dev/mapper/3600508b1001c725ab3a5a49b0ad9848ep1 is missing under /dev/mapper:

# ll /dev/mapper/  total 0  crw-rw—- 1 root root 10, 58 Jul 9 2013 control  lrwxrwxrwx 1 root root 7 Jul 9 2013 mpatha -> ../dm-1  lrwxrwxrwx 1 root root 7 Jul 9 2013 mpathap1 -> ../dm-2  lrwxrwxrwx 1 root root 7 Jul 9 2013 mpathb -> ../dm-0  lrwxrwxrwx 1 root root 7 Jul 9 2013 mpathc -> ../dm-3  lrwxrwxrwx 1 root root 7 Jul 9 2013 mpathcp1 -> ../dm-4  lrwxrwxrwx 1 root root 7 Jul 9 2013 mpathcp2 -> ../dm-5  lrwxrwxrwx 1 root root 7 Jul 9 2013 vgroot-lvroot -> ../dm-6  lrwxrwxrwx 1 root root 7 Jul 9 2013 vgroot-lvswap -> ../dm-7 

Mapping UUIDs to Multipath Names

From /etc/fstab, it is found that the UUID of the device is specified:

UUID=6dfd9f97-7038-4469-8841-07a991d64026 /boot ext4 defaults 1 2 

Using blkid, we can see the device associated with the UUID. The blkid command prints the attributes of all block devices in the system:

# blkid  /dev/mapper/mpathcp1: UUID=”6dfd9f97-7038-4469-8841-07a991d64026″ TYPE=”ext4″ 

Remounting the /boot mount point shows the user-friendly name /dev/mapper/mpathcp1:

# df  Filesystem 1K-blocks Used Available Use% Mounted on  [..] /dev/mapper/mpathcp1 198337 61002 127095 33% /boot

Analyzing User-Friendly Name Configurations

From this, we can understand that the system is booting with the WWID as the device name, but later the device name is converted into a user-friendly name. In this multipath configuration, user_friendly_names is enabled:

# grep user_friendly_names /etc/multipath.conf user_friendly_names yes  

As per Red Hat documentation:

“When the user_friendly_names option in the multipath configuration file is set to yes, the name of a multipath device is of the form mpathn. For the Red Hat Enterprise Linux 6 release, n is an alphabetic character, so that the name of a multipath device might be mpatha or mpathb. In previous releases, n was an integer.”

As the system is mounting the right disk after booting up, the problem should be with the user-friendly name configuration in the initramfs.

Root Cause: Missing Bindings in initramfs

Extracting the initramfs file and checking the multipath configuration shows that the user_friendly_names parameter is enabled:

# cat initramfs/etc/multipath.conf  defaults {  user_friendly_names yes 

The interesting point is that /etc/multipath/bindings is missing in initramfs, although the file exists in the system. The /etc/multipath/bindings file is used to map a WWID to an alias:

# cat /etc/multipath/bindings  # Multipath bindings, Version : 1.0  # alias wwid  mpathc 3600508b1001c725ab3a5a49b0ad9848e  mpatha 36782bcb0005dd607000003b34ef072be  mpathb 36782bcb000627385000003ab4ef14636 

Verifying the Solution with Dracut

initramfs can be created using the dracut command. Building a test initramfs file shows that a newly created image includes /etc/multipath/bindings:

# dracut -v -f test.img 2.6.32-131.0.15.el6.x86_64 2> /tmp/test.out # grep -ri bindings /tmp/test.out  I: Installing /etc/multipath/bindings 

So, here is what is happening: When the system boots up, initramfs looks for /etc/multipath/bindings to use aliases for user-friendly names. Because it cannot find the file, it defaults to the WWID. Once the system has fully booted, the bindings file is present, and device names change to user-friendly names.

This likely happened because the multipath configuration was performed after the kernel installation and initial initrd generation. Even if the system root device is not on multipath, it can be included in the initrd if, for example, the root device is on LVM.

Final Fix: Rebuilding the Initrd

To solve the issue, we need to rebuild the initrd and restart the system. Re-installing the existing kernel or installing a new kernel would also fix the issue, as the initrd is rebuilt in both cases.

# dracut -v -f 2.6.32-131.0.15.el6.x86_64

Managed IT Consulting Services

Are you ready to save up to 60% in operational costs?