Blueprints: Installing Linux Distributions on Multipath Devices

Blueprints

Installing Linux Distributions on Multipath Devices

Blueprints

Installing Linux Distributions on Multipath Devices

Note

Before using this information and the product it supports, read the information in Appendix D,

“Notices,” on page 27.

First Edition (August 2008)

US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents

Preface ...... v Appendix B. Additional multipath

Introduction ...... v configuration information for IBM

storage ...... 23

Installing Linux Distributions on

Multipathed Devices ...... 1 Appendix C. Related information . . .25

What is Multipath Connectivity ...... 1

Hardware Setup for Test Environment ...... 2 Appendix D. Notices ...... 27

Multipath installation on Red Hat Enterprise Linux Trademarks ...... 28

5.2 ...... 2 Terms and conditions ...... 29

Multipath installation on SUSE Linux Enterprise

Server 10 SP2 ...... 11

Appendix A. Troubleshooting tips . . .21

Preface

Introduction

This blueprint provides step by step instructions for installing Red Hat Enterprise

Linux 5.2 and SUSE Linux Enterprise Server 10 SP2 on a LUN in a multipath disk

storage environment. The procedure demonstrated here is performed on a System

x host connected to a DS6000 storage server through Fibre Channel Fabric, but it

can be adapted for installing either of the Linux distribution onto other supported

models of storage devices.

Intended audience

This document is intended to be used by Linux system administrators who have

experience in installing Red Hat Enterprise Linux 5 or SUSE Linux Enterprise

Server 10 and have a moderate level of knowledge in Device Mapper (DM)

multipath.

Scope and purpose

This document is intended to serve any Linux administrator who wants to take

advantage of the multiple paths available to their storage devices. It provides

installation steps to complete a Linux distribution install to a selected multipath

device that is connected through Fibre Channel.

Installation onto hosts and storage subsystems that are connected through iSCSI or

other protocols are beyond the scope of this blueprint. Also configuration and

setup of storage device are not be covered in this document. Refer to appropriate

storage manual for those details.

Hardware requirements

A System x or p server and a supported multipath storage device. The two are

connected via a Fibre Channel. The multipath device is setup to be the bootable

device.

Refer to IBM Storage Interoperability matrices (http://www...ibm.com/

systems/storage/product/interop.html) for supported storage configurations.

IBM Services

® Linux offers flexibility, options, and competitive total cost of ownership with a

world class enterprise operating system. Community innovation integrates leading

edge technologies and best practices into Linux.

® IBM is a leader in the Linux community with over 600 developers in the IBM

Linux Technology Center working on over 100 open source projects in the

community. IBM supports Linux on all IBM servers, storage, and middleware,

offering the broadest flexibility to match your business needs.

For more information about IBM and Linux, visit us at ibm.com/linux

(https://www.ibm.com/linux)

IBM Support

Questions and comments regarding this documentation may be posted on the

DeveloperWorks Multipath Blueprint Community Forum:

http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1334

The IBM developerWorks discussion forums let you ask questions, share

knowledge, ideas, and opinions about technologies and programming techniques

with other developerWorks users. Use the forum content at your own risk. While

IBM will attempt to provide a timely response to all postings, the use of this

DeveloperWorks forum does not guarantee a response to every question that is

posted, nor do we validate the answers or the code that are offered.

Typographic conventions

The following typographic conventions are used in this blueprint:

Bold Identifies commands, subroutines, keywords, files, structures,

directories, and other items whose names are predefined by the

system. Also identifies graphical objects such as buttons, labels,

and icons that the user selects.

Italics Identifies parameters whose actual names or values are to be

supplied by the user.

Monospace Identifies examples of specific data values, examples of text

similar to what you might see displayed, examples of portions of

program code similar to what you might write as a programmer,

messages from the system, or information you should actually type.

vi Blueprints: Installing Linux Distributions on Multipath Devices

Installing Linux Distributions on Multipathed Devices

What is Multipath Connectivity

The connection from the server through the Host Bus Adapter (HBA) to the

storage controller is referred as a path. Within the context of this blueprint,

multipath connectivity refers to a system configuration where multiple connection

paths exist between a server and a storage unit (Logical Unit (LUN)) within a

storage subsystem. This configuration can be used to provide redundancy or

increased bandwidth. Multipath connectivity provides redundant access to the

storage devices, for example, to have access to the storage device when one or

more of the components in a path fail. Another advantage of using multipath

connectivity is the increased throughput by way of load balancing.

Note that multipathing protects against the failure of path(s) and not the failure of

a specific storage device.

A common example of multipath connectivity is a SAN connected storage device.

Usually one or more Fibre Channel HBAs from the host will be connected to the

fabric switch and the storage controllers will be connected to the same switch.

A simple example of multipath connectivity could be: two HBAs connected to a

switch to which the storage controllers are connected. In this case the storage

controller can be accessed from either of the HBAs and hence we have multipath

connectivity.

In the following diagram each host has two HBAs and each storage device has two

controllers. With the given configuration setup each host will have four paths to

each of the LUNs in each of the storage devices.

Hardware Setup for Test Environment

™ For the examples included in this blueprint, the host system is a System x x346

server with one HBA connected to a Fibre Channel Fabric. This Fibre Channel

™ Fabric is also connected to a DS6000 storage server with five storage devices.

Each storage device has two fabric connections.

Multipath installation on Red Hat Enterprise Linux 5.2

This section describes the steps for installing Red Hat Enterprise Linux 5.2 on a

multipath device.

1. Determine which multipath device your machine boots from. To do this, note

down the LUN (Logical Unit) number of the bootable device that is made

available by your Fibre Channel Adapter card during firmware boot.

In the test environment, the firmware displays the boot disk, showing lun

number 0.

2 Blueprints: Installing Linux Distributions on Multipath Devices

2. Start the installation by providing the keyword mpath in the kernel command

line. In the test environment, linux mpath vnc was used.

The Partitioning screen displays a list of multipath devices as

mapper/mpath*.

Installing Linux Distributions on Multipathed Devices 3

3. Determine the multipath device that corresponds to the bootable device for

the host being installed. Go back to the console window if using VNC, or start

an alternate console. In the test environment, the System x is hooked up to an

RSA. In the remote control console, we pressed Ctrl-Alt-F2 for an alternate

console.

From the output, we found the device that corresponds to the bootable LUN.

In the test environment, System x is connected to an RSA. In the console, we

ran the command:

multipath -ll | grep -E ’: | mpath’

The following output was displayed:

From the output of the command as shown above, dm-6, the fourth multipath

device listed, corresponds to LUN 0. First, look for the numbers in x:x:x:x

4 Blueprints: Installing Linux Distributions on Multipath Devices

format. The devices with x:x:x:0 are the devices correspond to LUN 0. In this

case, dm-6, with the two multipaths 0:0:0:0 and 0:0:1:0, is the bootable

multipath device of the host. mapper/mpath4, which corresponds to dm-6, is

then the bootable multipath device for installation.

4. Continue the install by going back to your VNC session, or switching back to

the console window if installing through text mode. In the test environment,

we pressed Ctrl-Alt-F1 in the remote control console.

5. Select the appropriate multipath device for installation. In the test

environment, mapper/mpath4 is the only device checked.

6. Select Review and modify partitioning layout and then press Next.

7. Boot loader might not selected by default to install into the bootable multipath

device. In the test environment, the boot loader defaulted to

/dev/mapper/mpath0, as shown in the following screen. Note the first line of

the screen:

The GRUB boot loader will be installed on /dev/mapper/mpath0.

Installing Linux Distributions on Multipathed Devices 5

8. If the installer did not select the correct multipath bootable device, change

where the boot loader is installed by selecting Configure advanced boot

loader options and press Next.

6 Blueprints: Installing Linux Distributions on Multipath Devices

9. Press Change Drive Order and a new window pops up.

10. Highlight the correct bootable mulitpath device and use Up to move the

device up the list.

Installing Linux Distributions on Multipathed Devices 7

Repeat this step until the bootable mulitpath device is on top of the list.

8 Blueprints: Installing Linux Distributions on Multipath Devices

11. Press OK.

12. The bootable device is now selected for boot loader install. As in the test

environment, note that /dev/mapper/mpath4 is selected.

13. Continue and complete the installation.

Note: If you are a different IBM supported multipath storage, additional steps

are needed to finish the installation. See Appendix B, “Additional

multipath configuration information for IBM storage,” on page 23 for

more details.

14. Reboot the system.

15. Verify that the firmware boot sequence is properly configured to boot from the

correct multipath device.

In the test environment, we pressed F1 in the BIOS screen (where the IBM

eServer logo appears) to go to Configuration/Setup menu. Choose Start

Options and then Startup Sequence Options. Change the option to have Hard

Disk 0 as the first startup device to boot from the multipath device (mpath4).

16. Follow these steps to verify the installation was successful:

a. Run df and cat /proc/swaps to verify that the correct partitions are in a

multipath device. In the test environment, the following partitions are

displayed: root, /boot, and swap as /dev/mapper/mpath4p3,

/dev/mapper/mpath4p1, and /dev/mapper/mpath4p2 respectively. These

partitions are installed correctly to use the Device Mapper (DM) Multipath

feature of Linux.

Installing Linux Distributions on Multipathed Devices 9

b. Run multipath -ll to verify that the specific multipath device has as

many paths as were configured. In the test environment, two paths are

displayed, verifying that all the paths are properly configured.

c. If installed on LVM, run dmsetup ls and demsetup table to verify that the

LVM volumes are created as linear devices on top of the chosen multipath

device.

10 Blueprints: Installing Linux Distributions on Multipath Devices

Multipath installation on SUSE Linux Enterprise Server 10 SP2

This section describes the specific steps for installing SUSE Linux Enterprise Server

10 SP2 on a multipath device.

1. Determine which multipath device your machine boots from. To do this, note

down the LUN (Logical Unit) number of the bootable device that is made

available by your Fibre Channel Adapter card during firmware boot.

In the test environment, the firmware displays the boot disk, showing lun

number 0.

Installing Linux Distributions on Multipathed Devices 11

2. Start the installation normally; you do not need to specify any special

keywords.

3. Continue installation until you reach Installation Settings, then click

Partitioning.

12 Blueprints: Installing Linux Distributions on Multipath Devices

4. Select Create Custom Partition Setup in the Suggested Partitioning window

and press Next.

5. Click Custom Partitioning (for experts) and then click Next in the window. In

the test environment, a total of 10 disks were discovered.

Installing Linux Distributions on Multipathed Devices 13

6. When you reach the Expert Partitioner window, switch to an alternative

console. In the test environment, the System x is hooked up to an RSA. In the

remote control console, we pressed Ctrl-Alt-F2.

14 Blueprints: Installing Linux Distributions on Multipath Devices

7. Determine which of the SCSI devices listed in the installer are the ones that

correspond to the bootable device for the host being installed. In an alternate

console, run the command ls -1d /sys/block/sd*/device/scsi_device*. The

file where the last digit is the same as the LUN number of the bootable device

in your environment is the corresponding SCSI device.

In the test environment, as shown in the following screen, the LUN 0

corresponds to the SCSI devices sda and sdf. Note that these represent the

same storage device, but are found in two different paths. Therefore, this

install only uses partition sda.

8. While partitioning the appropriate device, verify that the Mount by option of

each partition on the multipath device is set correctly. If you are installing on

LVM , the partitions should be mounted by Volume Label. If you are

installing directly into disk, they should be mounted by Device ID. To set the

Mount by option, go back to the VNC screen and click Create or highlight

each disk partition that you are setting up and click Edit. You should see a

screen similar to the following:

Installing Linux Distributions on Multipathed Devices 15

9. In the Create or Edit Partition screen, click Fstab Options.

10. Choose Volume label for installing on LVM or Device ID for installing

directly to disk as the Mount in /etc/fstab by option of the disk partition that

you are creating or editing.

16 Blueprints: Installing Linux Distributions on Multipath Devices

11. Before continuing, verify that the Mount By option for all the partitions on the

device are set to L for installing on LVM, or I for installing directly on disk. In

the test example, sda was chosen for installation and three disk partitions

were chosen for /boot, swap, and root. Notice the letter I in each disk

partition’s Mount By column next to the disk partitions just created.

Installing Linux Distributions on Multipathed Devices 17

12. Continue and complete the installation.

Note: If you are a different IBM supported multipath storage, additional steps

are needed to finish the installation. See Appendix B, “Additional

multipath configuration information for IBM storage,” on page 23 for

more details.

13. Verify that the firmware boot sequence is properly configured to boot from the

correct multipath device.

In the test environment, press F1 in the BIOS screen (where the IBM eServer

logo appears) to go to Configuration/Setup menu. Choose Start Options and

then Startup Sequence Options. Changed the option to have Hard Disk 0 as

the first startup device to boot from the multipath device (mpath4).

14. Reboot the system.

15. Verify that all the disk partitions on the multipath device are referenced in

/etc/fstab and the boot loader files (in the test environment, this is

/boot/grub/menu.lst) by /dev/disk/by-id/* instead of /dev/sd* names. If the

disk partitions are not referenced correctly, change the reference by editing the

corresponding entries in /etc/fstab and the boot loader files.

Find the /dev/disk/by-id names by looking under the /dev/disk/by-id

directory and determine which link points to root, swap and boot partitions. If

using LVM, make sure the device are referenced by their label.

16. Enable the multipath function by running the following commands:

chkconfig boot.multipath on

chkconfig multipathd on

17. Add multipath module to initrd

Edit the file /etc/sysconfig/kernel and add dm-multipath to

INITRD_MODULES.

18 Blueprints: Installing Linux Distributions on Multipath Devices

In the test environment, the INITRD_MODULES line looks like this in the

/etc/sysconfig/kernel file:

INITRD_MODULES="lpfc sym53c8xx dm-multipath"

18. Run mkinitrd and, if required, run lilo.

19. Reboot the system again.

20. Enter the command lsmod | grep dm-multipath to see if the module is loaded

successfully.

21. System should now be up with root, boot and swap devices on multipath.

Verify the installation:

a. Run df and cat /proc/swaps to verify that the correct partitions are in a

multipath device. In the test environment, the following partitions are

displayed: root, /boot, and swap as /dev/dm-7, /dev/dm-5, and /dev/dm-6

instead of /dev/sda1, /dev/sda2, and /dev/sda3 respectively. These

partitions are installed correctly to use the Device Mapper (DM) Multipath

feature of Linux.

b. Run dmsetup ls to list all DM devices by name followed by its major

number and minor number in a pair of parenthesizes separated by a

comma. Note that the minor number here corresponds to the dm number

listed in df and cat /proc/swap. In the test environment, dm-7 is the root

directory mounted by Device ID. The corresponding dmsetup ls entry is

the one with minor number 7.

c. Run multipath -ll to verify that the specific multipath

device has as many paths as configured. In the test environment, two

paths are displayed, verifying that all the paths are properly configured.

d. If installed on LVM, run dmsetup table to verify that the LVM volumes

are created as linear devices on top of the chosen multipath device.

Installing Linux Distributions on Multipathed Devices 19 20 Blueprints: Installing Linux Distributions on Multipath Devices

Appendix A. Troubleshooting tips

This topic discusses troubleshooting tips and caveats.

If Your System does not Boot Up

After the installation is complete, if the system does not boot up, verify that the

server supports booting from Fibre Channel connected storage by looking into the

firmware boot options, and that you are using the appropriate LUN shown as a

boot device by your HBA.

In Red Hat 5, If the installation menu does not show the devices in

mapper/mpath0 format, verify two things:

v You provided mpath in the command line booting up to install.

v Your hardware is configured appropriately, such that the storage devices are

seen through multiple paths.

Tips for LVM on Multipath Devices

When using LVM on dm-multipath devices, it is preferred to turn LVM scanning

off on the underlying SCSI devices. This can be done by changing the filter

parameter in /etc/lvm/lvm.conf file to be filter = [ "a/dev/mapper/.*/",

"r/dev/sd.*/" ].

If your root device is also a multipath LVM device, then make the above change

before you create a new initrd image.

Appendix B. Additional multipath configuration information

for IBM storage

For certain IBM storage devices, a ready-made Multipath configuration file is

available. It is important to use the recommended Multipath configuration file. If

the model number of your IBM storage is not listed in the following links but it is

a supported Multipath storage device, no additional steps should be needed.

Instructions on how to use the Multipath configuration file should be based on the

information provided at the System Storage Interoperation Center (SSIC)

(http://www.ibm.com/systems/support/storage/config/ssic/) site and the

corresponding Multipath configuration file and its ReadMe file. The following

steps are provided for reference only.

After the first boot, perform the following steps:

1. Make a copy of /etc/multipath.conf file

2. Replace the original /etc/multipath.conf file by the one available for download

from the Subsystem Device Driver for Linux site (http://www.ibm.com/ support/docview.wss?rs=540&context=ST52G7&dc=D430&uid=ssg1S4000107

&loc=en_US&cs=utf-8&lang=en#DM).

For Red Hat only:

3. Run initrd and create a new initrd image to include the changes made in

/etc/multipath.conf file.

Run mkinitrd /boot/initrd.final.img

4. Edit the bootloader file (for example, /boot/grub/menu.lst) and replace the

initrd file name with /boot/initrd.final.img.

5. Reboot the system again.

Note: For SUSE Linux Enterprise Server 10, the /etc/multipath.conf file is not

created by default. See the /usr/share/doc/packages/multipath-tools/

directory for related information. In the directory, find a template called

multipath.conf.synthetic and a how-to guide called

multipath.conf.annotated.

The option user_friendly_names is not supported by initrd. When using

user_friendly_names in your /etc/multipath.conf file, either do not specify it

immediately or comment it out. After running the mkinitrdand command and

before you reboot the system, add user_friendly_names back to

/etc/multipath.conf file.

Appendix C. Related information

Listed here are Web sites that relate to multipath.

v DS8000 and DS6000 Hardware and Logical Configuration Concepts

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS2069

v Multipath Usage Guide

http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide

v System Storage Interoperation Center (SSIC)

http://www.ibm.com/systems/support/storage/config/ssic/

v Subsystem Device Driver for Linux http://www.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D430

&uid=ssg1S4000107&loc=en_US&cs=utf-8&lang=en#DM

v IBM Storage Interoperability matrices

http://www.ibm.com/systems/storage/product/interop.html

v http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1334

http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1334

Appendix D. Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in

other countries. Consult your local IBM representative for information on the

products and services currently available in your area. Any reference to an IBM

product, program, or service is not intended to state or imply that only that IBM

product, program, or service may be used. Any functionally equivalent product,

program, or service that does not infringe any IBM intellectual property right may

be used instead. However, it is the user’s responsibility to evaluate and verify the

operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter

described in this document. The furnishing of this document does not give you

any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation

North Castle Drive

Armonk, NY 10504-1785

U.S.A.

The following paragraph does not apply to the United Kingdom or any other

country where such provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS

PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER

EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS

FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or

implied warranties in certain transactions, therefore, this statement may not apply

to you.

This information could include technical inaccuracies or typographical errors.

Changes are periodically made to the information herein; these changes will be

incorporated in new editions of the publication. IBM may make improvements

and/or changes in the product(s) and/or the program(s) described in this

publication at any time without notice.

Licensees of this program who wish to have information about it for the purpose

of enabling: (i) the exchange of information between independently created

programs and other programs (including this one) and (ii) the mutual use of the

information which has been exchanged, should contact:

IBM Corporation

Dept. LRAS/Bldg. 905

11501 Burnet Road

Austin, TX 78758-3498

U.S.A.

Such information may be available, subject to appropriate terms and conditions,

including in some cases, payment of a fee.

The licensed program described in this document and all licensed material

available for it are provided by IBM under terms of the IBM Customer Agreement,

IBM International Program License Agreement or any equivalent agreement

between us.

For license inquiries regarding double-byte (DBCS) information, contact the IBM

Intellectual Property Department in your country or send inquiries, in writing, to:

IBM World Trade Asia Corporation

Licensing

2-31 Roppongi 3-chome, Minato-ku

Tokyo 106-0032, Japan

IBM may use or distribute any of the information you supply in any way it

believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of

those products, their published announcements or other publicly available sources.

IBM has not tested those products and cannot confirm the accuracy of

performance, compatibility or any other claims related to non-IBM products.

Questions on the capabilities of non-IBM products should be addressed to the

suppliers of those products.

Any references in this information to non-IBM Web sites are provided for

convenience only and do not in any manner serve as an endorsement of those Web

sites. The materials at those Web sites are not part of the materials for this IBM

product and use of those Web sites is at your own risk.

This information contains examples of data and reports used in daily business

operations. To illustrate them as completely as possible, the examples include the

names of individuals, companies, brands, and products. All of these names are

fictitious and any similarity to the names and addresses used by an actual business

enterprise is entirely coincidental.

Trademarks

® IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of

International Business Machines Corporation in the United States, other countries,

or both. If these and other IBM trademarked terms are marked on their first

occurrence in this information with a trademark symbol (®tm; and &tm;), these

symbols indicate U.S. registered or common law trademarks owned by IBM at the

time this information was published. Such trademarks may also be registered or

common law trademarks in other countries. A current list of IBM trademarks is

available on the Web at Copyright and trademark information at

www.ibm.com/legal/copytrade.shtml

® ® Adobe , the Adobe logo, PostScript , and the PostScript logo are either registered

trademarks or trademarks of Adobe Systems Incorporated in the United States,

and/or other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other

countries, or both.

Other company, product, or service names may be trademarks or service marks of

others.

28 Blueprints: Installing Linux Distributions on Multipath Devices

Terms and conditions

Permissions for the use of these publications is granted subject to the following

terms and conditions.

Personal Use: You may reproduce these publications for your personal,

noncommercial use provided that all proprietary notices are preserved. You may

not distribute, display or make derivative works of these publications, or any

portion thereof, without the express consent of the manufacturer.

Commercial Use: You may reproduce, distribute and display these publications

solely within your enterprise provided that all proprietary notices are preserved.

You may not make derivative works of these publications, or reproduce, distribute

or display these publications or any portion thereof outside your enterprise,

without the express consent of the manufacturer.

Except as expressly granted in this permission, no other permissions, licenses or

rights are granted, either express or implied, to the publications or any data,

software or other intellectual property contained therein.

The manufacturer reserves the right to withdraw the permissions granted herein

whenever, in its discretion, the use of the publications is detrimental to its interest

or, as determined by the manufacturer, the above instructions are not being

properly followed.

You may not download, export or re-export this information except in full

compliance with all applicable laws and regulations, including all United States

export laws and regulations.

THE MANUFACTURER MAKES NO GUARANTEE ABOUT THE CONTENT OF ″ ″ THESE PUBLICATIONS. THESE PUBLICATIONS ARE PROVIDED AS-IS AND

WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED,

INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF

MERCHANTABILITY, NON-INFRINGEMENT, AND FITNESS FOR A

PARTICULAR PURPOSE.

Appendix D. Notices 29 30 Blueprints: Installing Linux Distributions on Multipath Devices

Printed in USA