HP-UX Technical

Maximum LUN configuration and considerations for HP-UX

January 2002 A Whitepaper on Maximum LUN limitations Version 1.0 on HP-UX

U.S.A.

©2002 Hewlett-Packard Company Table of Contents 1. Abstract ...... 1 2. Definitions...... 1 3. Supported LUN Limits on HP-UX ...... 2 4. Device LUN Limits...... 3 5. Architectural Limits...... 3 6. Considerations...... 3 6.1 Server considerations ...... 4 6.2 Kernel Tunables...... 4 6.3 Volume Manager considerations...... 4 6.4 File System considerations ...... 4 6.5 Scan/Startup/Shutdown times...... 4 7. Roadmap / Futures...... 5 8. More Information...... 5

ii Maximum LUN Configurations and Consideration for HP-UX

1. Abstract

Mass storage requirements have expanded dramatically over the last few years. High-end HP-UX customers now require Terabytes of storage and thousands of storage devices (LUNs) accessible from a single HP-UX server. This paper will address HP-UX limits on the maximum number of LUNs accessible in various ways from an HP-UX server, and significant LUN limits that exist at a sub- component level such as per-HBA (Host Bus Adapter) limits. The nature of these limits will be defined, and issues, recommendations, or restrictions that apply when operating at or near these limits will be discussed, along with expectations for the future and what to do if you need to exceed existing supported levels.

2. Definitions

The primary mass storage device protocol in use on HP-UX and in the workstation and server industry in general is the SCSI device protocol, which is used across both parallel SCSI and in HP-UX today, and will be used across other links in the future. The SCSI protocol defines the term LU (Logical Unit) to refer to an independently controllable device or to an independently controllable piece of storage in a storage array device. Technically the term LUN (Logical Unit Number) refers to an addressing identifier of the LU, but colloquially it is used to refer to the device or piece of storage, or even a hardware path to the device. In this paper, to distinguish between the actual device and a path to that device, we'll use the term "LUN device" to mean the actual piece of storage, and we'll use the term "LUN path" or simply "LUN" to mean "a specific hardware path to the piece of storage". There can be multiple LUN paths to a given LUN device.

On an HP-UX system, a “LUN” can be opened, accessed via appropriate I/O or control operations, and closed. For user access a LUN typically has at least one device file associated with it. Volume Managers or dynamic-multi-pathing modules running in HP-UX may abstract which path is being accessed for a given LUN device, in which case an open of such an abstracted LUN path provides a mechanism for access to the LUN device independently of any specific path.

Two cases of abstracted LUN paths currently used on HP-UX occur in alternate path failover functionality (e.g., LVM’s PVLinks), and in dynamic-multi-pathing (DMP) functionality (e.g., Veritas’ Volume Manager (VxVM) and EMC’s PowerPath product). “Dynamic-multi-pathing” (DMP) refers to the capability to determine the various paths to a given LUN device at runtime and to dynamically load- balance I/Os across those paths. As a result, all LUN paths that are accessed via a DMP module are considered load-bearing and hence “active” in the definitions below. DMP modules tend to include the alternate path failover functionality as well, but the key characteristic of DMP modules important for this paper is that they increase the number of “active” paths that exist on the system.

LVM’s PVLinks provides a mechanism for abstracting multiple paths (traditionally only 2 paths per LUN device, but recently on 11.0 this has been extended to up to 8 paths per LUN device) for purposes of failover. Only one of the paths is a load-bearing path at any point in time.

For purposes of LUN count limitations we’ll define the following terms:

· Active LUN: A load-bearing path to a LUN device. · Open LUN: An "Active LUN" or a LUN that is open with only occasional I/O to the LUN. · Visible LUN: An "Open LUN" or a LUN that is typically not open but is visible to ioscan.

Page 1 Maximum LUN Configurations and Consideration for HP-UX

Each of these three definitions will be used in defining various limits, as described in the next section below.

Note that all PVLinks paths to a given LUN device are considered “Open” but only one of the paths is active at any point in time. Hence if a system has 2000 LUN devices attached to it with 2 paths to each device for a total of 4000 LUN paths, all of which are PVLink pairs, then the number of Active LUNs would be 2000 and the number of Open and Visible LUNs would be 4000.

On the other hand, if a system has 2000 attached LUN devices with 2 paths to each device, all of which are accessed through a DMP module the number of active LUNs would be 4000.

The term HBA (Host Bus Adapter) in this paper is used to refer to a mass storage adapter (e.g., SCSI or Fibre Channel) that attaches to a system I/O bus such as PCI.

3. Supported LUN Limits on HP-UX

The limits associated with LUNs on HP-UX are for the most part tested limitations, not architectural. That is, the limits are the limit of what has been tested on HP-UX, and unless otherwise specified are not hard architected limits that would require hardware or software redesign to move beyond. These limits specify the levels at which HP provides general support on HP-UX 10.20, 11.0, and 11i. Customers must contact their HP support personnel to discuss support of higher LUN levels than those indicated below.

As of the writing of this paper, HP-UX supports maximum LUN configurations as defined in the following table:

Table 1 - Max LUN Limits

HP-UX Releases 10.20 11.0 11i Active LUNs 768 2400 4096 Open/Visible LUNs 1536 4800 8192 LUNs/HBA (non-HA) 512 512 512 LUNs/HBA (HA) 320 320 512

The LUNs/HBA (HA and non-HA) rows indicate the maximum number of Visible LUNs supported through a single HBA port. In this context “HA” refers to usage of the HBA in an environment in which the MC-ServiceGuard or ServiceGuard-OPS (Oracle Parallel Server) products are being used.

These limits may change over time as larger configurations are tested and supported, and to support ever larger requirements on future releases of HP-UX. Examples of supported configurations on HP-UX 11i, given the definitions and limits in Table 1, would include:

a) 4096 LUN devices, with 2 paths to each device, all controlled by LVM’s PVLinks: Þ 4096 Active LUNs, 8192 Open and Visible LUNs

b) 4100 LUN devices, with 2 paths to each device, and 4090 of them controlled by LVM’s PVLinks and the other 10 LUNs only used sporadically: Þ 4090 Active LUNs, 8180 Open LUNs, and 8190 Visible LUNs

Page 2 Maximum LUN Configurations and Consideration for HP-UX

c) 2048 LUN devices, with 4 paths to each device, all controlled by LVM’s PVLinks: Þ 2048 Active LUNs, 8192 Open and Visible LUNs

d) 1024 LUN devices, with 4 paths to each device, all accessed via DMP modules: Þ 4096 Active LUNs, 4096 Open and Visible LUNs

e) 256 LUN devices, with 16 paths to each device, all accessed via DMP modules: Þ 4096 Active LUNs, 8192 Open and Visible LUNs

4. Device LUN Limits

In addition to the system limits specified above, the storage device itself can have various LUN limitations both internally and with respect to the configuration/topology that it can reside within. Make sure you consider all of the possible limits across the storage device, adapter, and the system in your LUN count determinations. Contact the storage device vendor or your HP support personnel for details on device-specific limits.

5. Architectural Limits

Architectural limits that exist on HP-UX are well above the supported limits listed above and are not expected to be encountered by customers at those levels. One key architectural limit that currently exists, which will be removed in future releases of HP-UX, is a 32768 LUN limit that results from the 15 bits currently available for LUN addressing in the device file’s minor number. This limit includes so-called “LUN holes” (non-existent LUNs between two existing LUNs, or non-existent LUNs after an existing LUN up to the next multiple of LUN number 128 in a disk array), and is also decreased by approximately 128 for every attached parallel SCSI bus (or more accurately 128 minus the number of LUNs on the parallel SCSI bus, since those LUNs can be counted in the “visible” total). So this effectively decreases this architectural limit with respect to visible LUNs in a sparse LUN configuration to something less than 32768, but should still be well above the 8192 supported limit given in Section 3. In any event, this architectural limit will be removed in future releases of HP-UX.

Another key architectural limit that currently exists, and which is also expected to be removed in future releases of HP-UX, is a limit on the number of physical volumes that can be created using LVM or VxVM volume managers. Currently the volume managers on HP-UX are limited to 256 volume groups (VGs) with 255 physical volumes per VG, for a total of 65280 physical volumes. Clearly this is well above the supported limits listed above and customers are not expected to be impacted by this architectural limit. As noted in the “LVM Considerations” section below, we currently recommend that the number of physical volumes be limited to 32 per VG for a well-balanced system, but this is not required.

6. Considerations

Whenever a system is configured, the number of LUNs must be properly balanced with respect to the various I/O links and buses and with respect to the configuration of volume manager volumes and volume groups, the size of memory, the number of CPUs, the corresponding application workload, etc. The details of system and mass storage configuration and tuning to support large LUN configurations are beyond the scope of this paper, but a few general guidelines and specific considerations applicable to large LUN configurations are outlined here.

Page 3 Maximum LUN Configurations and Consideration for HP-UX

6.1 Server considerations

Generally speaking the larger LUN levels are expected to only be obtained on the larger HP-UX servers such as the Superdome, V-Class, or N-Class. Smaller servers can share in the use of large numbers of LUNs in a cluster, but due to memory and slot limitations would generally not be able to in and of themselves support large numbers of LUNs.

6.2 Kernel Tunables

As the number of LUNs is increased, the “maxfiles” and “maxfiles_lim” kernel tunables will typically also need to be increased. Contact HP support personnel for help in tuning your specific configuration.

6.3 Volume Manager considerations

The LVM (Logical Volume Manager) product in HP-UX currently allows for an architectural limit of 256 VGs (volume groups) with up to 256 PVs (physical volumes) per VG. Therefore, given the 11i limits specified above, you can configure a system with only 16 VGs, each of which contain 256 PVs, for a total of 4096 PVs (where a PV is equivalent to a LUN). However, for a well-balanced configuration a limit of 32 PVs per VG is recommended.

Also, for planning purposes, you should keep in mind that you effectively limit the total amount of disk space that a volume group can contain when the VG is created, based on the extent size and the maximum number of extents specified at VG creation. Therefore, to ease future expansion, the limits specified at VG creation should be carefully considered.

6.4 File System considerations

The use of HFS file systems with large numbers of LUNs is not recommended due to recovery times; use the VxFS file system with large LUN levels.

6.5 Scan/Startup/Shutdown times

As the number of LUNs increase, the time it takes to discover LUNs via ioscan or other device query utilities increases. Future versions of HP-UX are expected to significantly flatten out the increase, but currently a system with 8000 visible LUNs can take on the order of 30 minutes or more to scan and probe the hardware. Because ioscans are done during boot, this can also significantly impact the boot time. Similarly, it can impact application startup times, for applications that do an ioscan or similar general device discovery.

A set of patches have been released for both HP-UX 11.00 and 11.11 releases which, when applied together, reduce the ioscan time significantly. The following table lists these patches. Apply these patches (or patches which supersede) together for reducing ioscan time.

Table 2 : ioscan performance improvement patches

HP-UX 11.00 PHCO_24199 PHKL_24165 PHKL_24187 HP-UX 11.11 PHCO_24198 PHKL_24163 PHKL_25165

Page 4 Maximum LUN Configurations and Consideration for HP-UX

7. Roadmap / Futures

To meet ever-increasing customer requirements, the LUN limitations in HP-UX will continue to increase over time and with the release of new versions of HP-UX, both in terms of the supported LUN limits and architectural limits. Also, HP-UX will be working on significantly decreasing and flattening the overheads associated with managing, scanning, and displaying large numbers of LUNs, providing for streamlined management of large storage attachments.

8. More Information Visit http://docs.hp.com/netcom web site for more information on HP's Fibre Channel and SCSI host bus adapters. http://www.hp.com/products1/storage/san/index.html gives information on (SAN) connectivity options for HP-UX.

Page 5 Maximum LUN Configurations and Consideration for HP-UX

Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this white paper, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. ©Copyright 2001 Hewlett-Packard Company, all rights reserved. HP-UX® is a registered trademark of the Hewlett-Packard Corporation. All other trademarks and registered trademarks are the property of the respective corporations. Reproduction, adaptation, or translation of this document without prior written permission is prohibited, except as allowed under the copyright laws.

Page 6