HP-UX Technical Maximum LUN configuration and considerations for HP-UX January 2002 A Whitepaper on Maximum LUN limitations Version 1.0 on HP-UX U.S.A. ©2002 Hewlett-Packard Company Table of Contents 1. Abstract ......................................................................................................................................... 1 2. Definitions...................................................................................................................................... 1 3. Supported LUN Limits on HP-UX ....................................................................................................... 2 4. Device LUN Limits............................................................................................................................ 3 5. Architectural Limits.......................................................................................................................... 3 6. Considerations................................................................................................................................ 3 6.1 Server considerations ................................................................................................................. 4 6.2 Kernel Tunables......................................................................................................................... 4 6.3 Volume Manager considerations................................................................................................. 4 6.4 File System considerations .......................................................................................................... 4 6.5 Scan/Startup/Shutdown times.................................................................................................... 4 7. Roadmap / Futures......................................................................................................................... 5 8. More Information............................................................................................................................ 5 ii Maximum LUN Configurations and Consideration for HP-UX 1. Abstract Mass storage requirements have expanded dramatically over the last few years. High-end HP-UX customers now require Terabytes of storage and thousands of storage devices (LUNs) accessible from a single HP-UX server. This paper will address HP-UX limits on the maximum number of LUNs accessible in various ways from an HP-UX server, and significant LUN limits that exist at a sub- component level such as per-HBA (Host Bus Adapter) limits. The nature of these limits will be defined, and issues, recommendations, or restrictions that apply when operating at or near these limits will be discussed, along with expectations for the future and what to do if you need to exceed existing supported levels. 2. Definitions The primary mass storage device protocol in use on HP-UX and in the workstation and server industry in general is the SCSI device protocol, which is used across both parallel SCSI and Fibre Channel in HP-UX today, and will be used across other links in the future. The SCSI protocol defines the term LU (Logical Unit) to refer to an independently controllable device or to an independently controllable piece of storage in a storage array device. Technically the term LUN (Logical Unit Number) refers to an addressing identifier of the LU, but colloquially it is used to refer to the device or piece of storage, or even a hardware path to the device. In this paper, to distinguish between the actual device and a path to that device, we'll use the term "LUN device" to mean the actual piece of storage, and we'll use the term "LUN path" or simply "LUN" to mean "a specific hardware path to the piece of storage". There can be multiple LUN paths to a given LUN device. On an HP-UX system, a “LUN” can be opened, accessed via appropriate I/O or control operations, and closed. For user access a LUN typically has at least one device file associated with it. Volume Managers or dynamic-multi-pathing modules running in HP-UX may abstract which path is being accessed for a given LUN device, in which case an open of such an abstracted LUN path provides a mechanism for access to the LUN device independently of any specific path. Two cases of abstracted LUN paths currently used on HP-UX occur in alternate path failover functionality (e.g., LVM’s PVLinks), and in dynamic-multi-pathing (DMP) functionality (e.g., Veritas’ Volume Manager (VxVM) and EMC’s PowerPath product). “Dynamic-multi-pathing” (DMP) refers to the capability to determine the various paths to a given LUN device at runtime and to dynamically load- balance I/Os across those paths. As a result, all LUN paths that are accessed via a DMP module are considered load-bearing and hence “active” in the definitions below. DMP modules tend to include the alternate path failover functionality as well, but the key characteristic of DMP modules important for this paper is that they increase the number of “active” paths that exist on the system. LVM’s PVLinks provides a mechanism for abstracting multiple paths (traditionally only 2 paths per LUN device, but recently on 11.0 this has been extended to up to 8 paths per LUN device) for purposes of failover. Only one of the paths is a load-bearing path at any point in time. For purposes of LUN count limitations we’ll define the following terms: · Active LUN: A load-bearing path to a LUN device. · Open LUN: An "Active LUN" or a LUN that is open with only occasional I/O to the LUN. · Visible LUN: An "Open LUN" or a LUN that is typically not open but is visible to ioscan. Page 1 Maximum LUN Configurations and Consideration for HP-UX Each of these three definitions will be used in defining various limits, as described in the next section below. Note that all PVLinks paths to a given LUN device are considered “Open” but only one of the paths is active at any point in time. Hence if a system has 2000 LUN devices attached to it with 2 paths to each device for a total of 4000 LUN paths, all of which are PVLink pairs, then the number of Active LUNs would be 2000 and the number of Open and Visible LUNs would be 4000. On the other hand, if a system has 2000 attached LUN devices with 2 paths to each device, all of which are accessed through a DMP module the number of active LUNs would be 4000. The term HBA (Host Bus Adapter) in this paper is used to refer to a mass storage adapter (e.g., SCSI or Fibre Channel) that attaches to a system I/O bus such as PCI. 3. Supported LUN Limits on HP-UX The limits associated with LUNs on HP-UX are for the most part tested limitations, not architectural. That is, the limits are the limit of what has been tested on HP-UX, and unless otherwise specified are not hard architected limits that would require hardware or software redesign to move beyond. These limits specify the levels at which HP provides general support on HP-UX 10.20, 11.0, and 11i. Customers must contact their HP support personnel to discuss support of higher LUN levels than those indicated below. As of the writing of this paper, HP-UX supports maximum LUN configurations as defined in the following table: Table 1 - Max LUN Limits HP-UX Releases 10.20 11.0 11i Active LUNs 768 2400 4096 Open/Visible LUNs 1536 4800 8192 LUNs/HBA (non-HA) 512 512 512 LUNs/HBA (HA) 320 320 512 The LUNs/HBA (HA and non-HA) rows indicate the maximum number of Visible LUNs supported through a single HBA port. In this context “HA” refers to usage of the HBA in an environment in which the MC-ServiceGuard or ServiceGuard-OPS (Oracle Parallel Server) products are being used. These limits may change over time as larger configurations are tested and supported, and to support ever larger requirements on future releases of HP-UX. Examples of supported configurations on HP-UX 11i, given the definitions and limits in Table 1, would include: a) 4096 LUN devices, with 2 paths to each device, all controlled by LVM’s PVLinks: Þ 4096 Active LUNs, 8192 Open and Visible LUNs b) 4100 LUN devices, with 2 paths to each device, and 4090 of them controlled by LVM’s PVLinks and the other 10 LUNs only used sporadically: Þ 4090 Active LUNs, 8180 Open LUNs, and 8190 Visible LUNs Page 2 Maximum LUN Configurations and Consideration for HP-UX c) 2048 LUN devices, with 4 paths to each device, all controlled by LVM’s PVLinks: Þ 2048 Active LUNs, 8192 Open and Visible LUNs d) 1024 LUN devices, with 4 paths to each device, all accessed via DMP modules: Þ 4096 Active LUNs, 4096 Open and Visible LUNs e) 256 LUN devices, with 16 paths to each device, all accessed via DMP modules: Þ 4096 Active LUNs, 8192 Open and Visible LUNs 4. Device LUN Limits In addition to the system limits specified above, the storage device itself can have various LUN limitations both internally and with respect to the configuration/topology that it can reside within. Make sure you consider all of the possible limits across the storage device, adapter, and the system in your LUN count determinations. Contact the storage device vendor or your HP support personnel for details on device-specific limits. 5. Architectural Limits Architectural limits that exist on HP-UX are well above the supported limits listed above and are not expected to be encountered by customers at those levels. One key architectural limit that currently exists, which will be removed in future releases of HP-UX, is a 32768 LUN limit that results from the 15 bits currently available for LUN addressing in the device file’s minor number. This limit includes so-called “LUN holes” (non-existent