Multi-Path I/O Computer Solution Blueprints with Serial Attached SCSI Devices

Abstract This paper provides several Multi-Path I/O (MPIO) computer solution blueprints for computer storage planning with Serial Attached SCSI products. The SCSI, Fibre Channel and Serial ATA (SATA) devices have been widely used in the MPIO storage designs and computer cluster system implementations. To satisfy the data center requirement of scalability, performance, reliability, and manageability, Serial Attached SCSI (SAS) is developed to utilize a common electrical and physical connection interface from Serial ATA. SAS MPIO storage solutions are designed for customers running enterprise applications that require high-performance, minimized downtime, and expandable storage architecture. In this paper, we will discuss SAS device implementations in the MPIO, in the computer RAID storage system, and in the computer cluster. The conclusion section discusses the benefits of SAS MPIO implementation in the current market, and the future work section discusses performance enhancement and distance fault-tolerance development.

Contents Overview of Serial Attached SCSI Architecture...... 3 SAS Connections with Physical links, Ports and phys...... 3 Port Name and SAS Addresses...... 3 SAS Dual-Port Disk Drive...... 4 SAS Connections with Expanders...... 4 SAS Pathways...... 4 SAS Domain and SATA Domain...... 4 Construct SAS MPIO Paths with SAS Devices...... 6 Summary of Storage Devices’ Characteristics...... 6 SAS Devices’ Characteristics for MPIO Solutions...... 6 General MPIO Functions and Configurations...... 7 MPIO and Cluster Configuration Blueprints using SAS Products...... 7 Construct MPIO Paths with SAS Devices...... 7 SAS MPIO Configuration and Management Software Requirement...... 9 Automatic Load-Balancing...... 10 Transparent Failover...... 10 Automatic Failback...... 10 Scalability with Minimized Downtime...... 10 SAS MPIO Implementation Blueprints...... 10 Single Server MPIO Configuration with Dual-Port SAS Disk Drives...... 11 MPIO Cluster Configuration with SAS Storage Devices...... 12 Multi-Node MPIO Cluster Configuration with SAS...... 12 Conclusions and Future Works...... 13 Acronyms...... 14 Terms...... 14 Multi-Path I/O Computer Solution Blueprints with Serial Attached SCSI Devices - 2

Windows Hardware Engineering Conference

Author's Disclaimer and Copyright: The copyright of this white paper is owned by LSI Logic. LSI Logic Corporation is a leading designer and manufacturer of communications, consumer and storage semiconductors for applications that access, interconnect and store data, voice and video. In addition, the company supplies storage network solutions for the enterprise. LSI Logic is headquartered at 1621 Barber Lane, Milpitas, CA 95035, http://www.lsilogic.com.

WinHEC Sponsors’ Disclaimer: The contents of this document have not been authored or confirmed by Microsoft or the WinHEC conference co-sponsors (hereinafter “WinHEC Sponsors”). Accordingly, the information contained in this document does not necessarily represent the views of the WinHEC Sponsors and the WinHEC Sponsors cannot make any representation concerning its accuracy. THE WinHEC SPONSORS MAKE NO WARRANTIES, EXPRESS OR IMPLIED, WITH RESPECT TO THIS INFORMATION.

Microsoft, Windows, and Windows Server are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Other product and company names mentioned herein may be the trademarks of their respective owners.

WinHEC 2003 Microsoft Windows Hardware Engineering Conference Multi-Path I/O Computer Solution Blueprints with Serial Attached SCSI Devices - 3

Overview of Serial Attached SCSI Architecture

Rapidly shrinking form factors and rising performance expectations are placing a new set of challenges in front of storage system designers. Just as smaller, denser blade-type servers are driving the deployment of smaller form-factor disk drives, the limitations of existing bus architectures and parallel interfaces are making it increasingly difficult for designers to match drive performance to capacity. Serial Attached SCSI (SAS) technology is developed to meet future I/O and direct- attached storage requirements. In Figure 1, a SAS initiator could directly connect to a SAS target or through a SAS expander.

Target SAS Initiator

Target SAS Initiator SAS Expander

Figure 1: SAS Device Connections

SAS Connections with Physical links, Ports and phys SAS connections are constructed with physical links, ports, phys, and expanders. A physical link is a set of four wires used as two differential signal pairs. One differential signal transmits in one direction while the other differential signal transmits in the opposite direction. Data may be transmitted in both directions simultaneously. A phy is a transceiver; it is the object in a device that electrically interfaces to a physical link. Phys are contained in ports. A narrow port has only one phy, and a wide port has more than one phy in the port. A wide link is the set of physical links that attach a wide port to another wide port, and a narrow link is the physical link that attaches a narrow port to another narrow port.

Port Name and SAS Addresses Each SAS initiator port, SAS target port, SAS target/initiator port, and SAS expander port shall have a SAS address. The SAS address shall be worldwide unique. Device names are worldwide unique names for devices within a transport protocol. Port names are worldwide unique names for ports within a transport protocol. SAS port identifiers are the values by which ports are identified within a domain, and are used as SAS addresses. Phy identifiers are unique within a device. Each phy in a device shall be assigned a unique 8-bit identifier within the device. The phy identifier is used for management functions.

SAS Dual-Port Disk Drive A dual-transceiver SAS disk drive that doesn’t support a wide port must be implemented as two target devices, each with a unique device name, each containing a single port. Two target devices are allowed to share the same logical unit. When they do so, they present them with different LUNs on different virtual devices. A dual-transceiver disk drive that does support a wide port is implemented as one target device, with one device name, containing a single port.

SAS Connections with Expanders SAS Expander is a device that contains expander ports. Expander ports may support being attached to SAS initiator ports, SAS and/or SATA target ports, and to other expander ports. An expander device receives primitives, device service requests, and task management requests on one expander port and routes those requests to another expander port. An expander device supports Serial Management Protocol (SMP) and may also contain SCSI devices or ATA devices. SAS expander is capable of supporting multiple initiator and multiple target configurations. There are two types of expander devices differentiated by the routing attributes of their phys. One is edge expander, and the other is fanout expander. Edge expander devices may contain phys with the subtractive routing attribute. A fanout expander device does not contain phys with the subtractive routing attribute. Edge expander devices and the fanout expander device may also contain phys with the direct routing and table routing attributes. Expander devices are part of the SAS configuration implementation. A Fanout Expander could connect up to 128 Edge Expander device set, and an Edge Expander device set could contain up to 128 physical links. In Figure 2, one Fanout Expander could connect up to 16384 SAS devices.

Fanout Expander Device

Edge Expander Edge Device 128 Edge Expander Set Edge Expander Expander Device Device Device Set Set

Edge Edge 128 physical Links Expander Expander Edge Expander per Edge Device Device Device Expander Device Ser

Initiator or Target Initiator or Target Initiator or Target Initiator or Target Device Device Initiator or Target Initiator or Target InitiatorDevice or Target InitiatorDevice or Target Device Device Initiator or Target Initiator or Target InitiatorDevice or Target InitiatorDevice or Target Device Device Device Device

Figure 2: SAS Fanout Expander and Edge Expander device connections

SAS Pathways An SAS pathway is the physical route of a connection. In the case, where an initiator phy is directly attached to a target phy, the pathway and the physical link are identical. In some cases, where there are expander devices between an initiator

WinHEC 2003 Microsoft Windows Hardware Engineering Conference Multi-Path I/O Computer Solution Blueprints with Serial Attached SCSI Devices - 5 phy and a target phy, the pathway consists of all the physical links required to route dwords between the initiator phy and the target phy. The physical links may or may not be using the same physical link rate. A SAS partial pathway is the set of physical links participating in a connection request that has not reached the destination (for example, the connection request (OPEN address frame) has been transmitted by the connection originator and the originator has received at least one arbitration in progress status). A partial pathway is blocked when path resources it requires are held by either another connection or another partial pathway.

SAS Domain and SATA Domain A SAS domain may consist of ATA devices, SCSI devices, or SAS devices with SAS ports interfacing to the service delivery subsystem through phys. The service delivery subsystem in a SAS domain may include expander devices. Expander devices contain expander ports interfacing to the service delivery subsystem through phys. Serial SCSI Protocol (SSP), Serial ATA Tunneled Protocol (STP) and Serial Management Protocol (SMP) are three transport protocols used in the SAS interconnect. SSP is a mapping of SCSI protocol supporting multiple initiators and targets, STP is a mapping of SATA protocol expanded to support multiple initiators and targets, and SMP is a management protocol.

SAS Initiator SATA Initiator SAS Initiator SATA Initiator Port(s) Port(s) Port(s) Port(s)

SSP STP

SAS Expander SSP SATA Device(s)

SSP STP

SAS Target SATA Target SAS Target SATA Target Port(s) Port(s) Port(s) Port(s)

SAS Domain SAS Domain SATA Domain

Figure 3: SAS Domain and SATA Domain

Construct SAS MPIO Paths with SAS Devices SAS MPIO configuration allows a server to be connected to a SAS RAID storage system through multiple and independent I/O paths. In an unlikely event of a host- to-storage path failure, SAS MPIO automatically redirects I/O throughput to a redundant I/O path without interruption to data transfer. This automatic failover assures users with continuous access to the stored data assets and uninterrupted operation of software applications. With SAS MPIO, the data transition is not affected by the failure of vital components of the storage system. Automatic load balancing, transparent failover, automatic failback, and scalability are the main features of the SAS MPIO implementations.

Summary of Storage Devices’ Characteristics

Interface type Max.Transfer Rates Interface type Connection Max. Devices Cable Length Applications SCSI 160 MBps /320 MBps parallel 50/68-pin cable 16 SE-6m Server and High (8/16-bit) internal and LVD-12m End Workstation external HVD-25m Storage Fibre Channel 100 MBps / 1 Gbps serial Copper/Optical 16 Million - 1 30m (copper) High End Computer 200 MBps / 2 Gbps external and in 10Km (optical) Storage Systems Storage systems

ATA (IDE) 100 MBps parallel 40-pin cable 2 0.4m Internal storage (16-bit) serial ATA in 150 MBps / 1.5 Gbps serial Copper 1 1m Internal storage 2002 multiple ports/PC Serial 300 MBps, 600 MBps serial Copper Max to 16383 8m Internal storage or Attached SCSI using Expanders external storage

USB 1 MBps / 12 Mbps serial 6-pin USB 127 5m Interconnection PC connector w/Peripherals USB2 60 MBps / 480 Mbps serial 6-pin USB 127 5m Interconnection PC (in future) connector w/Peripherals

Table 1: Summary of Storage Devices’ characteristics

SAS Devices’ Characteristics for MPIO Solutions  With SAS Expander routing and transmission topologies, SAS can support multiple initiators and multiple targets in the same SAS domain. SAS is an ideal solution for the MPIO system in a rack, cluster-in-a-rack, or cluster-in-a-room configuration.  One SAS initiator can support up to 16383 devices in one SAS transmission bus, and the cable length can support up to 8 meters.  SAS pathway with SAS connection ports, as well as SAS connection phys and expanders could provide a highly redundant storage structure for the high- availability computer storage solution.

General MPIO Functions and Configurations MPIO allows multiple Host Bus Adapters (HBAs) to be connected to the same bus and multi-ported devices. This serves two purposes: First, it provides high availability to the devices when an HBA fails. Upon controller failure, the operating system will redirect I/O through another I/O path to the device. Second, it provides load balancing of I/O to the device across multiple controllers. The primary functions of multi-path I/O storage devices are:

 High availability—Minimize unexpected and scheduled disk downtime.  Performance—Enhance the performance through dynamic load balancing for those supported storage devices. Currently, the MPIO configuration has been implemented in the high-end single server or the highly available cluster server system with shared SCSI or Fibre Channel devices.

Target Server

Target

RAID Controller Initiator 1 Target

Target RAID Controller Initiator 2 Target

Figure 4: Single Server with MPIO configuration

Server 1 Target

Target Initiator 1,2 RAID Controller 1 Inter-connect Target Server 2 Target RAID Controller 2 Initiator 3,4 Target

Figure 5: A Cluster Server System with MPIO configuration

MPIO and Cluster Configuration Blueprints using SAS Products In this section, we discuss the MPIO Path ID and the implementations of MPIO Path ID. SAS MPIO configuration operates on a server connected to a SAS RAID storage system by multiple and independent I/O paths. In the unlikely event of a host-to-storage path failure, SAS MPIO automatically redirects I/O throughput to a redundant I/O path without interruption to data transfer. This automatic failover assures users with continuous access to stored data assets, and uninterrupted operation of software applications. With SAS MPIO, the data transition is not affected by the failure of vital components of the storage system. Automatic load balancing, transparent failover, automatic failback and scalability are the main features of the SAS MPIO implementations.

Construct MPIO Paths with SAS Devices SAS MPIO configuration allows a server connected to a SAS RAID storage system through multiple and independent I/O paths. SAS Management Protocol (SMP) and Serial SCSI Protocol (SSP) provide SAS pathway configuration and management scheme to build up SAS MPIO structure. With the unique and identifiable port address and SAS pathway, SAS MPIO path ID could be constructed with the SAS pathway ID. With MPIO path ID, the SAS MPIO configuration software and SAS controller firmware could assign Target’s owner preference to any available MPIO path.

E2-1 Dual-Port A-1 E1-1 Target SAS RAID Controller Port SAS Initiator 1 E1-2 Address Port_SAS Address A-2 SAS Edge Expander Port_SAS Address E2-2

Dual-Port Target SAS RAID Controller Port SAS Initiator 2 Address Port_SAS Address

Figure 6: MPIO Pathway construction with SAS RAID Controller and Dual- Port disk drive In figure 6, there are eight pathways between two Controllers and two dual-port disks. The MPIO path could be determined by the SAS pathway between the initiator port and target port. I/O request routing among the expander and dual-port disk drives could be invisible to the computer operating system.

The MPIO configuration could also utilize the SAS expanders. In figure 7, the MPIO path could be determined by SAS pathway in between the SAS initiator port and SAS RAID Controller port. The RAID configuration behind SAS RAID controller could be invisible to the MPIO configuration. To provide a complete fault-tolerance function, it is necessary for SAS storage system to provide RAID configuration and function. Without the disk RAID function, one single disk failure could disable the MPIO function.

A-1 Target Port SAS SAS RAID SAS Initiator 1 Address Controller 1 R-1 E-1 Port_SAS Address Port_SAS Address

A-2 R-2 E-2 Target Port SAS RAID SAS EdgeExpander SAS SAS Initiator 2 Controller 2 Port_SAS Address Address Port_SAS Address Port_SAS Address

Figure 7: MPIO Pathway construction with External SAS RAID storage

SAS MPIO Configuration and Management Software Requirement SAS MPIO configuration and management software development is based on the above SAS MPIO architecture and SMP. Storage system administrators can easily configure and modify the MPIO paths and automatic load-balancing configuration on their storage system to meet specific application requirements. The SAS MPIO configuration and management software needs to provide a user interface through the system OS, so system administrators can improve system performance and maximize path efficiency by balancing the I/O workload across multiple MPIO paths. With SAS MPIO comprehensive path monitoring and reporting features, storage system administrator could ensure pre-emptive measures against system failure and performance. Data path failures and events are reported and displayed on the log viewer, and recorded in the event log. The path monitoring and error reporting capabilities allows pre-emptive measures to be taken to ensure continuous data availability. This MPIO path monitoring and error reporting utility should be part of the MPIO Management software. Through the path monitoring result and error report, storage system administrator could manually operate the path load- balancing and failure recovery. The following is a list of features supported by SAS MPIO Management Software.

Automatic Load-Balancing SAS MPIO automatic load-balancing capabilities enable you to improve system performance by balancing the I/O workload and maximize path and storage resource efficiency. SAS MPIO configuration software automatically reallocates LUNs among a server’s available I/O paths. By assuming the available bandwidth on other paths, SAS MPIO configuration software ensures that a single I/O path does not become congested. As a result, resource utilization is enhanced and response times become more uniform, enhancing overall data throughput. The optimized queuing models could be also applied into the load-balancing schemes.

Transparent Failover SAS MPIO’s transparent failover capability ensures continuous, uninterrupted business operations. A server can have multiple and separate I/O paths to a storage system. MPIO configuration software automatically detects failed elements in an I/O path and reroutes I/O traffic across an alternate data path without interruption of data transfer. Failover occurs within seconds and is transparent to users.

Automatic Failback After failover occurs, SAS MPIO configuration software continues to monitor the components in the failed data path, and restores the path when it becomes available. MPIO configuration software retries failed paths, recovers resources on the fly, and restores the system to maximum efficiency. The ability to perform automatic failback maintains system integrity by ensuring that the I/O traffic does not remain on the failover path.

Scalability with Minimized Downtime With hot-plug support in the SAS devices, users could add disk drives or SAS expanders into the configuration without shutdown server or storage system. The feature provides minimized downtime to expected and unexpected storage system maintenance and upgrade. SAS MPIO Implementation Blueprints The following section introduces two SAS MPIO implementation blueprints. The blueprints present SAS MPIO implementation in the mid-range storage system designs for single server with MPIO, two-node computer cluster, and multi-node computer cluster system.

Single Server MPIO Configuration with Dual-Port SAS Disk Drives The hardware requirement for building a single server MPIO configuration with dual- port SAS disk drives is as follows.

 Server with more than one available PCI Slot.  Two SAS RAID Controllers.  Two SAS Edge Expanders.  Three or more dual-port SAS disk drives.

Target

Target Server Target

SAS Edge SAS RAID Controller Expander 1 Target Initiator 1

Target SAS Edge SAS RAID Controller Expander 2 Initiator 2 Target

Target

Figure 8: A single server MPIO configuration with Dual-Port disk drives The single server MPIO configuration with dual-port SAS disk drives could provide multi-path I/O failover, failback, and load balancing functions. The MPIO Path ID could be determined by the SAS pathway from RAID controller port to disk drive port. This configuration could be implemented in the MPIO-in-a-Box solution. If the server serves as a database server and an e-mail server, the server could fully utilize the I/O bandwidth with both active paths. This MPIO configuration could add new SAS expanders or disk drives to the MPIO configuration without stopping the server or the storage system. The disk drives of the single server MPIO configuration could failover from one path to the other and follow this procedure: 1. Client or Server detects an application service failure. 2. The MPIO pathway, physical link, or heartbeat is verified. 3. If the MPIO Path fails, Targets (LUNs) failover to the other path. 4. If the MPIO Path is alive and healthy, an alert message to the server administrator is sent.

MPIO Cluster Configuration with SAS Storage Devices The hardware requirements for building a two-node cluster with SAS MPIO configuration:

 Two Servers.  Two SAS Controllers.  One SAS RAID Storage system with dual-controllers.

Server 1 Target

Target

SAS Initiator Target 1,2 SAS RAID SAS Edge Controller 1 Expander 1

Inter-connect Target SAS RAID Server 2 Controller 2 Target SAS Edge Expander 2 Target SAS Initiator 3,4 Target

Target

Figure 9: Two-Node Cluster with SAS MPIO configuration The two-node cluster with SAS MPIO configuration could provide multi-path I/O failover, failback, load balancing functions, and server service failover and failback functions. If the server serves as a database server and an e-mail server, the server could fully utilize the I/O bandwidth with both active paths and both active cluster nodes. In the current market, Oracle Parallel Server, Lotus Domino Cluster (IBM), DB2 Cluster, Microsoft SQL Server™ Cluster, Microsoft Exchange Server Cluster, and the basic Print and File System Server cluster have been widely used in the cluster systems. With the SAS MPIO configuration and management software, the cluster with SAS MPIO could deliver a robust high availability and scalability solution to enterprise system users. This two-node cluster with MPIO configuration could be delivered in a Cluster-in-Box or Cluster-in-a-Rack format. The failover scenario follows the next procedure. 1. Client or Server detects a Cluster service failure. 2. The MPIO pathway, physical link, or heartbeat is verified. 3. If the MPIO Path failed, Targets (LUNs) failover to the other path and reactivate the service. 4. If the MPIO Path is alive, cluster service group failover to the other server. 5. If the other server still fails to activate the service group, alert messages are sent to the system administrator.

Multi-Node MPIO Cluster Configuration with SAS The following Multi-Node cluster configuration could provide minimized downtime, full-scale server service redundancy, I/O path redundancy, and disk drive RAID functions. A Multi-Node cluster system requires a load-balancing optimization utility to control the failover and failback scenarios.

Target

Cluster1-Node 1 SAS Fanout Expander Target

SAS Edge SAS initiator SAS RAID Expander 1 Target 1,2 Controller 1

Target Cluster1-Node 2 SAS RAID Target Controller 2 SAS Edge SAS initiator Expander 2 3,4 Target Server Interconnect Network SAS Edge Target Expander 6 SAS Edge Target Cluster1-Node3 Expander 5

SAS Edge Target Expander 3 SAS initiator SAS RAID Controller 3 5,6 Target

Cluster1-Node4 Target SAS RAID Controller 4 Target SAS initiator SAS Edge 7,8 Expander 4 Target

Target

Figure 10: Multi-Node MPIO cluster configuration with SAS Microsoft Windows Server™ 2003 Data Center, Microsoft SQL Server Cluster, Microsoft Exchange Server Cluster, and the basic Print and File System Server cluster could be widely implemented in the Multi-Node MPIO cluster systems. To deliver a Multi-Node cluster system with SAS into the enterprise market, LSI Logic will develop enterprise solution packages with industry partners for customers’ solution implementation and support.

Conclusions and Future Works In this paper, we present SAS MPIO implementation blueprints for single server MPIO, two-node cluster with MPIO, and multi-node cluster with MPIO. Comparing the SAS MPIO to the MPIO products in the current market, SAS MPIO users could have the following benefits from implementing SAS MPIO into the high-availability system design.

 Lower cost in SAS hardware and software acquisition—The cost of SAS hardware and software would be lower than the fibre-channel hardware, and the cost of SAS hardware would be the same as or lower than the legacy SCSI hardware.  Enhanced hardware configuration—The SAS hardware offers more flexible and longer distance cabling and increases in-device addressability. One SAS connector could support up to 16383 physical links through one fanout expander and 128 edge expanders.  Greater Addressability and Connection Expansion—SAS address topology and SAS expander technology all lead to a new level of flexibility when deploying mainstream data center servers and sub-systems.  High Availability and Scalability Solutions with Legacy SCSI Knowledge—SAS hardware features and protocol design enables the system architect to deliver high availability and scalability solutions with logical SCSI compatibility. The SAS MPIO Cluster and Data Center provide no single point of failure from server to storage.  Support high performance disk drives—The SAS technology will provide robust supports to high performance SAS disk drives with 300 MBps or 600 MBps speeds.  Easy Hardware Setup and Configuration—The SAS configuration uses the SATA cable receptacle connector, cable, and SAS expander. With the reduced connector size, the physical hardware will be much easier that the legacy shared SCSI cables.

SAS MPIO could provide storage solutions to mission-critical computer systems, and SAS could achieve highly reliable and satisfied data transmission and storage requirement. Due to the SAS cable-length limitation, SAS MPIO solution is not suitable for distance fault-tolerance design. Integrating the SAS product with i-SCSI support may provide a distance fault-tolerance configuration solution. The flexibility of SAS device connection design also provides many performance enhancement options to the SAS MPIO configuration. Pursuing the optimal performance with different I/O control algorithms would be future work that is associated with the current SAS MPIO research.

Call to Action and Resources

Call to Action:  For firmware engineers: LSI Logic CO  For device manufacturers: LSI Logic CO  For driver developers: LSI Logic CO Feedback:  To provide feedback about SAS MPIO Solution, please send e-mail to [email protected]. Resources:  Information Technology Industry Council, Serial Attached SCSI (SAS) Protocol Standard Specification. Revision 3, Nov. 21, 2002. Acronyms MBps: megabytes per second (106 bytes per second) SAS: Serial Attached SCSI SATA: Serial ATA SMP: Serial Management Protocol SSP: Serial SCSI Protocol STP: Serial ATA Tunneled Protocol Terms The following terms are defined in SAS ANSI (American National Standard for Information Technology).

 AT Attachment (ATA): A standard for the internal attachment of storage devices to host systems.  ATA domain: An I/O system consisting of a set of ATA devices that communicates with one another by means of a service delivery subsystem.  ATA initiator port: An ATA initiator device object that acts as the connection between application clients and the service delivery subsystem through which requests and responses are routed. Equivalent to a host adapter in ATA (see ATA/ATAPI V1).  ATA port: An ATA initiator port or an ATA target port (see 4.1.1).  ATA target device: An ATA device containing logical units and ATA target ports that receives device service and task management requests for processing.  ATA target port: An ATA target device object that contains a task router and acts as the connection between device servers and task managers and the service delivery subsystem through which requests and responses are routed.  ATA target/initiator port: An ATA target/initiator device object that has all the characteristics of an ATA target port and an ATA initiator port.  Edge expander device: An expander device containing phys with subtractive routing attribute.

 Edge expander device set: A group of one or more edge expander devices.  Expander device: A device that contains expander ports. An expander device receives primitives, device service requests, and task management requests on one expander port and routes those requests to another expander port. An expander device supports SMP and may also contain a SCSI device and/or an ATA device.  Expander phy: A phy in an expander device.  Expander port: A SAS expander device object that routes SSP, SMP, and STP frames to and from physical links or to internal initiator ports and/or target ports. Contains one or more phys.  Narrow link: A physical link that attaches a narrow port to another narrow port.  Narrow port: A port that contains exactly one phy.  Partial pathway: The set of physical links participating in a connection request, which has not reached a SAS endpoint (that is, the connection request has been transmitted by the source device and confirmed as received by at least one expander device with AIP).  Pathway: A set of physical links between a SAS initiator port and a SAS target port.  Phy: A SAS device object that interfaces to a service delivery subsystem.  Physical link: Two differential signal pairs, one pair in each direction, which connect two phys.  Port: A SAS port or an expander port. Each port contains one or more phys.  SAS address: A worldwide unique name assigned to an initiator port, target port, expander device, initiator device, or target device.  SAS device: A SCSI device, an ATA device, or an expander device in a SAS domain.  SAS domain: The I/O system defined by this standard that may serve as an ATA domain and/or a SCSI domain.  SAS initiator device: A SCSI initiator device or an ATA initiator device in a SAS domain.  SAS initiator port: A SCSI initiator port or an ATA initiator port in a SAS domain.  SAS port: A SAS initiator port or a SAS target port.  SAS target device: A SCSI target device or an ATA target device in a SAS domain.  SAS target port: A SCSI target port or an ATA target port in a SAS domain.  SAS target/initiator device: A SCSI target/initiator device or an ATA target/initiator device in a SAS domain.  SAS target/initiator port: A SCSI target/initiator port or an ATA target/initiator port in a SAS domain.  SATA domain: The I/O system defined by SATA that serves as an ATA domain.

 SATA target port: An ATA target port containing one phy in a SAS domain or a SATA domain.  Serial ATA Tunneled Protocol (STP): The protocol defined in this standard used by ATA initiator ports to communicate with SATA target ports in a SAS domain.  Serial Attached SCSI (SAS): The protocol defined by this standard.  Serial Management Protocol (SMP): The protocol defined in this standard used by SAS devices to communicate management information with other SAS devices in a SAS domain.  Serial SCSI Protocol (SSP): The protocol defined in this standard used by SCSI initiator ports to communicate with SCSI target ports in a SAS domain.  SSP initiator port: A SCSI initiator port containing one or more phys in a SAS domain.  SSP target port: A SCSI target port containing one or more phys in a SAS domain.  STP initiator port: An ATA initiator port containing one or more phys in a SAS domain.  STP target port: An ATA target port containing one or more phys in a SAS domain.

WinHEC 2003 Microsoft Windows Hardware Engineering Conference