Marty Czekalski Vice President, Director – SCSI Trade Association Technical Marketing Manager – Maxtor Corporation

1 SerialSerial AttachedAttached SCSISCSI

• Serial Attached SCSI is the next evolution of SCSI beyond U320 • SAS has been designed to be a device and near cabinet interface only, not a network interface • Leverages an enhanced Serial ATA PHY while adding support for a second port • Serial SCSI utilizes features of AND compatibility with SATA drives in a point-to-point, switched architecture ¾ Up to 16k physical links per domain in a switched, point-to-point configuration for scalable performance. ¾ Scalability with wide ports and future speed increases ¾ Data frame based on FCP ¾ Includes rate matching to optimize device transfer speeds ¾ addressing ¾ Uses a simplified protocol that will minimize interoperability issues between controllers & drives ¾ Dual port drives for high availability

2 IndustryIndustry AdoptionAdoption TimelineTimeline

Serial Attached SCSI Spec Serial Attached SCSI Serial Attached SCSI Serial Attached SCSI Transferred to Product Test and Product T10 Design Integration Availability

2002 2003 2004

Interoperability Testing & Plugfests Serial Attached SCSI Specification Serial Attached SCSI Letter Ballot v1.0 Specification Complete

• Twenty-six companies have come together since August 2001 to develop a new industry standard addressing customer’s enterprise needs • SCSI Trade Association has assumed ownership of SAS marketing. • Specification transferred to T10 in May 2002 • Specification sent to letter ballot on Nov. 22, 2002. ¾ Letter ballot closed and approved with comments on Dec. 22, 2002 ¾ Editorial Board working to resolve comments.

3 SerialSerial AttachedAttached SCSISCSI TopologyTopology

Serial Attached SCSI or SATA disk drives Expanders (Targets) Hosts (Initiators)

4 TechnologyTechnology ComparisonComparison

Serial ATA •Serial Attached Fibre Channel AL •SCSI

Half-duplex •Full-duplex with Full Duplex •Link Aggregation •(Wide ports) Performance •1.5 Gb/sec 3.0 Gb/sec(at intro.) 2.0 Gb/sec •(3.0 Gb/s announced) (6.0 Gb/s planned) (4Gb/s announced)

•1m internal cable >6m external cable >15m external cable

Connectivity •One device >128 devices 127 devices •(fan-out devices Expanders Loop or loop switch •demonstrated) (16k Phys. max) SATA only SAS and SATA Fibre Channel only

•Single-port HDDs Dual-port HDDs Dual-port HDDs

Availability Single-host Multi-initiator Multi-initiator Point to point Point to point Shared media or point to point •Driver •Software transparent •Software transparent •Software transparent •Model •with Parallel ATA •with SCSI •with SCSI

5 PHYsPHYs andand LinksLinks

Phy Phy Physical link transmitter receiver -Signal Differential signal pair +Signal Links are full duplex differential pair connections receiver transmitter -Signal Differential signal pair +Signal

Narrow link Physical link

Port phy phy Port

Wide link Physical link One or more links can be phy phy

configured to be a port Wide port Wide port

Physical link phy phy

Each horizontal line represents a differential signal pair 6 DifferentDifferent Port,Port, PhyPhy,, andand LinkLink ConfigurationsConfigurations

• Devices can have one or more Device Device ports, each configured as HDD Port phy phy Port narrow or wide ports • Initiator to expander links Device Exp. may be composed of multiple phy phy links Wide port Wide port • Aggregates bandwidth phy phy • One initiator address shared

Exp. Device by all the links

phy phy •Command sent down one link; data may be returned on Wide port Wide port Host another (in a separate

phy phy connection)

Each horizontal line represents a differential signal pair

7 MultipleMultiple PathwaysPathways AllowAllow ConcurrentConcurrent OperationOperation 2 Link wide port 2 Link wide port

Initiator device Expander device Target device Expander port Expander A A port Phy Target Initiator Phy Phy Phy port port Subsystem

Phy Phy Phy C Phy B Target device Target Expander Phy HDD Host Phy port C Phy Phy port B

Target device Target Phy Phy Expander D port HDD port Phy Phy D

Initiator Expander port port Target device E E Phy Phy Phy Phy Target Expander port port Subsystem

F Phy Phy Phy Phy F

SAS port X X Connection 2 Link wide port 4 Link wide port Single physical link 8 MaximumMaximum ExpanderExpander DeviceDevice TopologyTopology

Up to 128 Phys per Edge Expander (subtractive routing)

Fanout Expander Up to 128 Phys (routing tables) Expanders per Fanout Expander

9 ExpanderExpander DeviceDevice TopologyTopology

Up to 128 Phys per Edge Expander

Up to 128 Phys per Fanout Expander

10 TwoTwo EdgeEdge ExpandersExpanders

Up to 128 Phys per Edge Expander

A maximum of two Edge Expanders can be connected without the use of a Fanout Expander

11 EdgeEdge ExpanderExpander SetsSets

Initial implementations Up to 128 Phys Per Edge Expander Set are expected to support 40-80 devices per “Edge Expander Set”

Edge Expander Set Edge Expander Device

To facilitate cost effective Edge Edge implementations, Edge Expander Expander Device Device Expanders can be comprised of multiple devices in an ‘Edge Expander Set”

12 SASSAS HostHost AdapterAdapter ProtocolProtocol LayersLayers

SCSI application ATA application Management application layer layer layer Commands

Firmware One transport Transfer of SSP transport layer STP transport layer SMP transport layer layer per Commands (ST) (TT) (MT) protocol

Connection SAS port layer (PL) to a link

SSP link layer STP link layer SMP link layer Establishing (SSP) (STP) (SMP) Hardware and maintaining connections, Digital Logic SAS Port Control (PC) SAS Port Control (PC) transferring SASSAS link layerPort Control(SL) (PC) frames on a link

SAS phy layer (SP) Encoding, bit High Speed Phy ordering Analog

SAS port One or more phys per SAS port Physical: SAS device One or more SAS ports per SAS device Signaling & connectors13 TypicalTypical SASSAS ConfigurationConfiguration

Internal (PCI-X’s, PCI Express, HyperTransport, etc.) or External (FC, iSCSI, Infiniband, SAS, etc.)

SAS RAID SAS RAID Controller Controller

Wide Ports (typ. 4 wide)

SAS Expanders SAS Expanders Edge expanders typ. enclosed in JBOD SAS Fanout expander for large configurations Drive SAS Drive

SATA Drive

SATA 2 Port to single port adapter Drive 14 Phys,Phys, Links,Links, Protocols,Protocols, ExpandersExpanders

•Phy ¾ Cables, Connectors, and ¾ Link rates ¾ Out of Band signaling ¾ Encoding •Link ¾ Primitives ¾ Scrambling ¾ Wide and narrow links ¾ Connections ¾ Rate matching • Protocols and Expanders ¾ SMP (Expander Management) ¾ SSP (SAS) ¾ STP (SATA)

15 SATASATA ConnectorsConnectors && PlacementsPlacements

Appearance of Serial ATA Connectors Device Connector Sizes and Locations (Drawing courtesy of Molex)

Device side connector Serial 2.5” power data Serial ATA Interface Connector Serial Serial ATA 3.5” Power Connector power data Legacy Power (vendor specific)

Parallel 3.5”

Host side connector parallel ATA data 4-pin power

Note: SATA defines the exact connector location on a drive, AND, the connector location is identical between 2.5” & 3.5” HDDs 16 SASSAS BackplaneBackplane ConnectorConnector acceptsaccepts bothboth SATASATA andand SASSAS drivesdrives

(Drawing courtesy of Molex) SAS Port 1

SAS Connector Power SAS Port 2 (on backside)

Power

SATA/ Power SAS Port 2 SAS Port 1 SATA Port

Note: SATA backplane connectors Will NOT accept SAS drives

17 PhysicalPhysical -- SASSAS externalexternal environmentenvironment

External cabled SAS external cable plug environment connectors (4 physical links)

SAS device SAS device

SAS external cable SAS external receptacle (1 to 4 physical links) SAS external receptacle connector (4 physical links) connector (4 physical links)

(SAS external cable connects Tx to Rx on each physical link)

18 PhysicalPhysical -- LinkLink ratesrates

• SAS will be introduced at 3Gbit/sec ¾ Devices will also support 1.5Gbit/sec • SATA rates supported ¾ 1.5 Gbit/sec ¾ 3.0 Gbit/sec (targeted) • Auto-detect link rate during ALIGNs after COMWAKE

19 OutOut--ofof--bandband (OOB)(OOB) signalingsignaling

• SATA out of band (OOB) special patterns (“signals”) ¾ Signals are sent after power-up to initialize the link ¾ Signal is a burst of ALIGN primitives, then idle time; repeated 6 times ¾ Detected by squelch detector and frequency comparators • SATA’s COMRESET, COMINIT, and COMWAKE signals are unchanged • COMSAS signal added ¾ Inserted after calibration sequence before COMWAKE ¾ Initiator: SAS (SSP and/or STP) ¾ Target: SAS protocol (SSP) ¾ Expander: always asserted (not present on SATA initiator-target)) • If both sides assert COMSAS, then the link is a SAS link rather than a SATA link

20 OutOut ofof BandBand SignalingSignaling (OOB)(OOB)

• Out of band signals are slow speed bursts used to initialize a link

SATA phy does not respond to COMSAS COM SAS phy A starts SAS OOB sequence WAKE COMINIT COMSAS COMINIT COMSAS

Phy A Tx/ X Phy A Tx/ Phy B Rx Phy B Rx A Step 1 Step 3 Step 4 COMSAS Time=0 Detect Time=z Time=0 Time=z Step 2 Timeout If drive does NOT Phy A Rx/ Phy A Rx/ X B Phy B Tx generate a Phy B Tx If drive COMSAS burst COMINIT COMINIT COMSAS generates SATA phy responds to COMSAS with COMINIT COMBSATA Drive SAS phy B start SAS OOB sequence COMINIT COMSAS COMSAS COMINIT COMSAS WAKE BSAS Drive Phy A Tx/ Phy A Tx/ A X Phy B Rx Phy B Rx Step 1 Step 3 Step 5 COMSAS Time=0 Time=z Detect Timeout Phy A Rx/ Time=0 Time=z B Step 2 Step 4 Phy B Tx

Phy A Rx/ X COMINIT COMSAS Phy B Tx A : SAS A Power-on COMINIT COMINIT B : SAS B Power-on X : Power-on Time 0: SAS phy reset sequence begins Time 0 : SAS phy reset sequence begins Time z : Post-COMRESET/COMINIT SAS- SATA reset sequence begins Presence or absence of COMSAS OOB signal easily

identifies a HDD as SATA or SAS 21 PhyPhy -- EncodingEncoding

• 8b10b coding ¾ As used in SATA, Fibre Channel, et al. ª Provides embedded clocking and DC balance ª Once a link is established, characters are continuously transmitted ¾ Character = 10 bits as transmitted on the wire ¾ Control characters Kxx.y - special uses ¾ Data characters Dxx.y - represent 8 bit data bytes ¾ Running disparity • Dword = 4 characters ¾ Everything in SAS is based on dwords • Primitive = dword starting with a control character

22 LinkLink -- PrimitivesPrimitives

• Primitive is a dword starting with a control character • Primitives have no endianness; just first, second, third, and last bytes • ALIGN starts with K28.5 • All other SATA primitives start with K28.3 • All SAS primitives start with K28.5 • Primitives may start/end with any disparity

23 LinkLink -- ScramblingScrambling

• Scrambling tries to randomize data ¾ XOR data with the contents of a linear feedback shift register at both sender and receiver ¾ changes constant 000000… and 111111… patterns into pseudo- random patterns of 1s and 0s ¾ Constant patterns occur more often than other patterns, including the worst case pattern that undoes the scrambling effect • Reduces EMI peaks ¾ Spread spectrum clocking addresses EMI for all patterns

24 LinkLink -- ConnectionsConnections

• All I_T communication occurs within an SSP, SMP or STP

connection Expander • Establishing connection through an expander involves arbitration Targets • OPEN address frame to make (SAS or connection request Initiator SATA) • Open timeout timer • OPEN_ACCEPT means connection is active

25 LinkLink -- OpeningOpening aa connectionconnection

• Responses to open request ¾ Arbitration in progress - AIP Initiator device Expander device Target device Expander port Expander A ª reset open timeout timer and A port Phy Target Initiator Phy Phy Phy port keep waiting port Phy Phy Phy C Phy ¾ Cross on wire - OPEN address B Target device frame Target Expander Phy Phy port ª Arbitration fairness dictates who C Phy Phy port B Target device Target Phy Phy Expander wins D port port Phy Phy D ¾ Accepted - OPEN_ACCEPT Initiator Expander port port Target device ¾ Rejected - OPEN_REJECT E E Phy Phy Phy Phy Target Expander port ª Numerous reasons - Retry, bad port

F Phy Phy Phy Phy protocol, deadlock avoidance, F etc. SAS port X X Connection

¾ Cancel - BREAK Single physical link ¾ No response - timeout and send BREAK

26 ArbitrationArbitration FairnessFairness

• Least recently used arbitration fairness ¾ Arbitration wait timer (in OPEN) ªSource maintained ªExpander increments and forwards as part of OPEN ªReset rules and crossing requests • Deadlock Prevention ¾ Partial Pathway timer ¾ Pathway Recovery ªPriority based on Pathway Blocked Count ªEqual counts resolved using SAS address • Scales well to large configurations

27 LinkLink -- RateRate matchingmatching

• When initiator port and target port are separated by an expander, their link rates may differ ¾ E.g. Initiator to expander 3 Gbit/sec; expander to target 1.5 Gbit/sec • Solution: insert ALIGNs on the faster links ¾ E.g. every other dword is used

1,5 Gbps negotiated Time Phy physical link rate dword 0 dword 1 dword 2

Expander added added added device

ALIGN(0) dword 0 ALIGN(1) dword 1 ALIGN(2) dword 2 3,0 Gbps Phy negotiated physical link rate

28 SMPSMP

• Used to manage Expanders • Expanders are required to support SMP • Target do not support SMP • Functions • REPORT (GENERAL, SATA CAPABILITIES, MANUFACTURER INFORMATION, PHY ERROR LOG, PHY SATA, ROUTE INFORMATION), PHY CONTROL, CONFIGURE ROUTE INFORMATION, DISCOVER • OPEN address frame to make connection request • 1KB (1024) maximum frame size

Time SMP SMP SOF frame frame ... CRC EOF dword 0 dword 1

29 SSPSSP (SCSI)(SCSI)

• Full duplex Expander • OPEN address frame to make connection request • SOF, frame dwords, CRC, EOF SAS • Each frame acknowledged with ACK, NAK Target • Credit with RRDY Initiator • SSP link layer state machine • 1KB (1024) max frame length • Error recovery defined for COMMAND TASK,RESPONSE frames • Errors in DATA frames results in termination of the command

Time SSP SSP last SSP SOF frame frame ... frame CRC EOF dword 0 dword 1 dword

30 SAS Expander Operation (with SAS drives)

HBA SATA SAS Mgmt.IF

STP SSP SMP Connection remains fixed until terminated SAS Link by a CLOSE 5 Exp. OPEN ACCEPT – returns C through the path created by SSP Frames R C the OPEN to the host indicating that the two 2 endpoints have agreed upon H H the connection. OPEN C OPEN C R ACCEPT R C Exp. C

1 C R SAS Frames C

OPEN – creates a path to the target using the WWN of the source and destination (SAS end point). Small routing tables in expanders know where WWNs live SAS frames pass through the return path created by the OPEN SAS 4 SAS frames pass through the path Drive created by the OPEN 3

NOTE: SAS drive handle SSP and handle the OPEN process themselves directly using their own WWN 31 STPSTP (ATA)(ATA)

• STP (ATA) connection Expander ¾ STP from initiator to last expander ¾ SATA from the expander to the SATA device

¾ After an STP connection is opened, follow SATA SATA rules Target

¾ Frame sent as: SATA_SOF, SATA frame, Initiator SATA_EOF ¾ Each frame receives SATA_R_OK or SATA_R_ERR ¾ SATA_X_RDY/SATA_R_RDY for permission to send another frame ¾ Max SATA frame is 8KB (8192) Time SATA SATA SATA SATA_SOF frame frame ... SATA_EOF CRC dword 0 dword 1

32 SASSAS ExpanderExpander OperationOperation (with(with SATASATA drives)drives)

HBA SATA SAS Mgmt.IF

STP SSP SMP Connection remains SAS Link 7 fixed until terminated by a CLOSE

Exp. OPEN ACCEPT – returns C through the path created by Payload FIS R 3 (Regular FIS to be transmitted) C the OPEN to the host indicating that the two endpoints have agreed upon H H the connection. OPEN C OPEN C R ACCEPT R C Exp. C

1 C Payload FIS R (Regular FIS to be transmitted) C

OPEN – creates a path to the target using the WWN of the source and destination (STP end point). Small routing tables in expanders know Standard SATA 1.0 frames pass where WWNs live through the return path (tunnel) SATA 6 created by the OPEN Standard SATA 1.0 frames pass Drive through the path (tunnel) created by 4 2 STP function provides an end the OPEN point for routing and supplies a 5 WWN on behalf of the drive. Also OPENs connection to the host when drive has frames to send Drive sees only NOTE: SAS drive handle SSP and handle the back and no connection is already standard SATA OPEN process themselves directly using OPEN their own WWN 1.0 frames

33 SummarySummary

• Serial Attached SCSI meets the needs of mainstream enterprise class storage systems ¾ Its point to point architecture will provide performance, configuration capability and scalability beyond what is available from today’s device level interconnects ¾ Provides a choice of devices use ª Low cost Serial ATA disks or ª High performance, dual port Serial Attached SCSI drives ª Simplifies purchasing and deployment to meet changing application needs ¾ Provides the flexibility to meet the cost, availability, scalability, and performance requirements for future generations of storage systems and applications www.scsita.org www.maxtor.com

34