Marty Czekalski Vice President, Director – SCSI Trade Association Technical Marketing Manager – Maxtor Corporation
1 SerialSerial AttachedAttached SCSISCSI
• Serial Attached SCSI is the next evolution of SCSI beyond U320 • SAS has been designed to be a device and near cabinet interface only, not a network interface • Leverages an enhanced Serial ATA PHY while adding support for a second port • Serial SCSI utilizes features of Fibre Channel AND compatibility with SATA drives in a point-to-point, switched architecture ¾ Up to 16k physical links per domain in a switched, point-to-point configuration for scalable performance. ¾ Scalability with wide ports and future speed increases ¾ Data frame based on FCP ¾ Includes rate matching to optimize device transfer speeds ¾ World wide name addressing ¾ Uses a simplified protocol that will minimize interoperability issues between controllers & drives ¾ Dual port drives for high availability
2 IndustryIndustry AdoptionAdoption TimelineTimeline
Serial Attached SCSI Spec Serial Attached SCSI Serial Attached SCSI Serial Attached SCSI Transferred to Product Test and Product T10 Design Integration Availability
2002 2003 2004
Interoperability Testing & Plugfests Serial Attached SCSI Specification Serial Attached SCSI Letter Ballot v1.0 Specification Complete
• Twenty-six companies have come together since August 2001 to develop a new industry standard addressing customer’s enterprise needs • SCSI Trade Association has assumed ownership of SAS marketing. • Specification transferred to T10 in May 2002 • Specification sent to letter ballot on Nov. 22, 2002. ¾ Letter ballot closed and approved with comments on Dec. 22, 2002 ¾ Editorial Board working to resolve comments.
3 SerialSerial AttachedAttached SCSISCSI TopologyTopology
Serial Attached SCSI or SATA disk drives Expanders (Targets) Hosts (Initiators)
4 TechnologyTechnology ComparisonComparison
Serial ATA •Serial Attached Fibre Channel AL •SCSI
Half-duplex •Full-duplex with Full Duplex •Link Aggregation •(Wide ports) Performance •1.5 Gb/sec 3.0 Gb/sec(at intro.) 2.0 Gb/sec •(3.0 Gb/s announced) (6.0 Gb/s planned) (4Gb/s announced)
•1m internal cable >6m external cable >15m external cable
Connectivity •One device >128 devices 127 devices •(fan-out devices Expanders Loop or loop switch •demonstrated) (16k Phys. max) SATA only SAS and SATA Fibre Channel only
•Single-port HDDs Dual-port HDDs Dual-port HDDs
Availability Single-host Multi-initiator Multi-initiator Point to point Point to point Shared media or point to point •Driver •Software transparent •Software transparent •Software transparent •Model •with Parallel ATA •with SCSI •with SCSI
5 PHYsPHYs andand LinksLinks
Phy Phy Physical link transmitter receiver -Signal Differential signal pair +Signal Links are full duplex differential pair connections receiver transmitter -Signal Differential signal pair +Signal
Narrow link Physical link
Port phy phy Port
Wide link Physical link One or more links can be phy phy
configured to be a port Wide port Wide port
Physical link phy phy
Each horizontal line represents a differential signal pair 6 DifferentDifferent Port,Port, PhyPhy,, andand LinkLink ConfigurationsConfigurations
• Devices can have one or more Device Device ports, each configured as HDD Port phy phy Port narrow or wide ports • Initiator to expander links Device Exp. may be composed of multiple phy phy links Wide port Wide port • Aggregates bandwidth phy phy • One initiator address shared
Exp. Device by all the links
phy phy •Command sent down one link; data may be returned on Wide port Wide port Host another (in a separate
phy phy connection)
Each horizontal line represents a differential signal pair
7 MultipleMultiple PathwaysPathways AllowAllow ConcurrentConcurrent OperationOperation 2 Link wide port 2 Link wide port
Initiator device Expander device Target device Expander port Expander A A port Phy Target Initiator Phy Phy Phy port port Subsystem
Phy Phy Phy C Phy B Target device Target Expander Phy HDD Host Phy port C Phy Phy port B
Target device Target Phy Phy Expander D port HDD port Phy Phy D
Initiator Expander port port Target device E E Phy Phy Phy Phy Target Expander port port Subsystem
F Phy Phy Phy Phy F
SAS port X X Connection 2 Link wide port 4 Link wide port Single physical link 8 MaximumMaximum ExpanderExpander DeviceDevice TopologyTopology
Up to 128 Phys per Edge Expander (subtractive routing)
Fanout Expander Up to 128 Phys (routing tables) Expanders per Fanout Expander
9 ExpanderExpander DeviceDevice TopologyTopology
Up to 128 Phys per Edge Expander
Up to 128 Phys per Fanout Expander
10 TwoTwo EdgeEdge ExpandersExpanders
Up to 128 Phys per Edge Expander
A maximum of two Edge Expanders can be connected without the use of a Fanout Expander
11 EdgeEdge ExpanderExpander SetsSets
Initial implementations Up to 128 Phys Per Edge Expander Set are expected to support 40-80 devices per “Edge Expander Set”
Edge Expander Set Edge Expander Device
To facilitate cost effective Edge Edge implementations, Edge Expander Expander Device Device Expanders can be comprised of multiple devices in an ‘Edge Expander Set”
12 SASSAS HostHost AdapterAdapter ProtocolProtocol LayersLayers
SCSI application ATA application Management application layer layer layer Commands
Firmware One transport Transfer of SSP transport layer STP transport layer SMP transport layer layer per Commands (ST) (TT) (MT) protocol
Connection SAS port layer (PL) to a link
SSP link layer STP link layer SMP link layer Establishing (SSP) (STP) (SMP) Hardware and maintaining connections, Digital Logic SAS Port Control (PC) SAS Port Control (PC) transferring SASSAS link layerPort Control(SL) (PC) frames on a link
SAS phy layer (SP) Encoding, bit High Speed Phy ordering Analog
SAS port One or more phys per SAS port Physical: SAS device One or more SAS ports per SAS device Signaling & connectors13 TypicalTypical SASSAS ConfigurationConfiguration
Internal (PCI-X’s, PCI Express, HyperTransport, etc.) or External (FC, iSCSI, Infiniband, SAS, etc.)
SAS RAID SAS RAID Controller Controller
Wide Ports (typ. 4 wide)
SAS Expanders SAS Expanders Edge expanders typ. enclosed in JBOD SAS Fanout expander for large configurations Drive SAS Drive
SATA Drive
SATA 2 Port to single port adapter Drive 14 Phys,Phys, Links,Links, Protocols,Protocols, ExpandersExpanders
•Phy ¾ Cables, Connectors, and Backplanes ¾ Link rates ¾ Out of Band signaling ¾ Encoding •Link ¾ Primitives ¾ Scrambling ¾ Wide and narrow links ¾ Connections ¾ Rate matching • Protocols and Expanders ¾ SMP (Expander Management) ¾ SSP (SAS) ¾ STP (SATA)
15 SATASATA ConnectorsConnectors && PlacementsPlacements
Appearance of Serial ATA Connectors Device Connector Sizes and Locations (Drawing courtesy of Molex)
Device side connector Serial 2.5” power data Serial ATA Interface Connector Serial Serial ATA 3.5” Power Connector power data Legacy Power (vendor specific)
Parallel 3.5”
Host side connector parallel ATA data 4-pin power
Note: SATA defines the exact connector location on a drive, AND, the connector location is identical between 2.5” & 3.5” HDDs 16 SASSAS BackplaneBackplane ConnectorConnector acceptsaccepts bothboth SATASATA andand SASSAS drivesdrives
(Drawing courtesy of Molex) SAS Port 1
SAS Backplane Connector Power SAS Port 2 (on backside)
Power
SATA/ Power SAS Port 2 SAS Port 1 SATA Port
Note: SATA backplane connectors Will NOT accept SAS drives
17 PhysicalPhysical -- SASSAS externalexternal environmentenvironment
External cabled SAS external cable plug environment connectors (4 physical links)
SAS device SAS device
SAS external cable SAS external receptacle (1 to 4 physical links) SAS external receptacle connector (4 physical links) connector (4 physical links)
(SAS external cable connects Tx to Rx on each physical link)
18 PhysicalPhysical -- LinkLink ratesrates
• SAS will be introduced at 3Gbit/sec ¾ Devices will also support 1.5Gbit/sec • SATA rates supported ¾ 1.5 Gbit/sec ¾ 3.0 Gbit/sec (targeted) • Auto-detect link rate during ALIGNs after COMWAKE
19 OutOut--ofof--bandband (OOB)(OOB) signalingsignaling
• SATA out of band (OOB) special patterns (“signals”) ¾ Signals are sent after power-up to initialize the link ¾ Signal is a burst of ALIGN primitives, then idle time; repeated 6 times ¾ Detected by squelch detector and frequency comparators • SATA’s COMRESET, COMINIT, and COMWAKE signals are unchanged • COMSAS signal added ¾ Inserted after calibration sequence before COMWAKE ¾ Initiator: SAS (SSP and/or STP) ¾ Target: SAS protocol (SSP) ¾ Expander: always asserted (not present on SATA initiator-target)) • If both sides assert COMSAS, then the link is a SAS link rather than a SATA link
20 OutOut ofof BandBand SignalingSignaling (OOB)(OOB)
• Out of band signals are slow speed bursts used to initialize a link
SATA phy does not respond to COMSAS COM SAS phy A starts SAS OOB sequence WAKE COMINIT COMSAS COMINIT COMSAS
Phy A Tx/ X Phy A Tx/ Phy B Rx Phy B Rx A Step 1 Step 3 Step 4 COMSAS Time=0 Detect Time=z Time=0 Time=z Step 2 Timeout If drive does NOT Phy A Rx/ Phy A Rx/ X B Phy B Tx generate a Phy B Tx If drive COMSAS burst COMINIT COMINIT COMSAS generates SATA phy responds to COMSAS with COMINIT COMBSATA Drive SAS phy B start SAS OOB sequence COMINIT COMSAS COMSAS COMINIT COMSAS WAKE BSAS Drive Phy A Tx/ Phy A Tx/ A X Phy B Rx Phy B Rx Step 1 Step 3 Step 5 COMSAS Time=0 Time=z Detect Timeout Phy A Rx/ Time=0 Time=z B Step 2 Step 4 Phy B Tx
Phy A Rx/ X COMINIT COMSAS Phy B Tx A : SAS A Power-on COMINIT COMINIT B : SAS B Power-on X : Power-on Time 0: SAS phy reset sequence begins Time 0 : SAS phy reset sequence begins Time z : Post-COMRESET/COMINIT SAS- SATA reset sequence begins Presence or absence of COMSAS OOB signal easily
identifies a HDD as SATA or SAS 21 PhyPhy -- EncodingEncoding
• 8b10b coding ¾ As used in SATA, Fibre Channel, et al. ª Provides embedded clocking and DC balance ª Once a link is established, characters are continuously transmitted ¾ Character = 10 bits as transmitted on the wire ¾ Control characters Kxx.y - special uses ¾ Data characters Dxx.y - represent 8 bit data bytes ¾ Running disparity • Dword = 4 characters ¾ Everything in SAS is based on dwords • Primitive = dword starting with a control character
22 LinkLink -- PrimitivesPrimitives
• Primitive is a dword starting with a control character • Primitives have no endianness; just first, second, third, and last bytes • ALIGN starts with K28.5 • All other SATA primitives start with K28.3 • All SAS primitives start with K28.5 • Primitives may start/end with any disparity
23 LinkLink -- ScramblingScrambling
• Scrambling tries to randomize data ¾ XOR data with the contents of a linear feedback shift register at both sender and receiver ¾ changes constant 000000… and 111111… patterns into pseudo- random patterns of 1s and 0s ¾ Constant patterns occur more often than other patterns, including the worst case pattern that undoes the scrambling effect • Reduces EMI peaks ¾ Spread spectrum clocking addresses EMI for all patterns
24 LinkLink -- ConnectionsConnections
• All I_T communication occurs within an SSP, SMP or STP
connection Expander • Establishing connection through an expander involves arbitration Targets • OPEN address frame to make (SAS or connection request Initiator SATA) • Open timeout timer • OPEN_ACCEPT means connection is active
25 LinkLink -- OpeningOpening aa connectionconnection
• Responses to open request ¾ Arbitration in progress - AIP Initiator device Expander device Target device Expander port Expander A ª reset open timeout timer and A port Phy Target Initiator Phy Phy Phy port keep waiting port Phy Phy Phy C Phy ¾ Cross on wire - OPEN address B Target device frame Target Expander Phy Phy port ª Arbitration fairness dictates who C Phy Phy port B Target device Target Phy Phy Expander wins D port port Phy Phy D ¾ Accepted - OPEN_ACCEPT Initiator Expander port port Target device ¾ Rejected - OPEN_REJECT E E Phy Phy Phy Phy Target Expander port ª Numerous reasons - Retry, bad port
F Phy Phy Phy Phy protocol, deadlock avoidance, F etc. SAS port X X Connection
¾ Cancel - BREAK Single physical link ¾ No response - timeout and send BREAK
26 ArbitrationArbitration FairnessFairness
• Least recently used arbitration fairness ¾ Arbitration wait timer (in OPEN) ªSource maintained ªExpander increments and forwards as part of OPEN ªReset rules and crossing requests • Deadlock Prevention ¾ Partial Pathway timer ¾ Pathway Recovery ªPriority based on Pathway Blocked Count ªEqual counts resolved using SAS address • Scales well to large configurations
27 LinkLink -- RateRate matchingmatching
• When initiator port and target port are separated by an expander, their link rates may differ ¾ E.g. Initiator to expander 3 Gbit/sec; expander to target 1.5 Gbit/sec • Solution: insert ALIGNs on the faster links ¾ E.g. every other dword is used
1,5 Gbps negotiated Time Phy physical link rate dword 0 dword 1 dword 2
Expander added added added device
ALIGN(0) dword 0 ALIGN(1) dword 1 ALIGN(2) dword 2 3,0 Gbps Phy negotiated physical link rate
28 SMPSMP
• Used to manage Expanders • Expanders are required to support SMP • Target do not support SMP • Functions • REPORT (GENERAL, SATA CAPABILITIES, MANUFACTURER INFORMATION, PHY ERROR LOG, PHY SATA, ROUTE INFORMATION), PHY CONTROL, CONFIGURE ROUTE INFORMATION, DISCOVER • OPEN address frame to make connection request • 1KB (1024) maximum frame size
Time SMP SMP SOF frame frame ... CRC EOF dword 0 dword 1
29 SSPSSP (SCSI)(SCSI)
• Full duplex Expander • OPEN address frame to make connection request • SOF, frame dwords, CRC, EOF SAS • Each frame acknowledged with ACK, NAK Target • Credit with RRDY Initiator • SSP link layer state machine • 1KB (1024) max frame length • Error recovery defined for COMMAND TASK,RESPONSE frames • Errors in DATA frames results in termination of the command
Time SSP SSP last SSP SOF frame frame ... frame CRC EOF dword 0 dword 1 dword
30 SAS Expander Operation (with SAS drives)
HBA SATA SAS Mgmt.IF
STP SSP SMP Connection remains fixed until terminated SAS Link by a CLOSE 5 Exp. OPEN ACCEPT – returns C through the path created by SSP Frames R C the OPEN to the host indicating that the two 2 endpoints have agreed upon H H the connection. OPEN C OPEN C R ACCEPT R C Exp. C
1 C R SAS Frames C
OPEN – creates a path to the target using the WWN of the source and destination (SAS end point). Small routing tables in expanders know where WWNs live SAS frames pass through the return path created by the OPEN SAS 4 SAS frames pass through the path Drive created by the OPEN 3
NOTE: SAS drive handle SSP and handle the OPEN process themselves directly using their own WWN 31 STPSTP (ATA)(ATA)
• STP (ATA) connection Expander ¾ STP from initiator to last expander ¾ SATA from the expander to the SATA device
¾ After an STP connection is opened, follow SATA SATA rules Target
¾ Frame sent as: SATA_SOF, SATA frame, Initiator SATA_EOF ¾ Each frame receives SATA_R_OK or SATA_R_ERR ¾ SATA_X_RDY/SATA_R_RDY for permission to send another frame ¾ Max SATA frame is 8KB (8192) Time SATA SATA SATA SATA_SOF frame frame ... SATA_EOF CRC dword 0 dword 1
32 SASSAS ExpanderExpander OperationOperation (with(with SATASATA drives)drives)
HBA SATA SAS Mgmt.IF
STP SSP SMP Connection remains SAS Link 7 fixed until terminated by a CLOSE
Exp. OPEN ACCEPT – returns C through the path created by Payload FIS R 3 (Regular FIS to be transmitted) C the OPEN to the host indicating that the two endpoints have agreed upon H H the connection. OPEN C OPEN C R ACCEPT R C Exp. C
1 C Payload FIS R (Regular FIS to be transmitted) C
OPEN – creates a path to the target using the WWN of the source and destination (STP end point). Small routing tables in expanders know Standard SATA 1.0 frames pass where WWNs live through the return path (tunnel) SATA 6 created by the OPEN Standard SATA 1.0 frames pass Drive through the path (tunnel) created by 4 2 STP function provides an end the OPEN point for routing and supplies a 5 WWN on behalf of the drive. Also OPENs connection to the host when drive has frames to send Drive sees only NOTE: SAS drive handle SSP and handle the back and no connection is already standard SATA OPEN process themselves directly using OPEN their own WWN 1.0 frames
33 SummarySummary
• Serial Attached SCSI meets the needs of mainstream enterprise class storage systems ¾ Its point to point architecture will provide performance, configuration capability and scalability beyond what is available from today’s device level interconnects ¾ Provides a choice of devices use ª Low cost Serial ATA disks or ª High performance, dual port Serial Attached SCSI drives ª Simplifies purchasing and deployment to meet changing application needs ¾ Provides the flexibility to meet the cost, availability, scalability, and performance requirements for future generations of storage systems and applications www.scsita.org www.maxtor.com
34