Performance Analysis of iSCSI Over TCP versus SCTP THESIS Submitted to the Faculty of Graduare Srudies In Partial FulfrILment of the Requirements for the Degree of MASTER OF SC/ENCE Department of Electrical and Computer Engineering, University of Manitoba Dong Zhang December 2001 THE UNTVERSITY OF MANITOBA FACULTY OF G*RADUATE STUDIES COPYRIGHT PERMISSION Performance Analysis of iSCSI Over TCP versus SCTP BY Dong Zhang A ThesislPracticum submitted to the Faculty of Graduate Studies of The University of Manitoba Ín partial fulfillment of the requirement of the degree MASTER OF SCIENCE Doug Zhang@ 2007 Permission has been granted to the University of Manitoba Libraries to lend â copy of this thesis/practicum, to Library and Archives Canada (LAC) to lend a copy of this thesis/practicum, and to LAC's agent (UMVProQuest) to microfilm, sell copies and to publish an abstract of this thesis/practicum. This reproduction or copy of this thesis has been made available by authority of the copyright owner solely for the purpose of private study and research, and may only be reproduced and copied as permitted by copyright laws or with express written authorization from the copyright owner. Abstract IP SANs (storage area networks) have become increasingly common in today's storage industry. Internet SCSI (iSCSI) [21] is a block level protocol defined by the IP Storage work group of IETF (in RFC3720). iSCSI encapsulates SCSI commands and Data for transfer over an IP network and thereby takes advantage of the investment in existing IP networks to facilitate and extend SANs. iSCSl-based IP SANs allow us ro reduce initial cosrs and the management costs of storage. Conceptually, iSCSI is designed to run over TCP/IP that provides Layer 3/4 network transport. SCTP (Stream Control Transmission Protocol) [23] is a relatively new IETF stan- dard transport protocol. SCTP was originally developed to transport Public Switched Tele- phone Network (PSTN) signaling messages over IP networks, but it has emerged as a general- purpose transport protocol. SCTP is a reliable message oriented protocol that combines certain desirable qualities of TCP and UDP and provides additional services unavailable from TCP (or UDP) as well. Attractive features of SCTP such as multistreaming and multihoming make it an alternative to TCP for many applications. In this thesis, we focus primarily on a performance study comparing SCTP and TCP as the underlying transport layer protocol for iSCSI using ns2 simulation. The results of simulation and performance analysis indicate that iSCSI over SCTP has advantages over TCP in the scenarios investigated. Acknowledgments First of all, I would like to thank my adviso¡ Dr. Robert D. Mcleod for his guidance and support throughout the entirety of my work on this subject. Our discussions helped me overcome many obstacles that came up. I would also like to thank Sajid who helped me a lot. I feel lucky for having the chance to work with him. Many thanks go also to committee members: Dr. Peter Graham and Dr. Jun Cai, for their valuable comments and suggestions. Finally, I want to thank my lovely wife, Yu, my sister, Cong, and my parent for their support during all these years. Also my daughter, Kyleen, who is and will always be a source of inspiration for me. Contents Glossary I Introduction r0 iSCSI 13 2.1 Protocol Model l3 2.2 Session Management l3 2.2.1 Login 15 2.2.2 Full Feature Phase 15 2.2.3 Shutdown 17 2.3 Sequencing l7 2.3.1 CommandNumbering t7 2.3.2 Status Numbering l8 2.3.3 Data Numbering l8 2.4 Name and Address l9 2.5 PDU Format . 19 2.5.1 Basic Header Segment (BHS) 20 2.6 Message Framing 20 2.6.1 Fixed Interval Markers 2l 2.6.2 Marker PDU Aligned Framing 22 2.7 Error Handling and Recovery 22 2.8 Summary 23 Tiansport Protocols 24 3.1 The Transport Layer in the OSI Model 24 3.2 User Datagram Protocol (UDP) 26 3.3 Transmission Control Protocol (TCP) 27 3.3.1 Header Format 27 3.3.2 Flow Control and Congestion Control 29 3.4 Stream Control Transmission Protocol (SCTP) 3t 3.4.1 Packets 3l 3.4.2 Association Establishment 32 3.4.3 Multihoming... 35 1.4.4 Multistreaming 36 3.4.5 Flow and Congestion Control 37 3.5 Comparison between Transport Layer Protocols 38 3.6 Summary 38 Simulation Design and Implementation 40 4.1 Network Simulator (ns2) 40 4.2 iSCSI Implementation 4l 4.2.1 Overview of iSCSI Classes 42 4.2.2 Write Command Simulation 44 4.2.3 Read Command Simulation 46 4.3 Introduction to the SCTP Implementation 46 4.3.1 MultihomingSimulation 47 4.4 iSCSI Over TCP Simulation Component Implemenration 48 4.4.1 Agent V/rapper and Message Boundaries Simulation 48 4.4.2 Tcl Script: iSCSI-MPA-TCP . 49 4.5 iSCSI Over SCTP Simulation Component Implementation 50 4.6 Summary 52 5 Performance Analysis 53 5.1 Performance Metrics and Variables 53 5.2 Command Completion Time (no loss) 55 5.3 Throughput in a Lossy Network 56 5.3.1 Head-of-LineBlocking 56 ' 5.3.2 Throughput vs. Loss Rate 56 5.4 Reliability 58 5.5 Security 61 5.6 Summary 62 6 Related Work 7 Conclusions and Future Work 65 7.1 Conclusions 65 1.2 Future Work 66 List of Figures 2.1 iSCSI protocol layering model. l4 2.2 Text Operational Keys r6 2.3 Basic Header Segment (48 Bytes). 20 2.4 List of opcodes 21 3.1 OSI reference model. 25 3.2 UDP datagram. 26 3.3 TCP three-way handshake. 2l 3.4 TCP segment. 28 3.5 SCTP packet format. 32 3.6 SCTP chunk types and description. JJ 3.7 SCTP association establishment. 34 3.8 SCTP's multihoming feature 35 3.9 SCTP's multistreaming feature 36 4.1 Application Class 42 4.2 The class hierarchy of the iSCSI simulation classes. 43 4.3 iSCSI'Write (a) and Read (b) command execurion. 45 4.4 Simulation of iSCSVMPA/TCP in ns2. 49 4.5 A simulation script of iSCSVMPA/TCP. 50 4.6 Parameters of the SCTP-aware application data. 5l 4.7 Simulation of iSCSI over SCTP (singlehomed) in ns2. 51 4.8 Simulation of iSCSI over SCTP (multihomed) in ns2. 5l 5.1 Comparison of UDP, TCP, and SCTP. 53 5.2 The command completion time for various command sizes under no loss net- work 55 5.3 TCP's Head-Of-Line blocking problem and SCTP's multisrreaming feature 57 5.4 Throughput performance in loss network 57 5.5 Default value of parameters for failover system. 58 5.6 iSCSI over SCTP with Multihoming (a) 59 5.7 iSCSI over SCTP with Multihoming (b) 60 5.8 iSCSI over SCTP with Multihoming (c) 60 5.9 iSCSI over SCTP with Multihoming (d) 61 5.r0 Protection against DoS attacks by four way handshake 62 6 Glossary The list of acronyms and other related terms are described here. ACK Acknowledgement ADU Application Data Unit AHS Additional Header Segment BHS Basic Header Segment CmdSN Command Sequence Number CRC Cyclic Redundancy Check DataSN Data Sequence Number DDP Direct Data Placement EDTL Expected Data Transfer Length EMSS Effective Maximum Segment Size ExpCmdSN Expected Command Sequence Number ExpDataSN Expected Data Sequence Number ExpStatSN Expected Status Sequence Number FC Fibre Channel FCIP Fibre Channel over Internet Protocol FFP full-Feature Phase FFPO Full-Feature Phase Only FIM Fix Interval Marker FPDU Framing Protocol Data Unit Gbps Gigabit per second iFCP Internet Fibre Channel Protocol UO Input/Output IO Initialize Only IP Internet Protocol ips IP storage IPsec/IPSec Internet Protocol Security rqn iSCSI qualified name iSAN iSCSI-based Storage Area Network iSER iSCSI Extension for RDMA iSCSÌ SCSI over IP ISID Initiator Session ID I-T Initiator-Target I_TI Initiator-TargetIUN ITN iSCSI Target Node or iSCSI Target Name I1"T Initiator Task Tag LAN Local A¡ea Network LBA Logical Block Address LO leading Only LONP Login Operational Negotiation Phase LU Logical Unit LUN Logical Unit Number MAN Metropolitan Area Network MaxCmdSN Maximum Command Sequence Number MC/S Multiple Connections per Session MD-5 Message Digest version 5 MPA Markers PDU Aligned MSS Maximum Segment Size MTU Maximum Transfer Unit NA Not Applicable NAS Network Attached Storage NFS Network File System NIC Network Interface Card NOP No Operation NSG Next Stage OS Operating System PDU Protocol Data Unit R2T Ready to Transfer R2TSN Ready to Transfer Sequence Number RDMA Remote Direct Memory Access RFC Request for Comments SAM SCSI Architectural Model SAM2 SCSI Architectural Model 2 SAN Storage Area Network SCSI Small Compurer Systems Inrerface SCTP Stream Transmission Control Protocol SID Stream ID SN Sequence Number SNACK sequence Number Acknowledgement or selective Negative Ac- knowledgement SSID Session ID SSN Stream Sequence Numbers StatSN Status Sequence Number TCB Task Control Block TCP Transmission Control Protocol TOE TCP/IP Offload Engineer TSID Target Session ID TSIH Target Session Identifying Handle TSN Transmission Sequence Number TUF TCP Upper-Level-Protocol Framing TTT Target Transfer Tag UDP User Datagram Protocol ULP Upper Level Protocol or Upper Layer Protocol ULPDU Upper Layer Protocol Data Unit URL Uniform Resource Locator URN Uniform Resource Name UTF Universal Transform ati on Format VPN Virtual Private Network WAN Wide Area Network WG Working Group Chapter 1 Introduction A Storage Area Network (SAN) is a network dedicated to transporting data for storage and retrieval. SANs have been proven to increase the performance and availability of corporate storage resources. A SAN can be deployed in two ways: using Fiber Channel or IP.