Performance Analysis and Testing of Storage Area Network Yao-Long Zhu, Shu-Yu Zhu and Hui Xiong Data Storage Institute, Singapore Email: [email protected] http://www.dsi.nubs.edu.sg Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Motivations •What should we do to optimize the storage system directly connected to storage network instead of directly connected to server? •How to compare IP storage, Fibre Channel, and InfinBand? •Do we need new algorithms to replace the RAID technology which introduced in 1980s? •How to evaluate and analyze the performance of the Storage Area Network easily and quickly? •Modeling and simulation is a faster way to study these questions than implementation and testing Block level FC protocol Storage Storage system system FC/DWDM FC/DWDM FC-SAN FC-SAN FCIP FCIP NAS Server Server NAS WAN iSCSI iSCSI LAN LAN Router Router Clients Clients File level IP protocol IP storage system IP storage system Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Key Points Build up a Queuing Network Model for total SAN performance analysis from perspective of networking Analyze the effects of the I/O workload on the SAN performance Analyze the effects of disk cache and Fork/Join model Analyze the FC-AL’s scheduling algorithms Comparison of the theoretical and experimental results Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Modeling of SAN Disk Client Server RAID FC-switch FC-AL Disk IP switch controller Client Server Disk Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Queuing Network Model Disk •Network’s view points Client Server RAID IP switch FC-switch FC-AL Disk •I/O response time controller Client Server Disk •Throughput (MB/s) •IOPs l lhost1 ldisk hda l Disk HDA fc-sw lda lfc-al Controller & center Cache Hosts lhost2 FC-SW Disk Array FC-AL Network Controller and Network Cache Disk Array Disk Unit IO Queue Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 SAN Performance for Big Sequential I/O 180 160 HDA 140 Diskcache FC-AL 120 dacc 100 FCF 80 Host 60 40 Utilizations of service nodes vary with system Response time (ms) 20 request rate for sequential 1Mb I/O size. 0 0 10 20 30 40 50 60 70 80 90 SAN performance (MB/s) 1 Host FCF 0.8 dacc System response time variesFC with system = Bottleneck FC-AL ! throughput for sequential 1MB I/O size. Diskcache 0.6 HDA 0.4 Utilizations I/O workload: read System configuration: 0.2 1 server+ 1 daccDACC + 5 FC-AL disks HDA: Head-Disk Assembly (mechanical part) 0 dacc: Disk array controller and cache 0 20 40 60 80 100 FCF: FC fabric Switch Request rate (IOPs) Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 SAN Performance for Big Random I/O 140 HDA System response time varies with system 120 Diskcache throughput for random 1MB I/O size. FC-AL 100 dacc 80 FCF Host 60 40 Response time (ms) 20 Utilizations of service nodes vary with system request rate for random 1MB I/O size 0 0 10 20 30 40 50 60 SAN performance (MB/s) 1 Host 0.8 FCF dacc HDA response time is the largest portion FC-AL 0.6 Diskcache (40%~60%) in the whole responsePhysical time Drive = BottleneckHDA ! distribution. 0.4 performance of Utilizations 0.2 5 hard disks : 66MB/s 0 15 Hard disks : 79MB/s 0 10 20 30 40 50 60 70 25 Hard disks : 82MB/s request rate(IOPs) Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 SAN Performance for Small Sequential I/O 4.0 1.0 HDA 3.5 Host Diskcache 0.8 FCF 3.0 FC-AL dacc FC-AL 2.5 dacc 0.6 Diskcache 2.0 FCF HDA Host 1.5 0.4 Utilizations 1.0 0.2 response time (ms) 0.5 0.0 0.0 0 2 4 6 8 10Host12 14 Server16 18 = Bottleneck0 1000 !2000 3000 4000 5000 System performance (MB/s) Request rate (IOPs) System response time varies with the throughput for Utilizations vary with system request rate for sequential 4KB I/O (single user) sequential 4KB I/O (single user). (4500 IOPs) 1.0 3.0 Host HDA FCF 2.5 Diskcache 0.8 dacc FC-AL FC-AL 2.0 0.6 Diskcache dacc HDA 1.5 FCF Host 0.4 1.0 Utilizations 0.2 Response time (ms) 0.5 0.0 0.0 0 6 12 18 24 30 36 42 48 Disk Array Controller = 0Bottleneck2000 4000 6000 !8000 10000 12000 14000 SAN performance (MB/s) Request rate (IOPs) System response time varies with the throughput for Utilizations varies with the system request rate for sequential 4KB I/O (2 users). (5600 IOPs) sequential 4KB I/O (2 users). Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 SAN Performance for Small Random I/O 1.2 70 Host HDA 60 1 FCF Diskcache dacc 50 FC-AL 0.8 FC-AL dacc Diskcache HDA 40 FCF 0.6 30 Host Utilizations 0.4 20 response time (ms) 0.2 10 0 0 0 100 200 300 400 500 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 performance(MB/s) Request rate(IOPs) System response time varies with the throughput for Utilizations vary with the system request rate for random 4KB I/O (5 HDDs). random 4KB I/O (5 HDDs). 70 60 HDA Diskcache 50 FC-AL Physical Drive = Bottleneck ! 40 dacc FCF Performance of 30 Host 20 5 hard disks : 420 IOPs Response time (ms) 10 0 15 hard disks : 1260 IOPs 0.0 0.4 0.8 1.2 1.6 2.0 2.4 2.8 3.2 3.6 4.0 4.4 4.8 SAN performance (MB/s) 25 hard disks : 2100 IOPs System response time varies with the throughput for random 4KB I/O (15 HDDs). Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Empirical Model Comparison 100 SAN performance varies with I/O size theoretical for sequential I/O 80 experimental workload 60 40 20 Throughput (MB/s) 0 1 2 4 8 16 32 64 128 256 512 1024 I/O size (KB) The tests are based on the multiple Pentium 733MHz hosts with 64bits and 66MHz PCI bus, HBA with Qlogic QLA 2200A, 1G FC switch with Brocade Silkwarm 2400, and a self-developed virtual FC Disk. IOMeter is used as the benchmark tool. Both of the testing and theoretic results show that the FC network is the system bottleneck for big I/O size (>32KB), and the storage system controller overhead is the system limitation for small I/O size (<16KB). Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Lesson #1: Disk Cache / Controller Performance Controller and 8 Cache center HDA center ldisk l HDA Large I/O : hda 6 DCC Physical Drive overhead dominates Small I/O : Hard Disk model 4 DCC overhead larger than Physical Drive 2 overhead Bandwidths of the disk unit, DCC and HDA vary with IO size for 0 single hard disk. 1 2 4 8 16 32 64 128 I/O size (KB) 45 40 Response time varies with I/O size 35 when the request rate is 200 IOPs 30 for hard disk. 25 20 15 Single Disk – Cache MAY be bottleneck Throughput( MB/s) 10 Disk Array – Cache depends largely on 5 0 Disk Stripe Size 1 2 4 8 16 32 64 128 256 512 1024 IO size (KB) Disk HDA Cache Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Lesson #2: Disk Array and Fork/Join Model ldisk 1.5 1.4 l lda lfc-al disk 1.3 l disk 1.2 Ratio Disk Array FC-AL access time 1.1 Controller and Network ldisk waiting time Cache Disks 1 0 10 20 30 Disk number Fork/Join model Fork/Join access time and queue waiting time ratio vary with disk numbers. Request j for disk 1 Request j out Request j Request j for disk k d * m 2 * r * E[TS 2 ] TR (l) = TS (0) + disk disk disk k k m - l disk disk Response time ratio varies with disk numbers and system utilization. Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Lesson #3: FC-AL Algorithms 4 4 FAA FAA 3 RCF 3 RCF FLF FLF 2 CF 2 CF 1 1 Data waiting time (ms) 0 0 Cmd waiting time (ms) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 FC-AL utilization FC-AL utilization Data average waiting time varies with FC-AL Command average waiting time varies with FC-AL utilization. utilization FAA: Fairness Access Algorithm RCF: Read Command First FLF: FL-port First CF: Commands First Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 Lastly…. The queuing network model has been shown to be a useful tool for analyzing the overall SAN system performance The model was also used to analyze our advanced SAN storage technology, DA2 We missed the proceedings publication. However, this paper will be made available on the conference website! More Questions? Visit our Poster Session Thank you for your attention! Zhu Yaolong 10th NASA Goddard Conference on Mass Storage Systems and Technologie s 2002 .