1 SPEK: A Storage Performance Evaluation Kernel Module for Block Level Storage Systems under Faulty Conditions Xubin He, Member, IEEE, Ming Zhang, Member, IEEE, and Qing (Ken) Yang, Senior Member, IEEE Abstract— This paper introduces a new benchmark tool, cables are also employed at the network level. Software SPEK (Storage Performance Evaluation Kernel module), mechanisms are designed to bypass failed components to for evaluating the performance of block-level storage provide continued data availability. Different topological systems in the presence of faults as well as under normal architectures and fault-tolerant mechanisms exist for a operations. SPEK can work on both Direct Attached SAN (storage area network), and new ideas and tech- Storage (DAS) and block level networked storage systems nologies emerge rapidly [3]. In order to make design de- such as storage area networks (SAN). Each SPEK consists of a controller, several workers, one or more probers, and cisions and provide optimal storage solutions, it is highly several fault injection modules. Since it runs at kernel level desirable to have efficient benchmark tools to quantita- and eliminates skews and overheads caused by file systems, tively evaluate performance of various SAN architectures SPEK is highly accurate and efficient. It allows a storage under faulty conditions. Current benchmark tools focus- architect to generate configurable workloads to a system ing on performance evaluation are not efficient enough under test and to inject different faults into various system to accurately measure the behavior of storage systems. components such as network devices, storage devices, and Under many circumstances, the performance of a storage controllers. Available performance measurements under system available to users is the result achieved by file different workloads and faulty conditions are dynamically collected and recorded in SPEK over a spectrum of systems. This result is influenced by many factors such as time. To demonstrate its functionality, we apply SPEK to file system caches, data organization, and buffer caches, evaluate the performance of two direct attached storage so it cannot represent the true performance of the storage. systems and two typical SANs under Linux with different For some applications, such as databases, which can fault injections. Our experiments show that SPEK is highly utilize the raw performance of a storage device directly, efficient and accurate to measure performance for block- it is desirable and necessary to measure and compare level storage systems. different storage systems at the raw (block) level. It is Index Terms— Measurement Techniques, Performance also important for file system and OS designers to know Analysis, Degraded performance, Data Storage, Disk I/O how much potential raw performance they could exploit and how much optimization they have made. Existing benchmark tools such as PostMark [4], Io- I. INTRODUCTION Zone [5], Bonnie++ [6], and IoMeter [7] are widely used EING able to access data efficiently and reliably to measure various storage systems. PostMark, IoZone, B has become the first priority of many organizations and Bonnie++ run at the file system level and there- in today's information age. To achieve this goal, a fore mainly characterize file system performance. Fig. typical data storage system has built-in redundancies at 1 shows experimental performance measurements of a various levels. At the storage device level, redundancy same SCSI disk under different file system options using is achieved using RAID (redundant array of inexpensive PostMark, IoZone, and Bonnie++. Although we use the disks) [1], [2]. At the controller level, multiple HBAs same disk and same measurement metric (throughput in (host bus adapters) and NICs (network interface cards) terms of KB/second), these benchmark tools produce are used. Redundant switches, bridges, and connecting completely different performance results. Such devia- X. He is with the Department of Electrical and Computer Engi- tions can be attributed to effects of the file system cache neering,Tennessee Technological University, Cookeville, TN 38505. as well as different characteristics of file systems [8]. E-mail: [email protected]. While IoMeter can run below file systems, its measured M. Zhang and Q. Yang are with the Department of Electrical performance on Linux fluctuates dramatically due to the and Computer Engineering, University of Rhode Island, Kingston, RI 02881. effects of buffer caches. Email: mingz,qyang ¡ @ele.uri.edu. Many operating systems provide a “raw” interface 2 4 4 x 10 x 10 7 7 2500 6 6 2000 5 5 Ext2 Ext2 1500 Ext2 4 4 Ext3 Ext3 Ext3 3 Ext2 Sync 3 Ext2 Sync 1000 Ext2 Sync 2 2 Throughput (KB/s) Throughput (KB/s) Throughput (KB/s) 500 1 1 0 0 0 Read Write Read Write Read Write Fig. 1. Measured throughput of PostMark (left), IoZone (middle), and Bonnie++ (right). Although using the same hardware, measured performance change dramatically with changes of file system options. bypassing the file system. Simply using such an interface measure storage performance at the file system level. will not efficiently produce accurate performance results Second, our benchmark tool produces degraded per- for block level storage systems for the following two formance measurements of storage architectures under reasons. First, the raw interface provides a character faulty conditions. Specifically SPEK measures perfor- device, as opposed to block device that is used by all mance levels in the presence of various faults over time storage systems. Second, since the “raw” interface is a instead of using an average percentage of “up” time as an user level interface, operating on it requires many context availability metric. It allows a user to generate workloads switches, giving rise to a great amount of overhead. for a block level storage system, to inject faults at Such overhead becomes significant when measuring high different parts of the storage system such as networks, performance storage systems such as RAID and SAN. storage devices, and storage controllers. By generat- Besides the accuracy problem of existing benchmark ing configurable workloads and injecting configurable tools, they also raise an efficiency issue. Because these faults, users can grab the dynamic changes of potentially benchmarks run in a user space, excessive number of compromised performance and therefore quantitatively system calls and context switches result in large amounts evaluate system performance of a measured SAN. To of overhead. This problem is more pronounced when demonstrate how SPEK works, we have performed sev- measuring high performance networked storage systems eral tests on direct-attached storage systems including because the intensity of traffic generated by these bench- single disks and disk arrays and two storage area network marks is limited due to excessive system overheads. As a systems: an iSCSI-based [14] SAN and a STICS-based result, a large number of clients are needed to saturate a [15] SAN. high performance block level networked storage system. The paper is organized as follows. In next Section, we In addition to performance measurement, degraded discuss the architecture and design of SPEK in detail. performance under faulty conditions is another concern In Section 3, we present measurement results using for any storage system. Trivedi et al. [9]–[12] have exten- SPEK on two direct attached storage systems and two sively studied availability modeling for multi-processor networked storage systems. We discuss related work in systems and wireless communication networks using Section 4 and conclude this paper in Section 5. Markov reward models. Little work has been done on benchmark tools considering the degraded performance II. ARCHITECTURE AND DESIGN OF SPEK of a networked storage system under faulty conditions. The overall structure of SPEK is shown in Fig. 2. One exception is the research by Brown and Patterson It consists of several components, including one SPEK [13] who advocated for availability benchmarking with controller, several SPEK workers, one or more SPEK a case study on evaluating the availability of software probers, and different types of fault injection modules. RAID systems. A SPEK controller resides on a controller machine which We introduce here a new benchmark tool for eval- is used to coordinate SPEK workers and probers. It can uating performance in consideration of various failure start/stop SPEK workers and probers, send commands, conditions of a storage system at the block level, which and receive responses from them. A Java GUI interface is referred to as SPEK (Storage Performance Evaluation allows a user to input configuration parameters such as Kernel module). The contributions of SPEK are two-fold. workload characteristics and to view measured results. First, we propose a benchmark tool to a storage designer Each SPEK controller also has a data analysis module or purchaser to evaluate the performance of a storage to analyze measured data. system at the block level. The tool is highly efficient One SPEK worker runs on each testing client to and accurate compared to existing benchmark tools that generate storage requests via the low level device driver 3 Worker Network Fault Injector Storage Controller Storage Device Worker Network Fault Injector Prober Storage Fault Injector Controller ... ... Controller Fault Injector Disk Drives Worker Network Fault Injector N−SPEK Component Network Connection Storage SCSI/FC/... cable Fig. 2. SPEK Structure. It contains one