I/O Workload Outsourcing for Boosting RAID Reconstruction Performance
Total Page:16
File Type:pdf, Size:1020Kb
WorkOut: I/O Workload Outsourcing for Boosting RAID Reconstruction Performance Suzhen Wu1, Hong Jiang2, Dan Feng1∗, Lei Tian12, Bo Mao1 1Key Laboratory of Data Storage Systems, Ministry of Education of China 1School of Computer Science & Technology, Huazhong University of Science & Technology 2Department of Computer Science & Engineering, University of Nebraska-Lincoln ∗Corresponding author: [email protected] {suzhen66, maobo.hust}@gmail.com, {jiang, tian}@cse.unl.edu, [email protected] Abstract ing reconstruction without serving any I/O requests from User I/O intensity can significantly impact the perfor- user applications, and on-line reconstruction, when the mance of on-line RAID reconstruction due to contention RAID continues to service user I/O requests during re- for the shared disk bandwidth. Based on this observa- construction. tion, this paper proposes a novel scheme, called WorkOut Off-line reconstruction has the advantage that it’s (I/O Workload Outsourcing), to significantly boost RAID faster than on-line reconstruction, but it is not practical reconstruction performance. WorkOut effectively out- in environments with high availability requirements, as sources all write requests and popular read requests orig- the entire RAID set needs to be taken off-line during re- inally targeted at the degraded RAID set to a surrogate construction. RAID set during reconstruction. Our lightweight pro- On the other hand, on-line reconstruction allows fore- totype implementation of WorkOut and extensive trace- ground traffic to continue during reconstruction, but driven and benchmark-driven experiments demonstrate takes longer to complete than off-line reconstruction as that, compared with existing reconstruction approaches, the reconstruction process competes with the foreground WorkOut significantly speeds up both the total recon- workload for I/O bandwidth. In our experiments we find struction time and the average user response time. Im- that on-line reconstruction (with heavy user I/O work- portantly, WorkOut is orthogonal to and can be easily loads) can be as much as 70 times (70×) slower than incorporated into any existing reconstruction algorithms. off-line reconstruction (without user I/O workloads) (see Furthermore, it can be extended to improving the perfor- Section 2.2). Moreover, while on-line reconstruction al- mance of other background support RAID tasks, such as lows foreground workload to be served, the performance re-synchronization and disk scrubbing. of the foreground workload might be significantly re- duced. In our experiments, we see cases where the user 1 Introduction response time increases by a factor of 3 (3×) during on- As a fundamental technology for reliability and availabil- line reconstruction(see Section 2.2). ity, RAID [30] has been widely deployed in modern stor- Improving the performance of on-line RAID recon- age systems. A RAID-structured storage system ensures struction is becoming a growing concern in the light of that data will not be lost when disks fail. One of the key recent technology trends: reconstruction times are ex- responsibilities of RAID is to recover the data that was pected to increase in the future, as the capacity of drives on a failed disk, a process known as RAID reconstruc- grows at a much higher rate than other performance pa- tion. rameters, such as bandwidth, seek time and rotational la- The performance of RAID reconstruction techniques tency [10]. Moreover, with the ever growing number of depends on two factors. First, the time it takes to com- drives in data centers, reconstruction might soon become plete the reconstruction of a failed disk, since longer re- the common mode of operation in large-scale systems construction times translate to a longer “window of vul- rather than the exception [4, 9, 18, 32, 34]. nerability”, in which a second disk failure may cause per- A number of approaches have been proposed to im- sistent data loss. Second, the impact of the reconstruction prove the performance of RAID reconstruction, includ- process on the foreground workload, i.e., to what degree ing for example optimizing the reconstruction work- are user requests affected by the ongoing reconstruction. flow [12, 22], the reconstruction sequence [3, 41] or the Current approaches for RAID reconstruction fall into data layout [14, 49]. We note that all these approaches two different categories: [11, 12]: off-line reconstruction, focus on a single RAID set. In this paper we propose a when the RAID devotes all of its resources to perform- new approach to improving reconstruction performance, USENIX Association FAST ’09: 7th USENIX Conference on File and Storage Technologies 239 exploiting the fact that most data centers contain a large evaluations of WorkOut based on a prototype implemen- number of RAID sets. tation are presented in Section 4. We analyze the relia- Inspired by recent work on data migration [2, 24, 46] bility of WorkOut in Section 5 and present related work and write off loading [27], we propose WorkOut, a frame- in Section 6. We point out directions for future research work to significantly improve on-line reconstruction per- in Section 7 and summarize the main contributions of the formance by I/O Workload Outsourcing. The main idea paper in Section 8. behind WorkOut is to temporarily redirect all write re- quests and popular read requests originally targeted at 2 Background and Motivation the degraded RAID set to a surrogate RAID set. The sur- In this section, we provide some background and key ob- rogate RAID set can be free space on another live RAID servations that motivate our work and facilitate our pre- set or a set of spare disks. sentation of WorkOut in later sections. The benefits of WorkOut are two-fold. WorkOut re- duces the impact of reconstruction on foreground traffic 2.1 Disk failures in the real world because most user requests can be served from the surro- Recent studies of field data on partial or complete disk gate RAID set and hence no longer compete with the re- failures in large-scale storage systems indicate that disk construction process for disk bandwidth. WorkOut also failures happen at a significant rate [4, 9, 18, 32, 34]. speeds up the reconstruction process, since more band- Schroeder & Gibson [34] found that annual disk re- width on the degraded RAID set can be devoted to the placement rates in the real world exceed 1%, with 2%- reconstruction process. 4% on average and up to 13% in some systems, much In more detail, WorkOut has the following salient fea- higher than 0.88%, the annual failure rates (AFR) spec- tures: ified by the manufacturer’s datasheet. Bairavasundaram [4] observed that the probability of latent sector (1) WorkOut tackles one of the most important factors et al. errors, which can lead to disk replacement, is 3.45% in adversely affecting reconstruction performance, their study. Those failure rates, combined with the con- namely, I/O intensity, that, to the best of our knowl- tinuously increasing number of drives in large-scale stor- edge, has not been adequately addressed by the pre- age systems, raise concerns that in future storage sys- vious studies [3, 41]. tems, recovery mode might become the common mode (2) WorkOut has a distinctive advantage of improv- of operation [9]. ing both reconstruction time and user response time. It is a very effective reconstruction optimiza- Another concern arises from recent studies showing a tion scheme focusing on optimizing write-intensive significant amount of correlation in drive failures, indi- workloads, a roadblock for many of the existing re- cating that, after one disk fails, another disk failure will construction approaches [41]. likely occur soon [4, 9, 18]. Gibson [9] also points out (3) WorkOut is orthogonal and complementary to and that the probability of a second disk failure in a RAID can be easily incorporated into most existing RAID system during reconstruction increases along with the reconstruction approaches to further improve their reconstruction time: approximately 0.5% for one hour, performance. 1.0% for 3 hours and 1.4% for 6 hours. (4) In addition to boosting RAID reconstruction perfor- All the above trends make fast recovery from disk fail- mance, WorkOut is very lightweight and can be eas- ures an increasingly important factor in building storage ily extended to improve the performance of other systems. background tasks, such as re-synchronization [7] 2.2 Mutually adversary impact of reconstruc- and disk scrubbing [36], that are also becoming tion and user I/O requests more frequent and lengthier for the same reasons that reconstruction is becoming more frequent and During on-line RAID reconstruction, reconstruction re- lengthier. quests and user I/O requests compete for the bandwidth of the surviving disks and adversely affect each other. Extensive trace-driven and benchmark-driven exper- User I/O requests increase the reconstruction time while iments conducted on our lightweight prototype im- the reconstruction process increases the user response plementation of WorkOut show that WorkOut sig- time. nificantly outperforms the existing reconstruction ap- Figure 1 shows the reconstruction times and user re- proaches PR [22] and PRO [42] in both reconstruction sponse times of a 5-disk RAID5 set with a stripe unit size time and user response time. of 64KB in three cases: (1) off-line reconstruction, (2) The rest of this paper is organized as follows. Back- on-line reconstruction at the highest speed (when RAID ground and motivation are presented in Section 2. We de- favors the reconstruction process), and (3) on-line recon- scribe the design of WorkOut in Section 3. Performance struction at the lowest speed (when RAID favors user 240 FAST ’09: 7th USENIX Conference on File and Storage Technologies USENIX Association Reconstruction Time Response Time Reconstruction Time Response Time 180 60 180 60 150 150 45 45 120 120 90 30 90 30 60 (minute) 15 (minute) 60 30 15 30 Response Time (ms) Response Time (ms) Reconstruction Time 0 0 Off-Line On-line On-line Recoonstruction Time 0 0 fastest slowest 9 43 59 150 200 Reconstruction Option I/O Intensity (IOPS) Figure 1: Reconstruction and its performance impact.