A Survey on Shared Disk I/O Management in Virtualized Environments Under Real Time Constraints
Total Page：16
File Type：pdf, Size：1020Kb
A survey on shared disk I/O management in virtualized environments under real time constraints Ignacio Sañudo Roberto Cavicchioli Nicola Capodieci University of Modena University of Modena University of Modena and Reggio Emilia and Reggio Emilia and Reggio Emilia ignacio.sanudoolmedo [email protected] nicola.capodieci @unimore.it @unimore.it Paolo Valente Marko Bertogna University of Modena University of Modena and Reggio Emilia and Reggio Emilia [email protected] [email protected] ABSTRACT the case of next-generation automotive architectures, where In the embedded systems domain, hypervisors are increas- cost-effective solutions ever more require sharing an on-board ingly being adopted to guarantee timing isolation and appro- computing platform among different applications with het- priate hardware resource sharing among different software erogeneous safety and criticality levels, e.g., the infotain- components. However, managing concurrent and parallel ment part on one side, and a safety-critical image processing requests to shared hardware resources in a predictable way module on the other side. These domains are independent, still represents an open issue. We argue that hypervisors with different period, deadline, safety and criticality require- can be an effective means to achieve an efficient and pre- ments. However, they need to be properly isolated with no dictable arbitration of competing requests to shared devices mutual interference, or a misbehaving module may endanger in order to satisfy real-time requirements. As a representa- the timely execution of a high-criticality domain, affecting tive example, we consider the case for mass storage (I/O) safety qualification. devices like Hard Disk Drives (HDD) and Solid State Disks In order to provide real-time guarantees, hypervisors ei- (SSD), whose access times are orders of magnitude higher ther dynamically schedule virtual machines according to a than those of central memory and CPU caches, therefore given on-line policy, or they statically partition virtual ma- having a greater impact on overall task delays. We provide chines to the available hardware resources. An example of a comprehensive and up-to-date survey of the literature on the first category is RT-Xen [23] (now merged into mainline I/O management within virtualized environments, focusing Xen [1]), which implements a hierarchical virtual machine on software solutions proposed in the open source commu- scheduler managing both real-time and non-real-time work- nity, and discussing their main limitations in terms of real- loads using the Global Earliest Deadline First (G-EDF) al- time performance. Then, we discuss how the research in gorithm. On the other hand, statically partitioned solutions this subject may evolve in the future, highlighting the im- tend to isolate virtual machines onto dedicated cores, with portance of techniques that are focused on scheduling not an exclusive assignment of hardware resources. An exam- uniquely the processing bandwidth, but also the access to ple of this approach is given by Jailhouse [18], which does other important shared resources, like I/O devices. not allow multiple virtual machines to share the same core. An advantage of this latter approach is that the resulting hypervisors have a typically smaller code footprint, imply- 1. INTRODUCTION ing much lower certification costs. Indeed, reducing the code A hypervisor, also called Virtual Machine Manager (VMM), size is a prominent characteristic of other recent VMMs, like 1 is a combination of software and hardware components that NOVA [19] and bhyve ). allow emulating the execution of multiple virtual machines No matter which virtualization approach is taken, I/O for upon the same computing platform by properly arbitrating storage devices might becomes a bottleneck. This is due to the concurrent access to shared hardware resources. Most of the added layer of complexity introduced by the hypervisor the available open source hypervisors are specifically tailored itself, as shown in section 5. most of the current literature to server applications and cloud computing. In these areas, on resource access arbitration for virtualized environments hypervisors are mainly designed to provide isolation, load mainly focuses on CPU scheduling (see for example surveys balancing, server consolidation and desktop virtualization [8] and [21]), neglecting the huge impact that the access within the managed virtual machines. However, the emerg- to other shared hardware resources, like Hard Disk Drives ing of new potential areas for VMMs, such as automotive (HDD) and Solid State Disks (SSD), may have on time- applications and other embedded systems, and the possi- critical tasks. In view of this consideration, this paper pro- bility to exploit multi-core embedded processors are pos- vides a survey on the state-of-the-art on I/O virtualization ing new challenges to real-time systems engineers. This is and concurrent HDD/SSD read/write operations. We will discuss the applicability of previously introduced solutions EWiLi'16, October 6th, 2016, Pittsburgh, USA. Copyright retained by the authors. 1https://wiki.freebsd.org/bhyve to I/O arbitration for enhancing the real-time guarantees obtain and may significantly vary on different architectures. that may be provided in a virtualized environment. Main This makes it extremely difficult to develop a tight tim- limitations of classic fair provisioning schemes to resource ing analysis even for simpler platforms. To sidestep these sharing will be highlighted. problems, we are studying scheduling solutions that aim at We are interested in software-based solutions that do not avoiding conflicts on the device arbiter, by properly shaping require customized device controllers and hardware mech- the device requests from the different cores. Hypervisors are anisms to obtain the desired behavior. Therefore, most of natural candidates in this sense, providing a centralized de- the addressed works deal with virtualized approaches that cision point with a global view of the requests from the var- schedule the access to storage devices by means of a hypervi- ious partitions. This would allow taking the unpredictable sor or similar mechanisms, shaping the I/O requests to guar- arbiters out of the scheduling loop, leaving the resource man- antee a given I/O bandwidth to multiple partitions/cores. agement at hypervisor level. Before implementing such a For each of the presented works, we will highlight the main solution, we examined the existing related approaches for weaknesses and limitations, in order to stimulate the real- managing shared resources in virtualized environments, tak- time research community to undertake a more rigorous and ing storage devices as a representative example. structured effort towards achieving the required predictabil- This choice is mainly due to the large interest in I/O ity guarantees. scheduling within the open source community. Modifica- Contributions are divided by contexts. In this respect, a tions to the current Linux schedulers are constantly being first coarse-grained distinction is made considering the tech- proposed and evaluated. For example, at the time of writ- nology used for data storage: rotational or non rotational. ing, a new scheduler denoted as BFQ (Budget Fair Queu- HDDs and flash-based devices, such as SSDs, may have sim- ing) [22] is undergoing evaluation for being merged into ilar issues when it comes to arbitration of concurrent ac- mainline Linux. This I/O scheduler is based on CFQ, the cesses, but their radically different operating principles en- default I/O scheduler in most Linux systems. Among other tail different problems to solve in order to ensure a pre- goals, BFQ is designed to outperform CFQ in terms of the dictable behavior. A finer-grained distinction is related to soft real-time requirements that can be guaranteed to mul- arbitration policies, examining how different I/O scheduling timedia applications. However, it remains unclear how the algorithms behave in a virtualized environment and whether proposed modifications can deal with harder real-time con- they are able to satisfy hard/soft real-time guarantees. It straints, given the unpredictable technical characteristics of is worth stressing that most of the work on virtualization storage devices. in I/O environments is based on or concerned with the Xen A second motivation descends from the consideration that hypervisor. sub-optimal arbitration policies of an I/O storage device can The rest of the paper is organized as follows. The next sec- be the primary cause of blocking delays and performance tion introduces the motivation behind the presented survey. drops. This is due to the considerably worse latencies and Section 3 describes the existing solution based on the Xen bandwidths of storage devices with respect to other shared hypervisor for I/O management. Section 4 discusses stat- resources, such as central memory or CPU caches. As an ically partitioned solutions for multi-core platforms. Sec- example, the random access times to L1, L2, L3 and DDR tion 5 highlights performance, predictability and security main memory on an Intel R i7 architecture are in the order issues related to the layered scheduling systems implied by of 1ns, 10ns, 50ns and 100ns 2, respectively. In contrast, the many virtualization techniques. Existing works introducing random access times to SSDs and HDDS are considerably real-time parameters within the I/O scheduler are summa- higher, in the order of 100us and 10ms 3, respectively. De- rized in Section 6, while Section 7 discusses the additional spite the cost of SSDs random accesses is predicted to drop predictability problems incurred with current SSD devices. to 10/50us in the next years, the gap from the main memory A final discussion is provided in Section 8 showing promis- access times would still be of at least two orders of magni- ing research lines to improve the predictable management tude. Due to this difference, it is of paramount importance of shared hardware resources by means of properly designed to properly schedule and coordinate the access to storage hypervisor mechanisms. peripherals.