Quest-V: A Virtualized Multikernel for High-Confidence Systems Ye Li Matthew Danish Richard West Boston University Boston University Boston University [email protected] [email protected] [email protected] Abstract processors pose significant challenges to operating system de- This paper outlines the design of ‘Quest- !" which is implemented sign 3:" ;" 8@4& Not only is it difficult to design software sys- as a collection of separate $ernels operating together as a dis- tems that scale to large numbers of processing cores, there are nu- tributed system on a chip. Quest-V uses virtualization techniques to merous micro-architectural factors that affect software e-ecution, isolate $ernels and prevent local faults fro# affecting remote $er- leading to reduced efficiency and unpredictability& Shared on-chip nels. This leads to a high-confidence multikernel approach, where caches 3A9" 1B4" memory %us %andwidth contention 3114" hard- failures of syste# subcomponents do not render the entire system ware interrupts 3184" instruction pipelines" hardware prefetchers" inoperable. * virtual machine monitor for each $ernel $eeps track amongst other factors" all contribute to variability in task e-ecution of shadow page table mappings that control immutable memory ac- times. cess capabilities. This ensures a level of security and fault tolerance +oupled with the challenges posed by multicore processors in situations where a service in one $ernel fails, or is corrupted by a are the inherent complexities in modern operating systems& Such malicious attack. +ommunication is supported between $ernels us- comple- interactions between software components inevitably lead ing shared memory regions for message passing. ,imilarly" device to program faults and potential compromises to system integrity& driver data structures are shareable between $ernels to avoid the arious faults may occur due to memory violations (e.g., stack and need for comple- ./0 virtualization, or communication with a ded- %uffer overflows, null pointer dereferences and jumps or stores to icated $ernel responsible for ./0& .n Quest- " device interrupts are out of range addresses 39E" A;4=" +PU violations (e.g., starvation delivered directly to a $ernel, rather than via a monitor that deter- and deadlocks=" and ./0 violations (e.g." mismanagement of access mines the destination. Apart from %ootstrapping each $ernel, han- rights to files and devices=& Device drivers, in particular" are a dling faults and managing shadow page tables" the monitors are not known source of potential dangers to operating systems, as they needed. This differs from conventional virtual machine systems in are typically written by third party sources and usually e-ecute with which a central monitor" or hypervisor" is responsible for schedul- $ernel privileges. To address this" various researchers have devised ing and management of host resources amongst a set of guest $er- techniques to verify the correctness of drivers, or to sandbox them nels. In this paper we show how Quest-V can implement novel fault fro# the rest of the $ernel 38E" 8:4& isolation and recovery techniques that are not possible with conven- In this paper" we present a new system design that uses both vir- tional systems& We also show how the costs of using virtualization tualization capabilities and the redundancy offered by multiple pro- for isolation of system services does not add undue overheads to cessing cores, to develop a reliable system that is resilient to soft- the overall syste# performance. ware faults& 0ur syste#" called ‘Quest- ! is designed as a multik- ernel 3:4" or distributed system on a chip. It uses virtualization tech- Categories and Subject Descriptors D.4.7 3Operating Systems45 niques to isolate $ernels on different cores of a multicore proces- Organization and Design sor& Shadow page tables 384 securely isolate separate $ernel images General Terms Design, Reliability in physical memory& These page tables map each $ernels ‘guest’ physical memory to host (or machine= physical memory& Changes Keywords irtualization" Multicore, 6ault Isolation and Recovery to protection bits within shadow page tables can only be performed by a monitor associated with the $ernel on the corresponding core. 1. Introduction This ensures that any illegal memory accesses (e.g&" write attempts Multicore processors are now ubiquitous in today’s microprocessor on read-only pages) within a $ernel are caught by the correspond- and microcontroller industry& It is common to see two to four cores ing monitor& Our system has similarities to Barrelfish, in that it is a per package in embedded and desktop platforms" with server-class multikernel, while also using virtualization similar to systems such processors such as the Sun Niagara 8 having 9: cores and up to as Xen 3E4& We differ from traditional virtualized systems 3@4 by ; hardware threads per core. Similarly" Intel’s Single-chip +loud only trapping into a monitor when a fault occurs. .n all other sit- +omputer <,++= supports 1; cores, and other manufacturers are uations" e-ecution proceeds within the $ernels or user-spaces on following suit with an increase in core count driven in part by trade- each core. Interrupts and communication directly involve $ernels" offs in power and computational demands. Many emerging multi- without monitor intervention. Thus, for the most part, only memory core processors now have support for hardware virtualization (e.g., virtualization using shadow paging is used for fault isolation. Intel VT and AMD- +PUs). irtualization has re-emerged in the We show how Quest- does not incur significant operational last decade as a way to consolidate workloads on servers, thereby overheads compared to a non-virtualized version of our system, providing an effective means to increase resource utilization while simply called Quest, designed for SMP platforms& We observe that still ensuring logical isolation between guest virtual machines& communication, interrupt handling, thread scheduling and system Hardware advances with respect to multicore technology have call costs are on par with the costs of conventional SM> systems" not been met by software developments& In particular" multicore with the advantage that Quest-V can tolerate syste# component a system. .n other respects" Quest-V allows sandbox $ernels to failures without the need for syste# reboots. communicate via shared memory channels that allow the exchange We show how Quest-V can recover from component failure us- of messages, and via which services in one sandbox can access ing a network device driver example, whereby we detect software those in another& Similarly" physical devices are shareable across failure and restart a comparable driver either in a locally-recovered sandbox $ernels and do not require a monitor to manage them on $ernel or in an alternate $ernel on another core& This serves as an behalf of guests& example of the ‘self-healing! characteristics of Quest-V" with on- Finally" although this is not the primary focus of this paper" line fault recovery being useful in situations where high-confidence Quest-V is designed around time as a first-class resource& We have (or high availability= is important. This is typically the case with been developing the system to support virtual +PUs (VCPUs=" many real-time and embedded mission-critical systems found in which we will briefly describe in Section 8&1& 6undamentally" the healthcare, avionics, factory automation and automotive systems" goal here is to ensure that time management of +PU resources for example. are accurately accounted on behalf of all threads, either those as- In the following two sections, we describe the Quest-V architec- sociated with conventional tasks or interrupt handlers& .nterrupt tural design, introducing the goals first, followed by an overview of handlers are schedulable in Quest- and their priority is carefully the system architecture. This is followed in ,ection 1 by an exper- aligned with any thread that may have initiated the#" such as via a imental evaluation of the system& ?ere, we show the overheads of prior ./0 request. We have shown this leads to a system that ensures online device driver recovery for a network device, along with the temporal isolation %etween tasks and interrupts, so that no such en- costs of using hardware virtualization to isolate $ernels and system tity ever e-ecutes for longer than its %udgeted time in any window components. Section E describes related work, while conclusions of real-time. and future work are discussed in Section :& " Design Goals % Quest-V Architecture &'er'ie( The design and ongoing development of Quest- is centered As stated earlier" Quest-V relies on virtualization support for safety around three main goals5 safety" predictability and ef)ciency& As and isolation of system software components& The current imple- part of our investigations into system safety" we have studied var- mentation runs in 32-bit mode and is designed specifically for Intel ious methods for hardware and software fault isolation, including -;: processors with virtual machine extensions. Plans are under- the use of type-safe languages 32" AB" A1" AE4 and hardware fea- way to support AMD and future *RM processors. tures such as paging, segmentation 399" 8;4" and virtual machine Unlike #ost of the traditional hypervisor and virtual machine support 384& systems, Quest-V primarily uses virtualization to implement an ex- Quest-V is intended for safety-critical application domains, re- tra logical ring of memory protection into which sandbox address quiring high confidence. The National Coordination 0ffice for Net- spaces are mapped. We define a sandbo- domain to be a collec- working and Information Technology Research and Development tion of one or more CPUs and the host physical memory that in- <7+0/NITRD) defines a high confidence system as follows 39245 cludes the local monitor" its sandbox $ernel and all local applica- “A high confidence system is one in which the consequences of tions. There is no need for CPU virtualization as the instructions its behavior are well understood and predictable.