MASTER’S THESIS | LUND UNIVERSITY 2013 A thin MIPS hypervisor for embedded systems Mikael Sahlström Department of Computer Science Faculty of Engineering LTH ISSN 1650-2884 LU-CS-EX 2013-38 A thin MIPS hypervisor for embedded systems Mikael Sahlström [email protected] October 15, 2013 Master’s thesis work carried out at SICS Swedish ICT. Supervisor: Arash Vahidi, [email protected] Examiner: Per Andersson, [email protected] Abstract Embedded systems are becoming more and more important as both the systems and their applications are getting more advanced. This in- creases the demand of and emphasizes the need for securing these sys- tems. Virtualization is one tool in solving this problem by providing isolation between trusted critical applications and untrusted feature rich applications. In this thesis we investigate how a thin hypervisor can provide isolation on an MIPS based embedded system with a minimal footprint and performance impact as well as implement and evaluate it. Keywords: Hypervisor, MIPS, embedded systems, virtualization, security Abstrakt Inbyggda system blir mer och mer viktiga då både systemen och pro- gram som körs på dem blir mer avancerade. Detta ökar kraven på och betydelsen av säkerhet i dessa system. Virtualisering är en lösning på det här problemet och kan isolera kritiska program från icke betrodda pro- gram med mer och större funktioner. I den här rapporten så undersöker vi hur en tunn hypervisor kan förse ett inbyggt MIPS baserat system med isolering utan att offra för mycket prestanda. Vi kommer göra detta genom att implementera och evaluera en tunn hypervisor. 2 Acknowledgements I would like to thank Arash Vahidi for the guidance and support, Viktor Do for putting up with all my questions and everyone else at the SICS security lab in Lund. I would also like to thank Per Andersson and Jonathan Kämpe for the feedback. 3 4 Contents 1 Introduction 11 1.1 Purpose and goals ............................ 12 1.2 Thesis overview .............................. 12 2 Virtualization 13 2.1 Isolation .................................. 14 2.2 Types of virtualization .......................... 15 2.2.1 ISA translation .......................... 15 2.2.2 Para-virtualization ........................ 15 2.2.3 Hardware support ........................ 15 2.3 Virtual addressing ............................ 16 2.4 Page table ................................. 17 2.5 Putting it all together .......................... 18 3 The MIPS architecture 19 3.1 Overview ................................. 19 3.1.1 Registers ............................. 20 3.1.2 Coprocessors ........................... 20 3.1.3 Pipeline .............................. 21 3.1.4 Memory .............................. 21 3.2 Exceptions and interrupts ........................ 23 3.3 Coprocessor 0 ............................... 24 3.3.1 Coprocessor 0 registers ...................... 25 3.3.2 CP0 hazards ........................... 27 3.4 Caches ................................... 27 3.5 Memory management unit (MMU) and the TLB ............ 28 3.5.1 CP0 TLB registers ........................ 28 3.5.2 TLB entries ............................ 29 3.5.3 A TLB refill handler ....................... 30 5 CONTENTS 4 Implementation of a thin hypervisor 33 4.1 Structure ................................. 33 4.2 Memory management ........................... 35 4.2.1 Memory layout .......................... 36 4.2.2 Translation lookaside buffer ................... 36 4.2.3 Page tables ............................ 37 4.3 Isolation .................................. 38 4.4 Hypercalls ................................. 38 5 Evaluation 41 5.1 Methods .................................. 41 5.2 TLB refills ................................. 42 5.3 Hypercalls ................................. 43 5.4 Interrupts ................................. 44 5.5 System calls ................................ 44 5.6 Multiple applications or multiple guests ................. 45 5.7 Isolation .................................. 46 5.8 Hypervisor size .............................. 47 6 Conclusions 49 6.1 Future work ................................ 49 Bibliography 51 6 List of Figures 2.1 Virtualization in a nutshell. ....................... 13 2.2 Virtual addressing of the pages A, B, C and D [7, p. B-41]. ...... 16 2.3 Mapping of a virtual address to physical memory with a 2-level page table [7, p. B-45]. ............................. 17 3.1 MIPS32 memory map [6, p. 51]. ..................... 22 3.2 Fields in the status register [6, p. 60]. .................. 25 3.3 A TLB entry [6, p. 132]. ......................... 29 3.4 EntryHi and PageMask register fields [6, p. 134]. ............ 29 3.5 EntryLo0-1 register fields [6, p. 136]. .................. 30 4.1 Basic hypervisor structure. ........................ 34 4.2 Memory layout of the hypervisor and guests. .............. 36 5.1 Memory accesses with both sequential and pseudorandom accesses. 43 5.2 Illustration of how isolation is maintained. ............... 46 7 LIST OF FIGURES 8 List of Tables 3.1 Registers and their use [6, p. 36]. .................... 20 3.2 Exceptions and their mnemonic [6, p. 66 - 67]. ............. 24 3.3 CP0 control registers for memory management [6, p. 133]. ...... 28 4.1 Hypercalls provided by the hypervisor. ................. 39 5.1 Amount of instructions spent on each hypercall. ............ 44 5.2 Amount of instructions spent on each interrupt. ............ 44 5.3 Amount of instructions spent on each system call. ........... 45 5.4 Running multiple applications in one guest compared to running mul- tiple guests with one application. .................... 45 5.5 Size of the hypervisor files in lines of code, including comments and empty lines. ................................ 47 5.6 The amount of memory used by the hypervisor where n is the amount of guests and r is the amount of page table entries. .......... 47 5.7 The amount of memory used by a guest without the hypervisor. ... 48 9 LIST OF TABLES 10 Chapter 1 Introduction Virtualization is an important tool in securing applications by providing isolation with a relatively low computational overhead. A hypervisor enables virtualization and lies between the hardware and one or more virtual machines as an abstrac- tion layer. By controlling hardware accesses the hypervisor can protect sensitive applications from each other and themselves [2]. As embedded systems are becoming more advanced with full featured systems like Android running third party applications, the need for securing these systems become more and more apparent. Developers are presented with an interesting challenge: how can critical software run securely side by side with untrusted software without sacrificing too much performance? Virtualization with a thin hypervisor, a minimal low footprint hypervisor, is one solution to this problem. For example could a real-time operating system run critical communication software while noncritical applications run in a more user oriented system like Android and the hypervisor makes sure these two systems are separated from each other. This approach lets critical systems be small and manageable, minimizing risk for bugs and can make verification of the critical software easier while still having a modern user experience that, if compromised, will not affect the secure system [4]. Virtualization also can provide better hardware utilization which can be utilized on servers where virtualized applications can be moved between hardware as load varies. However in embedded systems, there are usually not much room for advanced hardware utilization tricks. Here, it is the security aspects of virtualization that are interesting. A thin hypervisor can provide virtualization while having a minimal memory footprint and low overhead. By keeping the amount of hypervisor code to a min- imum, we can reduce the risk for bugs and simplify code audits and verifications. These aspects, among others, makes virtualization a very interesting technique to use in embedded systems where many applications can benefit from it. 11 1. Introduction 1.1 Purpose and goals The purpose of this thesis is to explore how a thin hypervisor can be implemented on the MIPS architecture and how it can provide virtualization with a minimal footprint and performance impact. This thesis is divided into four tasks: • A study of the MIPS architecture with most focus on parts relevant to hyper- visor design. • Design of a thin hypervisor for MIPS capable of running multiple guests. • Implementation of a thin hypervisor running on a simulated embedded plat- form. • Analysis of security and performance aspects. The hypervisor implementation will be done entirely within the context of this thesis and by the authors of it. To be able to run our hypervisor we will use the Open Virtual Platforms (OVP) simulation tool to simulate an embedded platform. This platform is constructed by writing C code describing the platform and compiling this together with the existing peripheral descriptions in OVP. The file containing the platform description code consists of less than 100 lines of code. We need guests to use when testing the implementation. These guests should be as close as possible to what would be used in real world applications. Therefore, we choose to base our guests on the FreeRTOS real-time operating system which is a viable and widely used real-time operating system for embedded systems and has support for over 33 architectures [18]. To be able to use FreeRTOS as a guest, we have to modify it so that it