JYU DISSERTATIONS 132 Roee Shimon Leon Applications of Hypervisors in Security JYU DISSERTATIONS 132 Roee Shimon Leon Applications of Hypervisors in Security Esitetään Jyväskylän yliopiston informaatioteknologian tiedekunnan suostumuksella julkisesti tarkastettavaksi yliopiston vanhassa juhlasalissa S212 lokakuun 24. päivänä 2019 kello 10. Academic dissertation to be publicly discussed, by permission of the Faculty of Information Technology of the University of Jyväskylä, in building Seminarium, auditorium S212, on October 24, 2019 at 10 o’clock. JYVÄSKYLÄ 2019 Editors Timo Männikkö Faculty of Information Technology, University of Jyväskylä Ville Korkiakangas Open Science Centre, University of Jyväskylä Copyright © 2019, by University of Jyväskylä Permanent link to this publication: http://urn.fi/URN:ISBN:978-951-39-7854-9 ISBN 978-951-39-7854-9 (PDF) URN:ISBN:978-951-39-7854-9 ISSN 2489-9003 ABSTRACT Leon, Roee Shimon Applications of Hypervisors in Security Jyväskylä: University of Jyväskylä, 2019, 88 p. (+ included articles) (JYU Dissertations ISSN 2489-9003; 132) ISBN 978-951-39-7854-9 (PDF) Finnish summary Diss. As malware continue to evolve, so do the countermeasures which attempt to fight them. A modern computer system typically has many security services installed on top of its operating system which include antivirus, application-control, IDS, firewall, and many more. Modern operating systems are a highly complex pieces of software which typically contains millions of lines of code. Furthermore, primarily due to endless hardware support, new code is regularly added, resulting in a security sink with an open drain. Most security services run on top of the operating system and, therefore, are subject to the security of the operating system and its applications. In case of a vulnerability, these services can be removed, thus rendering them them com- pletely useless. This thesis proposes a thin hypervisor-based architecture for a system on top of which a variety of security services can be implemented. These services run in a secure, isolated environment. Furthermore, the proposed system can hide the presence of these security services. The proposed system architec- ture provides strong security guarantees. The thesis presents four common, heavily researched security problems and proposes four solutions, which are all based on the proposed architecture. The proposed solutions can compete, and even outperform current solutions, both in terms of security and performance. Keywords: trusted computing, virtualization, hypervisor, thin hypervisor, unau- thorised execution, malware analysis, code encryption, memory foren- sics TIIVISTELMÄ (ABSTRACT IN FINNISH) Leon, Roee Shimon Hyperviisoreiden sovelluksia tietoturvassa Jyväskylä: University of Jyväskylä, 2019, 88 s. (+artikkelit) (JYU Dissertations ISSN 2489-9003; 132) ISBN 978-951-39-7854-9 (PDF) Haittaohjelmien kehittyessä, myös vastatoimet niitä vastaan kehittyvät yhä kiih- tyvällä tahdilla. Nykyaikaisessa tietokonejärjestelmässä on tyypillisesti käyttöjär- jestelmän päälle asennettu useita tietoturvapalveluita, jotka sisältävät esimerkiksi virustentorjunta-, sovellusohjaus-, IDS-, palomuuri ja muita suojausmekanisme- ja. Nykyaikaiset käyttöjärjestelmät ovat erittäin monimutkaisia ohjelmistopa- ketteja, jotka sisältävät tyypillisesti miljoonia koodirivejä. Lisäksi, pääasiassa val- tavan laitteistotuen vuoksi, uutta koodia lisätään säännöllisesti, mistä seuraa tie- toturvariskejä. Useimmat tietoturvapalvelut toimivat käyttöjärjestelmän päällä, ja siksi ne ovat käyttöjärjestelmän ja sen sovellusten turvallisuuden osia. Haavoittuvuuden vuoksi nämä palvelut voidaan poistaa, jolloin ne eivät tietenkään ole käytössä ja ovat siten täysin hyödyttömiä. Tämä opinnäyte ehdottaa ohuita hypervisori- pohjaisia arkkitehtuureja järjestelmälle, jonka päälle voidaan toteuttaa erilaisia turvallisuuspalveluita. Nämä palvelut toimivat turvallisessa, eristetyssä ympä- ristössä. Lisäksi ehdotettu järjestelmä voi piilottaa näiden tietoturvapalvelujen näkyvyyden. Ehdotettu järjestelmäarkkitehtuuri tarjoaa siten vahvat turvallisuus- takuut. Opinnäytetyössä esitetään neljä yleistä tutkittua turvallisuusongelmaa ja ehdotetaan niille neljää ratkaisuvaihtoehtoa, jotka kaikki perustuvat työssä suun- niteltuun arkkitehtuuriin. Ehdotetut ratkaisut voivat kilpailla ja toimia tehok- kaammin kuin nykyiset ratkaisut sekä turvallisuuden että suorituskyvyn kan- nalta. Avainsanat: luotettava tietojenkäsittely, virtualisointi, hypervisori (virtuaalikonemonitori), luvattomat suoritukset, haittaohjelman analyysi, koodin salaus, muistin analyysi Author Roee Shimon Leon Faculty of Information Technology University of Jyväskylä Finland Supervisors Professor Pekka Neittaanmäki Faculty of Information Technology University of Jyväskylä Finland Doctor Nezer Zaidenberg Faculty of Information Technology University of Jyväskylä Finland Reviewers Professor Vincenzo Piuri Dept. of Computer Science University of Milan Italy Associate Professor, Dr. Jalil Boukhobza Lab-STICC Lab. / Dept. of Computer Science University of Western Brittany France Opponent Associate Professor, Dr. Miguel Correia Dept. of Informatics University of Lisbon Portugal ACKNOWLEDGEMENTS First, I would like to express my sincere gratitude to my supervisors, Prof. Pekka Neittaanmäki and Docent Nezer Jacob Zaidenberg, for their guidance and sup- port throughout the process of this thesis. In addition, I would like to thank my external reviewers for their valuable insights. I am also immensely grateful to Michael Kiperberg, Anat Anatey Leon Zabag, Amit Resh, and Asaf Algawi for co-authoring the joint publications in- cluded in this thesis. I want to thank the COMAS Graduate School, which partially funded this research. Furthermore, I would like to thank the University of Jyväskylä staff for being so responsive and helpful during my years as a Ph.D. student. I wish to thank my beloved wife, Anat Anatey, without whom this thesis would not have been possible, for her continuous moral support, co-authoring, and help. Finally, I want to thank my parents, Anat and Eyal, for their love and sup- port throughout my entire life, not least during my work on this thesis. LIST OF FIGURES FIGURE 1 The Common Architecture of Our Method .............................. 22 FIGURE 2 The Transition from Guest to Host and Vice Versa.................... 28 FIGURE 3 Thin Hypervisor ................................................................... 29 FIGURE 4 Thin Hypervisor Security – Cache Eviction ............................. 36 FIGURE 5 Address Translation in the Guest and the Hypervisor............... 36 FIGURE 6 Last Level Cache Organisation Example.................................. 36 FIGURE 7 Thin Hypervisor Memory Modifications Transparency............. 38 FIGURE 8 Thin Hypervisor Performance – PCMark ................................ 40 FIGURE 9 Thin Hypervisor Performance – PassMark .............................. 40 FIGURE 10 Thin Hypervisor Performance – Novabench ............................ 41 FIGURE 11 Memory Forensics – Performance Degradation Due to Mem- ory Acquisition..................................................................... 44 FIGURE 12 Malware Analysis – NtOpenFile 3rd Parameter Information Hierarchy in an x86 Windows System ..................................... 48 FIGURE 13 Malware Analysis – Sandbox Configuration File Example for Windows 7 x86 .................................................................... 48 FIGURE 14 Malware Analysis – Windows Memory Layout After Writing to SYSENTER_EIP msr .......................................................... 50 FIGURE 15 Malware Analysis – Overhead Performance Impact ................. 53 FIGURE 16 Unauthorised Execution Prevention – Scanning of a User Space Executable Image.................................................................. 55 FIGURE 17 Unauthorised Execution Prevention – Page Verification Over- head .................................................................................... 57 FIGURE 18 Unauthorised Execution Prevention – Overall Performance Over- head .................................................................................... 58 FIGURE 19 Preventing Reverse Engineering of Software – Virtual Address Space Layouts of the Hypervisor and the Guest during Pro- tected Function Execution...................................................... 60 FIGURE 20 Preventing Reverse Engineering of Software – First Experi- ment Performance Results ..................................................... 64 FIGURE 21 Preventing Reverse Engineering of Software – Second Experi- ment Performance Results 1................................................... 65 FIGURE 22 Preventing Reverse Engineering of Software – Second Experi- ment Performance Results 2................................................... 65 FIGURE 23 Preventing Reverse Engineering of Software – Third Test Per- formance Results 1 ................................................................ 66 LIST OF TABLES TABLE 1 Preventing Reverse Engineering of Software – Third Test Per- formance Results 2 ................................................................ 66 TABLE 2 Preventing Reverse Engineering of Software – Third Test Per- formance Results 3 ................................................................ 67 LIST OF ALGORITHMS ALGORITHM 1 BIOS Memory Allocation and the Hypervisor’s Memory Map Interception Functions ............................................ 32 ALGORITHM 2 Thin Hypervisor Initialisation ........................................