THÈSE DE DOCTORAT DE CENTRALESUPELEC, RENNES COMUE UNIVERSITÉ BRETAGNE LOIRE ÉCOLE DOCTORALE N° 601 Mathématiques et Sciences et Technologies de l'Information et de la Communication Spécialité : Informatique Par Ronny Chevalier Detecting and Surviving Intrusions Exploring New Host-Based Intrusion Detection, Recovery, and Response Approaches Thèse présentée et soutenue à Rennes, le 17 décembre 2019 Unité de recherche : IRISA Thèse N° : 2019CSUP0003 Rapporteurs avant soutenance : Joaquin Garcia-Alfaro Professeur, Telecom SudParis Herbert Bos Full Professor, Vrije Universiteit Amsterdam Composition du Jury : Présidente : Laurence Pierre Professeur, Université Grenoble Alpes Examinateurs : Joaquin Garcia-Alfaro Professeur, Telecom SudParis Herbert Bos Full Professor, Vrije Universiteit Amsterdam Karine Heydemann Maître de conférences, Université Pierre et Marie Curie Guillaume Hiet Maître de conférences, CentraleSupélec Rennes David Plaquin Senior Research Scientist, HP Dir. de thèse : Ludovic Mé Advanced Research Position, Inria ABSTRACT Computing platforms, such as embedded systems or laptops, are built with layers of preventive security mechanisms to help reduce the likelihood of attackers successfully compromising them. Nevertheless, given time and despite decades of improvements in preventive security, intrusions still happen. Therefore, systems should expect intrusions to occur, thus they should be built to detect and to survive them. Systems are monitored with intrusion detection solutions, but their ability to survive them is limited. State-of-the-art approaches from in- dustry or academia either involve manual procedures, loss of availabil- ity, coarse-grained responses, or non-negligible performance overhead. Moreover, low-level components, such as the BIOS, are increasingly targeted by sophisticated attackers to implant stealthy and resilient malware. State-of-the-art solutions, however, mainly focus on boot time security, leaving the most privileged part of the BIOS—known as the System Management Mode(SMM)—a prime target. The introduction of new solutions raises various challenges such as the security of the monitor, its ability to gather information about its target, the detection models, responding to intrusions and maintaining the availability of the system despite the presence of an adversary. Our contribution is two-fold: • At theOS-level, we introduce an intrusion survivability approach aimed at commodity OSs. We combine intrusion recovery and The initial idea behind this contribution was presented as a fine-grained cost-sensitive intrusion response to leverage asafe short-paper at RESSI’18 then the degraded mode when an intrusion is detected. Such a degraded final work was published at mode prevents attackers to reinfect the system or to achieve their ACSAC’19. goals if they managed to reinfect it. It maintains the availability of core functions while waiting for patches to be deployed. • At the BIOS level, we introduce an event-based and co-processor- based behavior monitoring approach to detect intrusions target- ing the SMM on x86 platforms. We isolate the monitor using This contribution has been published at ACSAC’17. a co-processor to ensure its security and we bridge the seman- tic gap resulting from it by using a dedicated communication channel. This channel is used to send relevant information about the SMM code behavior that we compare with the model of its expected behavior—using invariants of its control-flow and relevant CPU registers. Keywords: Information Security, Intrusion Detection, Intrusion Re- sponse, Intrusion Recovery, Intrusion Survivability iii PUBLICATIONS This thesis is based on previously published papers written jointly with several collabo- rators: international conference papers • Ronny Chevalier, David Plaquin, Chris Dalton, and Guillaume Hiet. “Survivor: A Fine-Grained Intrusion Response and Recovery Approach for Commodity Op- erating Systems”. In: Proceedings of the 35th Annual Computer Security Applications Conference. ACSAC’19. ACM, Dec. 2019. doi: 10.1145/3359789.3359792. • Ronny Chevalier, Maugan Villatel, David Plaquin, and Guillaume Hiet. “Co- processor-based Behavior Monitoring: Application to the Detection of Attacks Against the System Management Mode”. In: Proceedings of the 33rd Annual Computer Security Applications Conference. ACSAC’17. ACM, Dec. 2017, pp. 399– 411. doi: 10.1145/3134600.3134622. national conference papers • Ronny Chevalier, David Plaquin, and Guillaume Hiet. “Intrusion Survivabil- ity for Commodity Operating Systems and Services: A Work in Progress”. In: Rendez-vous de la Recherche et de l’Enseignement de la Sécurité des Systèmes d’Information. RESSI’18. May 2018. In addition, during the three years of this Ph.D. I was employed by HP, patent applications related to the work, ideas, or solutions presented in this document were filed: patent applications • Ronny Chevalier, David Plaquin, Guillaume Hiet, and Adrian Baldwin. “Miti- gating Actions”. Pat. req. Hewlett-Packard Development Company, L.P. May 2018. • Ronny Chevalier, David Plaquin, Maugan Villatel, and Guillaume Hiet. “Intru- sion Detection Systems”. Pat. req. Hewlett-Packard Development Company, L.P. June 2017. v • Ronny Chevalier, David Plaquin, Maugan Villatel, and Guillaume Hiet. “Monitor- ing Control-Flow Integrity”. Pat. req. Hewlett-Packard Development Company, L.P. June 2017. Finally, during these years I also contributed to another related area, but the following publication is not discussed in this thesis: international conference paper • Ronny Chevalier, Stefano Cristalli, Christophe Hauser, Yan Shoshitaishvili, Ruoyu Wang, Christopher Kruegel, Giovanni Vigna, Danilo Bruschi, and Andrea Lanzi. “BootKeeper: Validating Software Integrity Properties on Boot Firmware Images”. In: Proceedings of the 9th ACM Conference on Data and Application Security and Privacy. CODASPY’19. ACM, Mar. 2019, pp. 315–325. doi: 10.1145/3292006. 3300026. vi ACKNOWLEDGMENTS–REMERCIEMENTS No one does everything alone. Many people contributed—sometimes even without knowing it—to this research and dissertation either intellectually, financially, logis- tically, or personally. This is my attempt at acknowledging their help, interest, and contributions over the years. First and foremost, I would like to thank Herbert Bos, Joaquin Garcia-Alfaro, Karine Heydemann, and Laurence Pierre for taking an interest in my work and for accepting to be members of the jury. Especially Herbert and Joaquin for their detailed review of this manuscript. I was fortunate to be advised by Boris Balacheff, Guillaume Hiet, Ludovic Mé, and David Plaquin. They all shared different responsibilities and duties during this work, but they all provided me with their expertise, helpful criticism, and time. I want to thank in particular Guillaume and David whose advice, comments, and discussions helped shape this dissertation and my research in many respects. During these three years I was also part of two teams: the CIDRE team at Centrale- Supélec and the Security Lab at HP. I want to thank them for giving me an academic and industry perspective on research. I would like to thank the members of CIDRE for the scientific and technical discussions that I had with them over the years, butalso all the "team building" we had at lUnchTime with the PhD students and interns. So thanks to all of them, especially to David Lanoë with his unrelenting force when hitting the cue ball, Pierre Graux with his nice collection of little orange men, Cédric Herzog with his love for Germany, Benoît Fournier who likes to keep a log of what we say, and Aïmad Berady le malicieux. I also want to thank HP and especially the Security Lab at Bristol. Working with them gave me an insight at what it is like to work with a competent industry research lab. In particular, I would like to thank Philippa Bayley and Boris who worked hard to make sure that I could work at HP for my PhD. I would also like to acknowledge the expertise and time that Chris Dalton and Maugan Villatel provided me over the years; I was fortunate to have them as co-authors on some of my papers. I would also like to thank Vali Ali, Pierre Belgarric, Rick Bramley, Carey Huscroft, Jeff Jeansonne, and Thalia Laing for their feedback and technical discussions on my work. I am also grateful to Daniel Ellam, Jonathan Griffin, and Stuart Lees for their help in setting up and running some experiments in their malware lab, and Josh Schiffman for giving me opportunities to present my work at HP. I would also like to thank François Bourdon and Laurent Jeanpierre. They were teachers of mine respectively in operating system and computer architecture during the first two years of my higher education. Both motivated me—probably without vii knowing it—to think about pursuing a career in research. François is also the reason why I went to Rennes for my studies. He told me that there was a research team in Rennes working on computer security, and he pointed me towards someone called Ludovic Mé. Unaware at the time that four years later I would be doing a PhD in this team and with Ludovic as my doctoral advisor. I would also like to thank Jérémy and Martin who went from classmates, to friends, to best men at my wedding, and who also happened to follow a similar career path as PhD students. Je vais terminer par remercier, en français, ma famille. En particulier, mes parents, qui m’ont toujours soutenu même s’ils ne réalisaient probablement pas que tout ce temps passé sur un ordinateur à l’époque allait être utile un jour. Puis, Léni, sans qui je ne me serais problablement jamais intéressé à l’informatique à l’origine. Enfin, Agathe, qui a relu le résumé français de ce manuscrit, et qui a surtout accepté de faire partie de ma vie. viii CONTENTS abstract iii publications v acknowledgments – remerciements vii acronyms xv i prologue 1 introduction3 1.1 Problem Statement . 3 1.1.1 Preventive Security is not Sufficient . 4 1.1.2 Commodity OSs Can Detect but Cannot Survive Intrusions . 5 1.1.3 Low-Level Components Increasingly Targeted . 6 1.2 Thesis . 7 1.3 Evaluation Approach . 8 1.4 Outline . 8 2 background: from x86 power-on to login prompt9 2.1 BIOS and UEFI-Compliant Boot Firmware .