Safe and Automatic Live Update Ph.D. Thesis Cristiano Giuffrida VU University Amsterdam, 2014 This work was funded by European Research Council under ERC Advanced Grant 227874. This work was carried out in the ASCI graduate school. ASCI dissertation series number 300. Copyright © 2014 by Cristiano Giuffrida. ISBN 978-90-5383-072-7 Cover design by Dirk Vogt. Printed by Wöhrmann Print Service. VRIJE UNIVERSITEIT SAFE AND AUTOMATIC LIVE UPDATE ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad Doctor aan de Vrije Universiteit Amsterdam, op gezag van de rector magnificus prof.dr. F.A. van der Duyn Schouten, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de Faculteit der Exacte Wetenschappen op donderdag 10 april 2014 om 13.45 uur in de aula van de universiteit, De Boelelaan 1105 door CRISTIANO GIUFFRIDA geboren te Rome, Italië promotor: prof.dr. A.S. Tanenbaum “All problems in computer science can be solved by another level of indirection.” Butler Lampson, quoting David Wheeler. Acknowledgements “Yesterday I learned that I received a 3-million euro grant from the EU—all for myself [. ]. Might you be interested?”. This is the email that started it all, on the now-distant date of August 15, 2008. I cannot recall many more details from that email, but I do recall that glancing over its first and last sentence was enough for me to conclude that my spam filter was far less aggressive than I had hoped for. I was literally about to discard the message, when the signature at the bottom caught my attention. It read “Andy Tanenbaum”, which initially only reinforced my belief that targeted phishing attacks had grown to become astonishingly common and sophisticated. After a sudden epiphany, I finally decided to keep on reading. That was the beginning of my Ph.D. journey at the Vrije Universiteit Amsterdam. A challenging and rewarding journey which ultimately culminated in this disserta- tion, but which would not have been possible without the help of others. First and foremost, I would like to thank my supervisor, Andy Tanenbaum, for his constant guidance and support during all these years. Not only did he teach me how to do research and write papers, but he helped me develop passion for simple and elegant solutions and showed me that critical thinking, dedication, and perseverance can take you a long way. I am deeply indebted to him for providing me with an excellent topic, while allowing me, at the same time, to independently explore many exciting research directions that have gradually shaped my scientific interests over the years. Next, I would like to express my gratitude to all the members of my Ph.D. thesis committee: Herbert Bos, Cristian Cadar, Bruno Crispo, Orran Krieger, and Liuba Shira. Their valuable comments greatly contributed to improving the quality of this dissertation. I am especially grateful to Orran Krieger, for his extensive and in- sightful comments on the text, and Cristian Cadar, for fruitful early discussions and feedback on the techniques presented in this dissertation. A special mention is in or- der for my internal referees. I am extremely grateful to Herbert Bos, for his constant vii viii ACKNOWLEDGEMENTS support and excellent suggestions throughout my research work, and Bruno Crispo, for strongly encouraging me to expand my research interests into systems security. I would also like to acknowledge the many other people I had the pleasure to work or spend time with at the Vrije Universiteit Amsterdam. First, I would like to thank those who shared their Ph.D. journeys with me, especially Stefano Ortolani, for his friendship and engagement in many joint “not-so-side” research projects, and my P4.69 office mates, Raja Appuswamy, Jorrit Herder, Tomas Hruby, Erik van der Kouwe, David van Moolenbroek, and Dirk Vogt, for creating an inspiring and enjoyable work environment. Next, I would like to thank all the other members of the MINIX 3 team, including Ben Gras, Philip Homburg, Lionel Sambuc, Arun Thomas, and Thomas Veerman, for their ability to challenge my most ambitious ideas and provide very valuable support on short notice. Needless to say, I will never forget the movie nights and the other team events we shared together. I am also grateful to the many excellent students who contributed to our research projects, especially Calin Iorgulescu and Anton Kuijsten, with whom I shared many sleepless nights before a paper deadline. Finally, I would like to thank all the other people from the Computer Systems group, who substantially contributed to making my doctoral years special both professionally and personally, Willem de Bruijn, Albana Gaba, Ana Oprescu, Georgios Portokalidis, Asia Slowinska, and Spyros Voulgaris, in particular. I am also grateful to Divyesh Shah, for inviting me to join the kernel memory team at Google and for providing me with all the necessary support during my in- ternship time in Mountain View, California. I would also like to thank all the other members of the kernel memory team, especially Ying Han and Greg Thelen, for their dedication and support. My experience at Google has further strengthened my interest in operating systems and memory management, while providing much in- spiration for the RCU-based quiescence detection protocol presented in Chapter 7. I would also like to extend my gratitude to the many friends who provided the much needed distraction and support during my Ph.D. journey, including: all my fellow “Djangonians”, especially the pioneers, Albana, Christian, Dirk, Stefano, and Steve, for the unforgettable moments shared together; Raja, for the way-too- many “Lost” evenings; Flavio, for being always there for me in the difficult times. Last but not least, I would like to thank my family for supporting me all these years. My mother, among so many other things, for being a constant source of inspi- ration throughout my life, encouraging me to pursue my passion for research, and getting me interested in computers at a young age—despite her passion for archeol- ogy. My uncle and aunt, for guiding me through important decisions in my life. My cousins, Chiara and Matteo, for joining me on so many adventures. Finally, Laura, for her love, support, and much needed patience throughout this endeavor. Cristiano Giuffrida Amsterdam, The Netherlands, December 2013 Contents Acknowledgements vii Contents ix List of Figures xv List of Tables xvii Publications xix 1 Introduction 1 2 Safe and Automatic Live Update for Operating Systems 9 2.1 Introduction . 10 2.1.1 Contribution . 12 2.2 Background . 12 2.2.1 Safe Update State . 12 2.2.2 State Transfer . 13 2.2.3 Stability of the update process . 15 2.3 Overview . 16 2.3.1 Architecture . 16 2.3.2 Update example . 17 2.3.3 Limitations . 18 2.4 Live Update Support . 19 2.4.1 Programming model . 19 2.4.2 Virtual IPC endpoints . 20 2.4.3 State filters . 20 ix x ACKNOWLEDGEMENTS 2.4.4 Interface filters . 21 2.4.5 Multicomponent updates . 21 2.4.6 Hot rollback . 22 2.5 State Management . 22 2.5.1 State transfer . 23 2.5.2 Metadata instrumentation . 24 2.5.3 Pointer transfer . 25 2.5.4 Transfer strategy . 26 2.5.5 State checking . 27 2.6 Evaluation . 28 2.6.1 Experience . 28 2.6.2 Performance . 31 2.6.3 Service disruption . 34 2.6.4 Memory footprint . 34 2.7 Related work . 35 2.8 Conclusion . 37 2.9 Acknowledgments . 37 3 Enhanced Operating System Security Through Efficient and Fine-grained Address Space Randomization 39 3.1 Introduction . 40 3.1.1 Contributions . 41 3.2 Background . 41 3.2.1 Attacks on code pointers . 41 3.2.2 Attacks on data pointers . 42 3.2.3 Attacks on nonpointer data . 42 3.3 Challenges in OS-level ASR . 42 3.3.1 W⊕X............................. 42 3.3.2 Instrumentation . 43 3.3.3 Run-time constraints . 43 3.3.4 Attack model . 43 3.3.5 Information leakage . 44 3.3.6 Brute forcing . 44 3.4 A design for OS-level ASR . 45 3.5 ASR transformations . 47 3.5.1 Code randomization . 48 3.5.2 Static data randomization . 49 3.5.3 Stack randomization . 51 3.5.4 Dynamic data randomization . 52 3.5.5 Kernel modules randomization . 52 3.6 Live rerandomization . 53 3.6.1 Metadata transformation . 53 3.6.2 The rerandomization process . 54 ACKNOWLEDGEMENTS xi 3.6.3 State migration . 55 3.6.4 Pointer migration . 56 3.7 Evaluation . 57 3.7.1 Performance . 57 3.7.2 Memory usage . 60 3.7.3 Effectiveness . 60 3.8 Related work . 63 3.8.1 Randomization . 63 3.8.2 Operating system defenses . 64 3.8.3 Live rerandomization . 64 3.9 Conclusion . 65 3.10 Acknowledgments . 65 4 Practical Automated Vulnerability Monitoring Using Program State In- variants 67 4.1 Introduction . 68 4.2 Program State Invariants . 69 4.3 Architecture . 70 4.3.1 Static Instrumentation . 71 4.3.2 Indexing pointer casts . 72 4.3.3 Indexing value sets . 72 4.3.4 Memory management instrumentation . 73 4.3.5 Metadata Framework . 73 4.3.6 Dynamic Instrumentation . 74 4.3.7 Run-time Analyzer . 75 4.3.8 State introspection . 76 4.3.9 Invariants analysis . 76 4.3.10 Recording . 77 4.3.11 Reporting . 77 4.3.12 Feedback generation . 78 4.3.13 Debugging . 78 4.4 Memory Errors Detected . 78 4.4.1 Dangling pointers . 78 4.4.2 Off-by-one pointers . 78 4.4.3 Overflows/underflows . 79 4.4.4 Double and invalid frees . 79 4.4.5 Uninitialized reads .