No Hypervisor Is an Island: System-wide Isolation Guarantees for Low Level Code OLIVER SCHWARZ Doctoral Thesis Stockholm, Sweden 2016 KTH TRITA-CSC-A 2016:22 School of Computer Science ISSN 1653-5723 and Communication ISRN-KTH/CSC/A--16/22-SE SE-100 44 Stockholm ISBN 978-91-7729-104-6 SWEDEN Akademisk avhandling som med tillstånd av Kungl Tekniska högskolan framläg- ges till offentlig granskning för avläggande av teknologie doktorsexamen i datalogi måndagen den 10 oktober 2016 klockan 14.00 i F3, Kungl Tekniska högskolan, Lindstedtsvägen 26, Stockholm. SICS Swedish ICT SICS Dissertation Series 75 ISSN 1101-1335 Cover picture: The photograph that serves as background of the cover picture is courtesy of Lukáš Poláček. © Oliver Schwarz, September 8, 2016 Tryck: Universitetsservice US AB iii Abstract The times when malware was mostly written by curious teenagers are long gone. Nowadays, threats come from criminals, competitors, and gov- ernment agencies. Some of them are very skilled and very targeted in their attacks. At the same time, our devices – for instance mobile phones and TVs – have become more complex, connected, and open for the execution of third-party software. Operating systems should separate untrusted software from confidential data and critical services. But their vulnerabilities often allow malware to break the separation and isolation they are designed to pro- vide. To strengthen protection of select assets, security research has started to create complementary machinery such as security hypervisors and separa- tion kernels, whose sole task is separation and isolation. The reduced size of these solutions allows for thorough inspection, both manual and automated. In some cases, formal methods are applied to create mathematical proofs on the security of these systems. The actual isolation solutions themselves are carefully analyzed and in- cluded software is often even verified on binary level. The role of other soft- ware and hardware for the overall system security has received less attention so far. The subject of this thesis is to shed light on these aspects, mainly on (i) unprivileged third-party code and its ability to influence security, (ii) peripheral devices with direct access to memory, and (iii) boot code and how we can selectively enable and disable isolation services without compromising security. The six papers included in this thesis are both design and verification oriented, however, with an emphasis on the analysis of instruction set archi- tectures. With the help of a theorem prover, we implemented various types of machinery for the automated information flow analysis of several processor ar- chitectures. We used these tools to make explicit which registers arbitrary and unprivileged software on ARM or MIPS platforms can access. The analysis is guaranteed to be both sound and accurate. To the best of our knowledge, we were the first to publish an automated analysis and verification of information flow properties for commodity instruction set architectures. iv Sammanfattning Förr skrevs skadlig mjukvara mest av nyfikna tonåringar. Idag är våra datorer under ständig hot från statliga organisationer, kriminella grupper, och kanske till och med våra affärskonkurrenter. Vissa besitter stor kompetens och kan utföra fokuserade attacker. Samtidigt har tekniken runtomkring oss (såsom mobiltelefoner och tv-apparater) blivit mer komplex, uppkopplad och öppen för att exekvera mjukvara från tredje part. Operativsystem borde egentligen isolera känslig data och kritiska tjänster från mjukvara som inte är trovärdig. Men deras sårbarheter gör det oftast möjligt för skadlig mjukvara att ta sig förbi operativsystemens säkerhetsme- kanismer. Detta har lett till utveckling av kompletterande verktyg vars enda funktion är att förbättra isolering av utvalda känsliga resurser. Speciella vir- tualiseringsmjukvaror och separationskärnor är exempel på sådana verktyg. Eftersom sådana lösningar kan utvecklas med relativt liten källkod, är det möjligt att analysera dem noggrant, både manuellt och automatiskt. I någ- ra fall används formella metoder för att generera matematiska bevis på att systemet är säkert. Själva isoleringsmjukvaran är oftast utförligt verifierad, ibland till och med på assemblernivå. Dock så har andra komponenters påverkan på systemets sä- kerhet hittills fått mindre uppmärksamhet, både när det gäller hårdvara och annan mjukvara. Den här avhandlingen försöker belysa dessa aspekter, hu- vudsakligen (i) oprivilegierad kod från tredje part och hur den kan påverka säkerheten, (ii) periferienheter med direkt tillgång till minnet och (iii) start- koden, samt hur man kan aktivera och deaktivera isolationstjänster på ett säkert sätt utan att starta om systemet. Avhandlingen är baserad på sex tidigare publikationer som handlar om både design- och verifikationsaspekter, men mest om säkerhetsanalys av in- struktionsuppsättningar. Baserat på en teorembevisare har vi utvecklat olika verktyg för den automatiska informationsflödesanalysen av processorer. Vi har använt dessa verktyg för att tydliggöra vilka register oprivilegierad mjukvara har tillgång till på ARM- och MIPS-maskiner. Denna analys är garanterad att vara både korrekt och precis. Så vitt vi vet är vi de första som har publicerat en lösning för automatisk analys och bevis av informationsflödesegenskaper i standardinstruktionsuppsättningar. v Acknowledgements No person is an island. Especially no PhD student. In my case, they even gave me two lands that took care of me: the SICS-land and the KTH-land. Both came with one advisor each and – maybe without being aware of it – the two complemented each other pretty well. My industrial advisor Christian Gehrmann understood to accompany my devel- opment with a supervision ever-aligned with the needs of my respective state. In the beginning he provided much initial guiding, followed by an increasing amount of trust, freedom, and responsibility – always on the right level, always lagom mycket to empower me. It is not granted for industrial PhD students to find that degree of freedom and time for research and studies that I received from Christian. He always cared about my progress and expressed his confidence in me. Thank you very much, Christian! My gratitude also goes to my academic advisor Mads Dam. He always found time and patience to discuss technical details or writing matters. And this is still an understatement. As for technical details, it is more precise to say that Mads taught me to never stop discussing them until I really understand their gist. And as for writing, he went through uncountable iterations of proofreading with me. As much as I was striving towards his level of writing, I did not even come close and he still remains a role model in that respect. I also want to thank Mads for the freedom he granted me in my research questions and his confidence in the paths I have chosen towards their solutions. Also, I want to thank Mads for creating a truly inspirational research environ- ment. The PROSPER group at KTH was always a place to discuss, receive help, and get inspired. Working together with that group really was a pleasure. I want to thank their members, both my co-authors and the rest, namely Mads, Narges Khakpour, Hamed Nemati, Christoph Baumann, Andreas Lindner, Dilian Gurov, Musard Balliu, Andreas Lundblad, and Roberto Guanciale. Roberto, I am sorry that I do not know how to put this in a less stereotypical way, but you are a true role model, both as a scientist and a person. Thanks go also to my co-authors and project partners at SICS, namely Chris- tian, Viktor Do, Arash Vahidi, Heradon Douglas, and Jonas Haglund. Especially in the early days, you were my tour guides through the – back then for me inscrutable – jungle of low-level hardware and system software matters. Whenever I had a hardware or implementation question, you were the right address to look for the answer. I also want to thank the other members of the SEC lab, Rolf Blom, Rikard Höglund, Marco Tiloca, Nicolae Paladi, Antonis Michalas, Mudassar Aslam, Lud- wig Seitz, our master students, and all our lab members in Lund, that I hopefully will get to know better in the future. Nicolae, thank you for proofreading so many of my papers – even though I use parentheses more often than your beloved dashes. Thank you also for all the interesting scientific discussions in your office. The rub- ber duck, the whiteboard pictures, and that strange little spinning tops created a pleasant and stimulating atmosphere for brainstorming. Marco, thank you for vi sharing your insights about the world, both in scientific and other respects. Mu- dassar, the same holds for you. It was great to share the office with you, share our views on being PhD students, and share our thoughts concerning the big questions of life. Back in KTH-land, they also put me into an office, specifically, into the best office of the entire KTH. Thank you Benjamin Greschbach, Emma Enström, and Guillermo Rodríguez Cano for having kept it such a welcoming place that always encouraged people to drop by. Another place at the TCS-department that I never will forget is its kitchen. It was the home for countless interesting lunch discus- sions. Many people deserve thanks for this, but in particular Gunnar Kreitz, Lukáš Poláček, and Lukáš’ mother. There are many more nice people at TCS. I like to thank them all for a nice working environment. I know these acknowledgements will be the most read part of the thesis and I know you are waiting for your names to be listed here. But I have been around for quite many years, so many people came and went, just too many to list all of you. But be sure that you are in my memories, nonetheless. The same goes for all the nice people at SICS. I cannot list you all, but I really enjoyed your company and help. Thanks to the old NETS lab, to the IT support, the receptionists, the administration, the Swedish teachers, the badminton partners, and all others.