Doctor of Philosophy

RICE UNIVERSITY Safe and Secure Subprocess Virtualization in Userspace By Bumj in Im A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE Doctor of Philosophy APPROVED, THESIS COMMITTEE ang chen Nathan Dautenhahn (Aug 12, 2021 19:01 CDT) ang chen (Aug 12, 2021 16:01 CDT) Nathan Dautenhahn Ang Chen Assistant Professor of Computer Science Assistant Professor of Computer Science Dan Wallach (Aug 12, 2021 16:05 CDT) Dan Wallach Professor of Computer Science and of Electrical and Computer Engineering Kaiyuan Yang (Aug 12, 2021 16:32 CDT) Kaiyuan Yang Assistant Professor of Electrical and Computer Engineering HOUSTON, TEXAS August 2021 Safe and Secure Subprocess Virtualization in Userspace Thesis by Bumjin Im Thesis for the Degree of Doctor of Philosophy Department of Computer Science Rice University (Houston, Texas) August, 2021 ABSTRACT Safe and Secure Subprocess Virtualization in Userspace by Bumjin Im Commodity operating systems isolate the application with process boundary, and all the developers develop the applications upon the principle. However, the applications cannot simply trust the process-based isolation. Virtually all the applications link at least one dynamic library on the runtime that the libraries share all the resources in the same process boundary. Unfortunately, the application developers do not fully understand the libraries they are using, and it could even be infeasible for some complex applications. If a single malicious or buggy library is linked to the application, it can breach the entire application due to its process boundary principle. Since the process-based isolation could continue for some time, it could be harder to achieve the least privilege. We propose a new process model, Endokernel, to resolve this issue. Endokernel contains a monitor inside the standard process in the commodity operating system and provides safe isolation between subprocess, maintenance, and the secure interactions between subprocesses. Endokernel also proposes a endoprocess virtualization technique. Utilizing endoprocess virtualization could realize a more ne-grained least privilege principle in the commodity computing environment. We develop Intravirt as the prototype of Endokernel. Intravirt realizes the Endokernel model on Intel CPU and Linux by actively utilizing Intel® Memory Protection Key(MPK) and Control-ow Enforcement Technology(CET) as the core security mechanisms. Since MPK and CET are hardware mechanisms, Intravirt aims to secure and high-performance endoprocess virtualization. We then evaluate the security and the performance of Intravirt by measuring microbenchmarks and the actual applications with several use cases for the secure computing environment. Throughout the research, we verify Endokernel is a feasible, lightweight, applicable, and eective security model. Acknowledgments It was a reckless decision as a mid-aged man to start an advanced academic degree in a foreign country with a foreign language after resigning from a well-paid and recently promoted job. Everyone did not understand this decision indeed, and many people said it is a mistake. However, I started a new life in Houston, Texas, being a student after 13 years, get a master’s degree, publish a conference paper, and nally get a ph.D. degree. This tremendous achievement could be impossible without enormous help and support from many people. Without them, there will be no research achievement, no conference paper, no admission to the university, and I will never be able to dream about this. Professor Dan Wallach guided me to join the ph.D. program at Rice University. Without him, I would never think of applying to Rice University. Instead of rushing me to nish the school work quickly, he gave me enough time to settle down to the new culture. Also, he gave me enormous advice as a father, neighbor, and teacher that helped me so much to carry out the program and to support my family members. Lastly, when I decided to change the advisor, he did not hesitate to allow and support my new decision that the lost momentum of the research was able to grow again. It was the beginning of my 5th year when I decided to join Nathan’s group. I was mid-40, have family, and the background knowledge is dierent from the group’s research projects. Hence, it was a risky gamble for him to admit me as his student. However, he welcomed me without hesitation and supported me in making such a decision. He also understood and waited patiently for my months-long distractive working environment and the slow progress due to the family support in the pandemic and the lack of knowledge. Without professor Nathan Dautenhahn, I would decide to stop the program during my 5th year. I think he certainly thought about admitting me as his rst graduate student in his academic career. Also, he would be anxious about the research after admission. I appreciate his endless patience and the waiting for my research progress. Mr. Hyunjin Choi became my boss about ten years after starting to work at Samsung. Working with him was an auspicious event for me. He tried to make the most rational and practical decision, and he always tried to reduce the unnecessary burden on my work. He always gave me his best advice not only for the project but also the career and personal issues that he was not a simple boss, but a teacher of my life. After a few years working with him, I was frustrated with continuing my career at Samsung and in Korea, his advice was to consider an advanced academic degree abroad and develop a new career there instead of telling me to work with him forever. Ordinary manager will tell his coworker to work together with sweet promises like promotion, but he guided me to a dierent career path to me, and he chose to let me go. He is indeed one of the people who inuenced my life. Fangfei Yang is my lucky elf in this research. At the beginning of the research, I could not code in assembly, no knowledge of low-level code and hardware in detail. The only thing I have was the research idea. His deep knowledge of the low-level operating systems and the hardware and the never-decreasing passion kept the research rolling all the time and injecting even more fascinating ideas into the research. I admire him as a fellow student and appreciate much for his eorts. Without his contribution, the research could stall at any time. Daniel Song joined Rice University 2years earlier than me, working with professor Dan Wallach, and he is a Korean. He gave me enormous help and tips to survive in a foreign country without trouble, and he kept in touch with my family as well, becoming an uncle to my kids. He still gives me even more tips and helps about the graduation and career paths, as well as his mistake stories. He spent a noticeable amount of his time and resources for my family and me that I could start my life in a foreign country without hassles, and my children got an uncle. Lastly, I have to say thank you to my family. Most of all, my wife gave up all the privileges and assets she possessed, and she just followed me that I appreciate her sacrice, and I also feel a deep sorry for her. Her husband, I, was a recognized employee at Samsung, her children enjoyed their school life, and there was no potential trouble that everyone else did not support my decision to go abroad for this program, she supported me from when I started thinking about the ph.D. program in Rice University, and she still struggles to live in a foreign country only with her direct family members. Also, she still makes an endless eort to support my program and overcoming this pandemic. She is the headstone of my life, without a doubt. I can recall clearly the my children’s rst day of school in Houston. They were dropped in unfamiliar schools, could not understand English at all, completely dierent culture, and no friends. But they did not complain about the new schools, and they quickly adapted, fortunately. The pandemic made my kids stuck at home all the time, but they are still not complaining about this, and they are keeping what they need to do. I really appreciate my adorable kids. Contents 1 Introduction 1 1.1 Ideal Solution: Use Safe Languages for Everything . .3 1.2 Straightforward Solution: More Process Separations . .3 1.3 Ecient Solution: Subprocess Isolations . .5 1.4 Problems in Subprocess Isolation . .6 1.5 Endokernel: Safe Subrocess Isolation in Commodity OS . .8 1.6 Contributions . .9 2 Subprocess Isolations and System Call Virtualizations 12 2.1 Subprocess Separation . 12 2.1.1 Language Based Separation . 13 2.1.2 Operating System Based Separation . 17 2.1.3 Hardware Accelerated Separation . 19 2.2 System Call and Signal Virtualization . 26 2.2.1 Linux Security Module . 26 2.2.2 System call Filtering . 27 2.2.3 System call tracing and interposition . 30 3 Threats 33 3.1 Unauthorized memory access . 33 3.2 Unauthorized le access . 35 3.3 Unauthorized system call execution . 35 3.4 Attack on Subprocess Isolation: PKU Pitfall . 36 4 Endokernel Architecture 38 4.1 Assumption . 38 4.2 Requirements . 38 4.3 Mechanisms Gaps and Challenges . 40 4.4 Endoprocess Model . 42 4.5 Design Principle . 44 4.6 Authority Model . 45 4.7 Nested Endokernel Organization . 47 4.7.1 In-Process Policy . 47 4.7.2 Interface . 48 4.8 Separation Facilities: Nested Boxing . 49 4.9 Intel® Memory Protection Key . 51 5 Design and Implementation 52 5.1 Privilege and Memory Virtualization . 52 5.1.1 Virtual Privilege Switch .

Load more