Agentless Cloud-wide Monitoring of Virtual Disk State Wolfgang Richter CMU-CS-15-138 October 2015 School of Computer Science Computer Science Department Carnegie Mellon University Pittsburgh, PA Thesis Committee Mahadev Satyanarayanan, Chair David G. Andersen Gregory R. Ganger Vasanth Bala (Google) Canturk Isci (IBM Research) Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Copyright © 2015 Wolfgang Richter This research was sponsored by the National Science Foundation under grant numbers CNS-0614679, CNS-0833882, IIS-1065336, DGE-0750271, and DGE-1252522, the Defense Advanced Research Projects Agency under grant number FA8721-5-C-0003, Intel ISTC-CC, IBM, and Vodafone. The views and conclu- sions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government oranyother entity. Keywords: Agentless, Agentless Monitoring, Agentless Cloud Monitoring, Cloud, Cloud Com- puting, Cloud Monitoring, Deduplication, Deduplicated Snapshotting, Distributed Streaming Virtual Machine Introspection, File Deduplication, File Snapshotting, File Monitoring, Incremental Hash- ing, Introspection, Optimistic File Snapshotting, Retrospection, Searchable Backup, Snapshotting, Virtual Disk, Virtual Storage, Virtual Machine, VM, Virtual Machine Introspection, VMI Abstract This dissertation proposes a fundamentally different way of monitoring persistent storage. It intro- duces a monitoring platform based on the modern reality of software defined storage which enables the decoupling of policy from mechanism. The proposed platform is both agentless—meaning it op- erates external to and independent of the entities it monitors—and scalable—meaning it is designed to address many systems at once with a mixture of operating systems and applications. Concretely, this dissertation focuses on virtualized clouds, but the proposed monitoring platform generalizes to any form of persistent storage. The core mechanism this dissertation introduces is called Dis- tributed Streaming Virtual Machine Introspection (DS-VMI), and it leverages two properties of modern clouds: virtualized servers managed by hypervisors enabling efficient introspection, and file-level duplication of data within cloud instances. We explore a new class of agentless monitoring applications via three interfaces with two different consistency models: cloud-inotify (strong consistency), /cloud (eventual consistency), and /cloud-history (strong consistency). cloud- inotify is a publish-subscribe interface to cloud-wide file-level updates and it supports event-based monitoring applications. /cloud is designed to support batch-based and legacy monitoring applica- tions by providing a file system interface to cloud-wide file-level state. /cloud-history is designed to support efficient search and management of historic virtual disk state. It leverages new fast-to- access archival storage systems, and achieves tractable indexing of file-level history via whole-file deduplication using a novel application of an incremental hashing construction. I dedicate this work to those that have helped me make my dreams become reality. First, I must thank my maternal grandparents who supported me academically and my, at times, expensive explorations with computers and electronics. Gratitude alone is not nearly enough. I will never forget your legacy. Second, I must thank my paternal grandparents for showing me how people can persevere through the worst of times the world has ever seen and thrive. Third, for my parents who fostered a home filled with the digital wonders of the modern world, and my siblings for always being there for me. Fourth, to Ma and Baba for support and guidance through the fog of stress. Finally, and most importantly, to Debjani Biswas my companion for life, my wife. Thanks for putting up with my long hours, wacky ideas, and going on adventures with me around the world. Acknowledgments The PhD process is a herculean effort, of which I am only one small part. Along the way I have been aided by the smartest people at the top of the field of Computer Science. My success is attributable to the great opportunities presented to me by the academic environment fostered at Carnegie Mellon University’s School of Computer Science. I have been honored with public support through the NSF, and must thank the American people for supporting scientific efforts across the country. Of course, my interactions with many people along the way have shaped me, formed my research focus, and honed my skills. Here I thank many by name, but there are many more than I could possibly name. Thank you to everyone that interacted with me during my tenure as a graduate student. We all truly stand on the shoulders of giants. Of course, a large part of my success is due to excellent guidance and tutelage under my advi- sor Mahadev Satyanarayanan. Over the many years, our group administrative assistants including Tracy Farbacher, Angela Miller, and Chase Klingensmith always made my day better and ensured academic life was smooth. Deborah Cavlovich is amazing taking away every possible headache that arose during graduate school. Without her, scheduling classes, staying on top of PhD administrative overhead, and practically anything else would have been a nightmare. Catherine Copetas is a friend and helper to all students at CMU, she enriched my life by offering me many opportunities to meet new and interesting visitors. Karen Lindenfelser and Joan Digney always helped organize events via the SDI and the PDL. Their efforts made our lives less stressful, and more fun. Of course, conver- sations and nights well spent with friends led to new ideas and filled my life with fun. Thank you especially Yoshihisa Abe, Leman Akoglu, Brandon Amos, Sivaraman Balakrishnan, Field Cady, Zhuo Chen, Jim Cipar, Bhavana Dalvi, Leigh-Ann DeLyser, Ruta Desai, Bin Fan, Kiryong Ha, Wenlu Hu, Shiva Kaul, Elie Krevat, Anvesh Komuravelli, Jayant Krishnamurthy, Meghana Kshirsagar, Hyeon- taek Lim, Michelle Mazurek, Brendan Meeder, Iulian Moraru, Richard Peng, Amar Phanishayee, Kai Ren, Mehdi Samadi, Raja Sambasivan, Vyas Sekar, Vivek Seshadri, Jiri Simsa, Wittawat Tantisiriroj, Ekaterina Taralova, Alexey Tumanov, Vijay Vasudevan, Kevin Waugh, Gabriel Weisz, Lin Xiao, and Erik Zawadzki. For welcoming me to CMU, thank you Matthew Delaney, Jason Calaiaro, and Michael Mackin. Thank you all for the wonderful memories, they will stay with me for life. Last, but not least, I must thank a lot of my mentors and close research peers. This includes Glenn Ammons, Hrishikesh Amur, David Andersen, Vasanth Bala, Nilton Bila, Sastry Duri, Canturk Isci, Michael Kaminsky, Todd Mummert, Padmanabhan Pillai, Darrell Reimer, and Srinivas Seshan. The research scientists in my group Benjamin Gilbert, Adam Goode, and Jan Harkes have each vii viii ACKNOWLEDGMENTS served as mentors and an inspiration as I developed my systems research skills. Thanks for keeping me technically honest, humble, and holding me to the highest of standards. Each and every one of you has a hand in my future career and success. Contents Contents ix List of Figures xii List of Tables xv 1 Introduction 1 1.1 Problem Statement and Scope .............................. 3 1.1.1 Bridging the Semantic and Temporal Gaps ................... 4 1.1.2 Bounding Overhead ............................... 4 1.1.3 Generality ..................................... 4 1.1.4 Storing and Indexing Historic Changes ..................... 5 1.2 Thesis Statement ...................................... 5 1.3 Contributions ....................................... 6 2 Distributed Streaming Virtual Machine Introspection (DS-VMI) 9 2.1 Overview of the DS-VMI Prototype ........................... 10 2.2 Hypervisor, Kernel, and File System Requirements ................... 12 2.2.1 Hypervisor Virtual Storage Hooks ....................... 12 2.2.2 Guest Kernel Invariants ............................. 14 2.2.3 File System Invariants .............................. 16 2.2.4 Correctness of DS-VMI Relative to Snapshotting . 17 2.3 Crawling Initial Virtual Disk State ............................ 18 2.3.1 Impact on Virtual Image Library Operations . 20 2.3.2 Crawling a Virtual Disk ............................. 23 2.3.3 An Example Journaling File System: ext4 ................... 24 2.3.4 An Example Closed Source File System: NTFS . 24 2.3.5 An Example Non-Journaling File System: FAT32 . 27 2.4 Asynchronous Queuing of VM Writes .......................... 28 2.5 Introspecting Live Virtual Disk Writes .......................... 29 2.6 Live Attachment and Detachment ............................ 30 ix x CONTENTS 2.6.1 Live Crawling and Attaching .......................... 31 2.6.2 Detaching Introspection ............................. 32 2.7 Integration with Existing Clouds ............................. 32 2.7.1 Designing an API within OpenStack ...................... 33 2.8 Evaluating Overall Overhead ............................... 34 2.8.1 Experimental Setup ................................ 34 2.8.2 DS-VMI Tunables ................................. 35 2.8.3 Light-rate Small Writes: Modified Andrew Benchmark . 36 2.8.4 Clustered Large Writes: Installing Software . 37 2.8.5 Moderate-rate Small Writes: PostMark ..................... 37 2.8.6 Reducing Memory Footprint: Lazily Loading Metadata . 39 2.8.7 Dropping Writes ................................. 40 2.9 Limitations of DS-VMI .................................