The Redirect-On-Write File System and Characterization of I/O Overheads in a Virtualized Platform
Total Page：16
File Type：pdf, Size：1020Kb
PROVISIONING WIDE-AREA VIRTUAL ENVIRONMENTS THROUGH I/O INTERPOSITION: THE REDIRECT-ON-WRITE FILE SYSTEM AND CHARACTERIZATION OF I/O OVERHEADS IN A VIRTUALIZED PLATFORM By VINEET CHADHA A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2008 1 °c 2008 Vineet Chadha 2 I dedicate this thesis to my parents 3 ACKNOWLEDGMENTS I would like to thank my advisor Dr. Renato Figueiredo for all the support he has provided me during last six years. He has been taking around the maze of systems research and shown me the right way whenever I felt lost. It has been a privilege to work with Dr. Figueiredo whose calmness, humble and polite demeanor is the one i would like to carry and apply further in my career. Thanks to Dr. Jose Fortes who provided me opportunity to work at Advanced Computing and Information System (ACIS) laboratory. He gave me encouragement and support whenever things were down. I would like to thank Dr. Oscar Boykin for serving in my committee and for all those fruitful discussions on Research, Linux, healthy food and Running. His passion to achieve perfection in every endeavors of life often eggs me to do better. Thanks to Dr. Alan George and Dr. Joseph Wilson for serving in my PhD program committee and motivating me through their courses and research work. I would like to thank my mentor, Ramesh Illikkal and manager, Donald Newell at Intel Corporation for the faith they have shown in me and egged me to work hard. It has been a privilege to work with Ramesh who taught me importance of teamwork, failure and success. Thanks to Dr. Padmashree Apparao and Dr. Ravishankar Iyer for guidance and encouragement to achieve my goals. Thanks is also due to Dr. Ivan krsul and Dr. Suma Adabala for guiding me not only during the development and research of In-VIGO project but also often sharing thoughts on a PhD program and expectations. Thanks is due to all the colleagues here at ACIS which made work environment fun to work in. I would like to thank Andrea and Mauricio for providing excellent research facilities and resources. Thanks to my officemates Arijit and Girish for all fruitful discussions. Thanks is due to Cathy for maintaining cordial environment in ACIS lab and extending support to me as a good friend whenever need arises. 4 Thanks further to rest of the team members in Many core architecture group at Intel Corporation. I am thankful to National Science Foundation and Intel Corporation for providing me financial assistance during my PhD program. Thanks is due to Department of Electrical Engineering and the Department of Computer Information Science and Engineering for maintaining all the paperwork related to my appointment, travel and PhD milestones that made my life easier in Gainesville. This dissertation would not have been possible without unwavering support from friends and family. I would like to thank my good friend, Hemang for support and encouragement he provided me in past six years. This dissertation is dedicated to my parents whose sacrifices can not be described in words. They have been source of strength all these years. The good values and importance of education they have imbibed in me has been a guiding light in my career. Thanks is due to my brother who often guided me the right way to achieve the goals in the life. Finally, I thank almighty for giving me strength to work hard. 5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................. 4 LIST OF TABLES ..................................... 9 LIST OF FIGURES .................................... 10 ABSTRACT ........................................ 13 CHAPTER 1 INTRODUCTION .................................. 16 1.1 Introduction ................................... 16 1.1.1 Virtual Network File System I/O Redirection ............. 16 1.1.2 Characterization of I/O Overheads in a Virtualized Environments . 18 1.1.2.1 Simulation ........................... 19 1.1.2.2 I/O Mechanisms ....................... 19 1.2 Dissertation Contributions ........................... 20 1.3 Dissertation Relevance ............................. 22 1.4 Dissertation Overview ............................. 23 1.5 Dissertation Organization ........................... 26 2 I/O VIRTUALIZATION: RELATED TERMS AND TECHNOLOGIES ..... 28 2.1 Introduction ................................... 28 2.2 Virtualization Technologies ........................... 29 2.3 Virtual Machine Architectures ......................... 32 2.3.1 I/O mechanisms in Virtual Machines ................. 32 2.3.2 Virtual Machines and CMP architectures ............... 34 2.4 Grid Computing ................................ 35 2.5 File System Virtualization ........................... 35 2.5.1 Network File System .......................... 36 2.5.2 Grid Virtual File System ........................ 38 3 REDIRECT-ON-WRITE DISTRIBUTED FILE SYSTEM ............ 41 3.1 Introduction ................................... 41 3.1.1 File System Abstraction ........................ 41 3.1.2 Redirect-on-Write File System ..................... 42 3.2 Motivation and Background .......................... 42 3.2.1 Use-Case Scenario: File System Sessions for Grid Computing .... 44 3.2.2 Use-Case Scenario: NFS-Mounted Virtual Machine Images and O/S File Systems ............................... 44 6 3.2.3 Use-Case Scenario: Fault Tolerant Distributed Computing with Virtual Machines ................................. 45 3.3 ROW-FS Architecture ............................. 47 3.3.1 Hash Table ................................ 48 3.3.2 Bitmap .................................. 49 3.4 ROW-FS Implementation ........................... 51 3.4.1 MOUNT ................................. 51 3.4.2 LOOKUP ................................ 53 3.4.3 GETATTR/SETATTR ......................... 54 3.4.4 READ .................................. 54 3.4.5 WRITE ................................. 55 3.4.6 READDIR ................................ 55 3.4.7 REMOVE/RMDIR/RENAME ..................... 58 3.4.8 LINK/READLINK ........................... 58 3.4.9 SYMLINK ................................ 60 3.4.10 CREATE/MKDIR ........................... 60 3.4.11 STATFS ................................. 60 3.5 Experimental Results .............................. 60 3.5.1 Microbenchmark ............................. 61 3.5.2 Application Benchmark ......................... 63 3.5.3 Virtual Machine Instantiation ..................... 66 3.5.4 File System Comparison ........................ 67 3.6 Related Work .................................. 68 3.7 Conclusion .................................... 68 4 PROVISIONING OF VIRTUAL ENVIRONMENTS FOR WIDE AREA DESKTOP GRIDS ......................................... 70 4.1 Introduction ................................... 70 4.2 Data Provisioning Architecture ........................ 71 4.3 ROW-FS Consistency and Replication Approach ............... 78 4.3.1 ROW-FS Consistency in Image Provisioning ............. 79 4.3.2 ROW-FS Replication in Image Provisioning .............. 80 4.4 Security Implications .............................. 81 4.5 Experiments and Results ............................ 84 4.5.1 Proxy VM Resource Consumption ................... 84 4.5.2 RPC Call Profile ............................ 86 4.5.3 Data Transfer Size ............................ 87 4.5.4 Wide-area Experiment ......................... 87 4.5.5 Distributed Hash Table State Evaluation and Analysis ........ 88 4.6 Related Work .................................. 88 4.7 Conclusion .................................... 90 7 5 I/O WORKLOAD PERFORMANCE CHARACTERIZATION .......... 91 5.1 Introduction ................................... 92 5.2 Motivation and Background .......................... 93 5.2.1 Full System Simulator .......................... 94 5.2.2 I/O Virtualization in Xen ........................ 94 5.3 Analysis Methodology ............................. 96 5.3.1 Full System Simulation: Xen VMM as Workload ........... 96 5.3.2 Instruction Trace ............................ 97 5.3.3 Symbol Annotation ........................... 98 5.3.4 Performance Statistics ......................... 98 5.3.5 Environmental Setup for Virtualized Workload ............ 100 5.4 Experiments and Simulation Results ..................... 102 5.4.1 Life Cycle of an I/O packet ....................... 102 5.4.1.1 Unprivileged Domain ..................... 103 5.4.1.2 Grant Table Mechanism ................... 105 5.4.1.3 Timer Interrupts ....................... 105 5.4.1.4 Privileged Domain ...................... 106 5.4.2 Cache and TLB Characteristics .................... 108 5.5 Cache and TLB Scaling ............................ 110 5.6 Related Work .................................. 114 5.7 Conclusion .................................... 115 6 HARDWARE SUPPORT FOR I/O WORKLOADS: AN ANALYSIS ...... 116 6.1 Introduction ................................... 116 6.2 Translation Lookaside Buffer .......................... 117 6.2.1 Introduction ............................... 117 6.2.2 TLB Invalidation in Multiprocessors .................. 118 6.3 Interprocessor Interrupts ............................ 120 6.4 Grant Table Mechanism: I/O Analysis .................... 121 6.5 Experiments and Results ............................ 123 6.5.1 Grant Table Performance ........................ 123 6.5.2 Hypervisor Global Bit .......................... 124 6.5.3 TLB Coherence Evaluation ....................... 125 6.6 Related Work .................................