C: 40 C: 40 M: 2 M: 2 Y: 10 Y: 10 K: 0 K: 0 Comparative Assessment of Cloud Compute Services using Run-Time Meta-Data Michael Menzel C: 76 C: 76 Michael Menzel: Comparative Assessment of Cloud Compute Services using Run-Time Meta-Data Services using Run-Time of Cloud Compute Assessment Comparative Michael Menzel: M: 47 M: 47 Y: 30 ISBN 978-3-7375-5175-5 Y: 30 K: 5 K: 5 Comparative Assessment of Cloud Compute Services using Run-Time Meta-Data A Framework for Performance Measurements and Virtual Machine Image Introspections vorgelegt von Dipl. Wirt.-Inf. Michael Menzel geb. in Mainz von der Fakultät IV - Elektrotechnik und Informatik der Technischen Universität Berlin zur Erlangung des akademischen Grades Doktor der Ingenieurwissenschaften - Dr.-Ing. - genehmigte Dissertation Promotionsausschuss: Vorsitzender: Prof. Dr. Uwe Nestmann Gutachter: Prof. Dr. Stefan Tai Gutachter: Prof. Dr. Alexander Mädche Gutachter: Prof. Dr. Odej Kao Tag der wissenschaftlichen Aussprache: 21.04.2015 Berlin 2015 ii Credits As most projects in life, a dissertation stands on the shoulders of giants. Foremost, it is the result of collaborations, ongoing discussions, and inspirations from many ac- quaintances, companions, and friends. A particularly inspiring giant that supervised the present work is Prof. Stefan Tai whom I like to thank for his patient advises, engage- ment, and steady support of this project. Furthermore, I am deeply thankful I had the luck to spend time with the most intriguing colleagues1. The way all of them engage in discussions and strive for the best is remarkable. Companionships between like-minded researchers develop unaffected by distance or time zones. I am thankful to have met Rajiv Ranjan and Shrikumar Venugopal at the University of New South Wales. Rajiv accompanied and contributed to some of my publications in the research community. He has been a valuable counterpart in discussions. The arrangements never allowed me to discuss my research topics with Shrikumar as intensively as I would’ve liked to, but his few hints unfolded as crucial advises. The support I have received from my friends, especially from Sebastian Roth and Jeanette Homenu-Roth & Tobias Roth, will never be forgotten. Last but even more no- tably, I am grateful for my parents Sylvia and Wolfgang, and siblings Julia Konstanze and Laura-Sophia, who never doubted I was able to complete this dissertation. Above all, it is nearly impossible to express how outermost thankful I am for the un- ending support from one special person. Knowing her for so many years, there exists no word to describe how gratified and lucky I must be to share life with such a patient, understanding, and loving person as is my true love, Anna-Karolina Braun. She contin- ually encouraged me to pursue a PhD and to engage in a project which is as demanding and fulfilling as is this dissertation. Thank you! Mannheim, April 2015 Michael Menzel 1In alphabetic order: Alexander Lenk, Bugra Derre, Christian Janiesch, Christian Zirpins, David Bermbach, Erik Wittern, Frank Pallas, Gregory Katsaros, Jens Nimis, Jörn Kuhlenkamp, Markus Klems, Nelly Schuster, Raffael Stein, Robin Fischer, Steffen Müller, Tilmann Kopp, Ulrich Scholten iii Abstract The available amount of meta-data about compute service offerings which proofs reli- able, timely, and comparable is unsatisfactory. For example, the meta-data published by compute service providers regarding performance attributes of their offers is typically restricted to hardware figures and, thus, not necessarily sufficient for comparisons or planning tasks, such as a thorough software system capacity planning. A similar prob- lem of meta-data scarcity affects the reuse of Virtual Machine (VM) images available in repositories from compute service providers. The contents of the VM images are not described by any available meta-data, yet. The present work contributes a framework of compute service assessment and com- parison methods to the research community. The methods enables compute cloud consumers to assess and compare compute services regarding diverse characteristics. As the purpose of the methods is to serve consumers, the general scheme is an ex- ploitation of the client-side remote access to VMs in order to gain meta-data at run- time. Therefore, an archetypical run-time assessment automation model is provided. The information extracted at run-time can furthermore be attached to compute ser- vices as meta-data through a generic and extensible meta-data model. Furthermore, a Multi-Attribute Decision-Making (MADM)-based scoring method is introduced by the framework which enables consumers to compare compute services regarding mul- tiple characteristics with a single score. Besides, a stopping rule approach is able to enforce cost budgets during sequential compute service assessments. Additionally, in a search for a highest scoring compute service the rule uses priorly available meta-data to skip presumably low scoring services. The framework is employed in two specific instantiations to assess compute services in regards of performance and VM image contents. In particular, this work features an instantiation of the framework which assesses compute services using performance measurements. Therefore, a method is presented that incorporates a procedure built upon the framework’s automation model. The method uses the procedure to measure compute services by injecting benchmarking scripts into VMs via remote interfaces from client-side. Aside from the procedure, the method features a definition language for configuring batches of performance measurements with repetitions and scheduled runs. Resulting performance meta-data can be captured with an extension of the frame- work’s meta-data model. For the comparison of performance meta-data gained with the benchmarking method, the framework’s MADM-based method is adapted. The adapted v method enables compute cloud consumers to compare compute services according to a score derived from a custom set of performance attributes. In the quest of containing costs from benchmarking various compute services, the framework’s stopping rule is configured to consider information about compute service hardware figures available beforehand. In regards of VM image repositories of compute services, the proposed framework is instantiated to generate a procedure to introspect the contents of VM im- ages from the client-side. An extension of the framework’s meta-data model is able to capture results from VM image introspections. The feasibility of the methods is confirmed by software prototypes presented in this work. The prototypes, furthermore, serve as a basis to conduct an evaluation of the pro- posed methods and models. Several experiments proof the validity of the approaches. Further examinations in the course of an evaluation address aspects such as the promp- titude of the assessment approaches and the computational complexity to combine mul- tiple meta-data attributes into a score. Besides, the methods are compared to the state of the art to expose advantages and shortcomings. vi Zusammenfassung Die Menge verfügbarer Meta-Daten über Compute Service-Angebote erweist sich hin- sichtlich Zuverlässigkeit, Aktualität und Vergleichbarkeit ungenügend. Zum Beispiel beschränken sich die von Compute Service-Anbietern veröffentlichten Meta-Daten über Leistungsattribute ihrer Angebote typischerweise auf Hardwarekennzahlen und sind da- her nicht verwendbar für Vergleiche oder auch Planungsaufgaben wie bspw. eine Ka- pazitätsplanung für ein Softwaresystem. Ähnlich sind auch Meta-Daten über Abbilder von virtuellen Maschinen (VM) in den Sammlungen der Compute Service-Anbieter spärlich. Die dargelegte Arbeit stellt der Forschergemeinschaft ein Rahmenwerk mit Bewertungs- und Vergleichsmethoden für Compute Services bereit. Die Methoden erlauben Compu- te Cloud-Konsumenten Compute Service hinsichtlich verschiedener Charakteristiken zu bewerten und zu vergleichen. Da die Methoden Cloud-Konsumenten dienen, basiert das generelle Vorgehen auf der Nutzung von Client-seitigen Remote-Verbindungen zu virtuellen Maschinen, um Meta-Daten zur Laufzeit zu gewinnen. Hierfür wird ein ar- chetypisches Automatisierungsmodell bereitgestellt. Die zur Laufzeit extrahierten In- formationen können Compute Services anhand eines generischen und erweiterbaren Meta-Datenmodells zugeordnet werden. Auserdem¨ wird durch das Rahmenwerk basie- rend auf Ansätzen der multi-attributiven Entscheidungsfindung (MADM) eine Metho- de eingeführt, die es Konsumenten ermöglicht Compute Services hinsichtlich mehrerer Charakteristiken anhand eines Scores zu vergleichen. Weiter kann ein Stoppregelansatz Kostenbudgets während sukzessiver Compute Service-Bewertungen durchsetzen. Die Regel schliest¨ bei der Suche nach einem Compute Service mit höchstem Score anhand von vorab verfügbaren Meta-Daten voraussichtlich schlecht abschneidende Services aus. Das Rahmenwerk wird in zwei spezifischen Instanzen eingesetzt, um Compute Services bezüglich ihrer Leistung und Inhalten von VM-Abbildern zu bewerten. Eine Instanz des Rahmenwerks erlaubt Bewertungen von Compute Services anhand derer Leistung. In diesem Zuge wird eine Methode präsentiert, die eine auf dem Automatisierungs- modell des Rahmenwerks basierenden Prozedur einsetzt, um Compute Services mit Benchmarking-Skripten zu messen, die über Remote-Schnittstellen von der Client- seite ausgeführt werden. Neben der Prozedur bietet die Methode eine Definitionss- prache für Stapelverarbeitung sowie die Wiederholung und Terminierung von