ALGORITHMS AND TECHNIQUES FOR AUTOMATED DEPLOYMENT AND EFFICIENT MANAGEMENT OF LARGE-SCALE DISTRIBUTED DATA ANALYTICS SERVICES By Anirban Bhattacharjee Dissertation Submitted to the Faculty of the Graduate School of Vanderbilt University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY in Computer Science February 29, 2020 Nashville, Tennessee Approved: Aniruddha S. Gokhale, Ph.D. Abhishek Dubey, Ph.D. Douglas C. Schmidt, Ph.D. Gabor Karsai, Ph.D. Hongyang Sun, Ph.D. DEDICATION To my late Grandmother, Bina Roy Chowdhury, infinitely inspirational and To my beloved wife, Malabika, unbelievably encouraging and To my parents, Ashok and Nilanjana Bhattacharjee, amazingly supportive ii ACKNOWLEDGMENTS I express my sincere gratitude to those who have contributed to this thesis and supported me during this fantastic journey. I am grateful to all of those with whom I have had the pleasure to work during these years. First and foremost, I would like to express my sincere thanks to my advisor Dr. Anirud- dha S. Gokhale, for providing me the opportunity to work in the Distributed Object Com- puting (DOC) group at the Vanderbilt Vanderbilt School of Engineering. His thoughtful advice, guidance, mentorship, and unwavering support over the years helped me at various stages of my research. I appreciate his valuable suggestions, comments, and leadership, which encouraged me to learn more every day and to become an independent researcher. His support has been the most worthy experience for me, and I owe him a big thanks once again for being a fantastic advisor and mentor. I want to thank Dr. Abhishek Dubey, Dr. Douglas C. Schmidt, Dr. Gabor Karsai, and Dr. Hongyang Sun, for serving on my dissertation committee. Each of my dissertation committee members has provided extensive professional guidance and taught me a great deal about scientific research. I want to acknowledge the collaboration from Dr. Hongyang Sun in my multiple research works. He helped me remarkably in formulating the research problems in various areas. Thank you, Dr. Hongyang Sun, for your valuable time, co- operation, and generosity, which set this dissertation work possible. I am grateful to Dr. Douglas Fisher for allowing me to join the Vanderbilt University and for mentoring me during the early stage of my Ph.D. program. I would especially like to thank Dr. Xenofon Koutsoukos for his mentorship, skeptical feedbacks, and deep insights at the DDDAS project meetings. I would also like to thank Dr. Zhifeng Yun for his mentorship and leadership during my internship days at ARM Ltd. This work would not have been possible without the financial support of various agen- cies, and I’m grateful for their generous support. This thesis was supported in part by iii NEC Corporation, Kanagawa, Japan, and NSF US Ignite CNS 1531079, AFOSR DDDAS FA9550-18-1-0126, and AFRL/Lockheed Martin’s StreamlinedML program. I appreciate the feedback and insights from our sponsors, Mr. Thomas Damiano of Lockheed Martin and Dr. Takayuki Kuroda of NEC Corporation. My sincere gratitude is reserved for my colleague, Ajay Dev Chhokra, for his collabo- ration and research contributions on multiple projects. His valuable insights and collabo- ration made the last chapter of the dissertation possible. His contribution towards the third chapter, Barista, is also distinguishable, where he helped me to devise the problem and algorithm. I would also like to thank the members of the DOC group, Shashank Shekhar, Yogesh Barve, Shweta Khare, Shunxing Bao, Subhav Pradhan, Prithviraj Patil, Zhuangwei Kang, and Robert Canady for their collaboration, feedback, and encouragement. I would especially like to thank Shashank Shekhar, Yogesh Barve, Shweta Khare, and Zhuangwei Kang for collaborating with me on multiple projects. I also thank Shreyas Ramakrishna for his deep insights and thoughtful ideas, which helped define the last chapter of this disser- tation. Nobody has been more valuable to me in the pursuit of this Ph.D. journey than my family members. I’m hugely thankful to my late grandmother, Bina Roy Chowdhury, with- out whose support the Ph.D. journey would have never begun. I want to acknowledge the patience, support, and encouragement of my parents, Ashok and Nilanjana Bhattacharjee. Most importantly, I would like to express my gratitude to my beloved wife, Malabika, for her continuous support during the ups and downs of my Ph.D. journey. iv TABLE OF CONTENTS Page DEDICATION . ii ACKNOWLEDGMENTS . iii LIST OF TABLES . xiii LIST OF FIGURES . xiv I Introduction . 1 I.1 Emerging Trends . 1 I.2 Key Research Challenges and Solution Needs . 5 I.2.1 Requirement 1: Automation of the ML Development Pipeline . 5 I.2.1.1 Challenge 1: Abstraction of ML Pipeline . 6 I.2.1.2 Challenge 2: Code-generation for ML Model Training and Evaluation . 6 I.2.1.3 Challenge 3: Support for ML Deployment . 7 I.2.2 Requirement 2: Automation of Infrastructure and Application Pro- visioning . 7 I.2.2.1 Challenge 4: Abstraction of Application and Infrastruc- ture details . 8 I.2.2.2 Challenge 5: Infrastructure Code-generation from Abstract Model . 8 I.2.2.3 Challenge 6: Verification of Abstract Deployment Model . 8 I.2.2.4 Challenge 7: Extensibility and Re-usability . 9 I.2.3 Requirement 3: Proactive Resource Management . 9 I.2.3.1 Challenge 8: Workload Variation . 9 I.2.3.2 Challenge 9: Optimal Resource Selection . 10 v I.2.3.3 Challenge 10: Proactive Resource Provisioning . 10 I.2.4 Requirement 4: Interference-aware Strategy for ML Model Update . 10 I.2.4.1 Challenge 11: Heterogeneity-aware Data Management . 10 I.2.4.2 Challenge 12: Resource Interference-awareness . 11 I.3 Organization of the Dissertation . 11 II Erudite: A Lifecycle Management Framework for Machine Learning based Pre- dictive Analytics Applications . 13 II.1 Introduction . 13 II.1.1 Emerging Trends . 13 II.1.2 Challenges and State-of-the-art Solutions . 13 II.1.3 Overview of Technical Contributions . 15 II.1.4 Organization of the Chapter . 17 II.2 Related Work . 17 II.3 Problem Formulation . 20 II.3.1 Motivating Case Study and Key Challenges . 21 II.3.1.1 Deployment Challenges . 22 II.3.1.2 Data Movement and Management Challenges . 22 II.3.1.3 Model Building and Dissemination Challenges . 23 II.3.1.4 Challenges in Determining the Right Hardware Needed . 24 II.3.1.5 Runtime Resource Monitoring Challenges . 24 II.3.2 Solution Requirements . 24 II.3.2.1 Requirement 1: Automated Deployment of Application components in Heterogeneous environment . 25 II.3.2.2 Requirement 2: Framework for Flexible ML Service De- velopment and Encapsulation . 25 II.3.2.3 Requirement 3: Performance Monitoring and Intelligent Resource Allocation . 26 vi II.4 Design and Implementation of Erudite . 26 II.4.1 Addressing Requirement 1: CloudCAMP - Automated Deployment of Application Components in Heterogeneous Resources . 27 II.4.1.1 Meta-model for Heterogeneous Resources . 28 II.4.1.2 Meta-model for Data Ingestion Frameworks . 29 II.4.1.3 Meta-model for Data Analytics Applications . 30 II.4.1.4 Meta-model for Data Storage Services . 30 II.4.2 Addressing Requirement 2: Erudite Development Kit for AI/ML Model Development . 30 II.4.2.1 Main Meta-model for Erudite Framework . 31 II.4.2.2 Meta-model for Machine Learning Algorithms . 32 II.4.2.3 Model Evaluation and Flexible ML Service Encapsulation 33 II.4.3 Addressing Requirement 3: Framework for Performance Monitor- ing and Intelligent Resource Management . 34 II.4.3.1 Performance Monitoring . 35 II.4.3.2 Resource Management . 35 II.4.4 Support for Collaboration and Versioning . 37 II.5 Evaluation . 38 II.5.1 Evaluating the Rapid Model Development Framework . 38 II.5.2 Evaluation of Rapid Application Prototyping Framework . 40 II.5.3 Performance Monitoring on Heterogeneous Hardware . 41 II.5.4 Resource Management . 43 II.6 Conclusion . 45 II.6.1 Summary . 45 vii III CloudCAMP: A Model-Driven Approach to Automate Cloud Services Deploy- ment and Management . 46 III.1 Introduction . 46 III.1.1 Motivation . 46 III.1.2 Requirements and State-of-the-art Solutions . 47 III.1.2.1 Requirement 1: Reduction in specification details needed for deployment . 47 III.1.2.2 Requirement 2: Auto-completion of Infrastructure Provi- sioning . 48 III.1.2.3 Requirement 3: Support for Continuous Integration, Mi- gration, and Delivery . 49 III.1.3 Overview of Technical Contributions . 51 III.1.4 Organization of the Chapter . 52 III.2 Related Work . 52 III.3 Design and Implementation of CloudCAMP . 55 III.3.1 System Architecture of CloudCAMP . 55 III.3.2 System Implementation of CloudCAMP . 57 III.3.3 CloudCAMP Domain-specific Modeling Language (DSML) . 58 III.3.3.1 Design Rationale for CloudCAMP Meta-models . 58 III.3.3.2 Meta-model for the Cloud Platforms . 59 III.3.3.3 Meta-model for Application Components . 61 III.3.3.4 Defining the Relationship among Components . 61 III.3.3.5 Extensibility of the Meta-model . 62 III.3.4 Design of CloudCAMP Knowledge Base . 63 III.3.4.1 Design of Knowledge Base Database . 63 III.3.4.2 Design of Knowledge Base Template . 64 III.3.4.3 Extensibility of the Knowledge Base . 64 viii III.3.5 Generative Capabilities of CloudCAMP DSML . 65 III.3.5.1 Knowledge Base for Generation of Infrastructure-as-code Solution for Deployment . 66 III.3.5.2 Determining the Order of Deployment and Execution . 66 III.3.5.3 Generation of Infrastructure-as-code for Migration . 68 III.3.5.4 Support for Continuous Delivery . 69 III.3.5.5 Constraints Checking for Correctness Business Models .