Brownout-Oriented and Energy Efficient Management of Cloud Data Centers Minxian Xu Submitted in total fulfilment of the requirements of the degree of Doctor of Philosophy November 2018 School of Computing and Information Systems THE UNIVERSITY OF MELBOURNE Copyright c 2018 Minxian Xu All rights reserved. No part of the publication may be reproduced in any form by print, photoprint, microfilm or any other means without written permission from the author except as permitted by law. Brownout-Oriented and Energy Efficient Management of Cloud Data Centers Minxian Xu Principal Supervisor: Prof. Rajkumar Buyya Abstract Cloud computing paradigm supports dynamic provisioning of resources for deliver- ing computing for applications as utility services as a pay-as-you-go basis. However, the energy consumption of cloud data centers has become a major concern as a typical data center can consume as much energy as 25,000 households. The dominant energy efficient approaches, like Dynamic Voltage Frequency Scaling and VM consolidation, cannot func- tion well when the whole data center is overloaded. Therefore, a novel paradigm called brownout has been proposed, which can dynamically activate/deactivate the optional parts of the application system. Brownout has successfully shown it can avoid overloads due to changes in the workload and achieve better load balancing and energy saving effects. In this thesis, we propose brownout-based approaches to address energy efficiency and cost-aware problem, and to facilitate resource management in cloud data centers. They are able to reduce data center energy consumption while ensuring Service Level Agreement defined by service providers. Specifically, the thesis advances the state-of-art by making the following key contributions: 1. An approach for scheduling cloud application components with brownout. The approach models the brownout enabled system by considering application com- ponents, which are either mandatory or optional. It also contains brownout-based algorithm to determine when to use brownout and how much utilization can be reduced. 2. A resource scheduling algorithm based on brownout and approximate Markov De- cision Process approach. The approach considers the trade-offs between saved en- ergy and the discount that is given to the user if components or microservices are deactivated. 3. A framework that enables brownout paradigm to manage the container-based en- vironment, and provides fine-grained control on containers, which also contains several scheduling policies for managing containers to achieve power saving and QoS constraints. 4. The design and development of a software prototype based on Docker Swarm to reduce energy consumption while ensuring QoS in Clouds, and evaluations of dif- ferent container scheduling policies under real testbeds to help service provider de- ploying services in a more energy-efficient manner while ensuring QoS constraint. iii 5. A perspective model for multi-level resource scheduling and a self-adaptive ap- proach for interactive workloads and batch workloads to ensure their QoS by con- sidering the renewable energy at Melbourne based on support vector machine. The proposed approach is evaluated under our developed prototype system. iv Declaration This is to certify that 1. the thesis comprises only my original work towards the PhD, 2. due acknowledgement has been made in the text to all other material used, 3. the thesis is less than 100,000 words in length, exclusive of tables, maps, bibliogra- phies and appendices. Minxian Xu, November 2018 v Preface This thesis research has been carried out in the Cloud Computing and Distributed Sys- tems (CLOUDS) Laboratory, School of Computing and Information Systems, The Uni- versity of Melbourne. The main contributions of the thesis are discussed in Chapters 2-7 and are based on the following publications: • Minxian Xu, Amir Vahid Dastjerdi, and Rajkumar Buyya, “Energy Efficient Schedul- ing of Cloud Application Components with Brownout,” IEEE Transactions on Sus- tainable Computing (T-SUSC), Volume 1, Number 2, Pages: 40-53, ISSN: 2377-3782, IEEE Computer Society Press, USA, July-Dec 2016. • Minxian Xu, Wenhong Tian, and Rajkumar Buyya, “A Survey on Load Balancing Algorithms for Virtual Machines Placement in Cloud Computing,” Concurrency and Computation: Practice and Experience (CCPE), Volume 29, No. 12, Pages: 1-16, ISSN: 1532-0626, Wiley Press, New York, USA, June 25, 2017. • Minxian Xu and Rajkumar Buyya, “Energy Efficient Scheduling of Application Components via Brownout and Approximate Markov Decision Process,” in Pro- ceedings of the 15th International Conference on Service-Oriented Computing (ICSOC), LNCS, Springer-Verlag Press, Berlin, Germany), Malaga,´ Spain, November 13-16, 2017. • Minxian Xu, Adel Nadjaran Toosi, and Rajkumar Buyya, “iBrownout: An Inte- grated Approach for Managing Energy and Brownout in Container-based Clouds,” IEEE Transactions on Sustainable Computing (T-SUSC), Volume 4, Number 1, Pages: 53-66, ISSN: 2377-3782, IEEE Computer Society Press, USA, Jan-Mar 2019. • Minxian Xu and Rajkumar Buyya, “Brownout Approach for Adaptive Manage- vii ment of Resources and Applications in Cloud Computing Systems: A Taxonomy and Future Directions,” ACM Computing Surveys (CSUR), Volume 52, No. 8, Pages: 1-27, ISSN: 0360-0300, ACM Press, New York, USA, February 2019. • Minxian Xu and Rajkumar Buyya, “BrownoutCon: A Software System based on Brownout and Containers for Energy Efficient Clouds,” Journal of Systems and Soft- ware (JSS), 2019 (under review). • Minxian Xu, Adel Nadjaran Toosi, and Rajkumar Buyya, “A Self-adaptive Ap- proach for Managing Applications and Harnessing Renewable Energy for Sustain- able Cloud Computing,” IEEE Transactions on Parallel and Distributed Systems (TPDS), 2018 (under review). viii Acknowledgements Time flies in the blink of an eye. This year is the tenth year when I started my study at university. I hope this thesis is a new starting point of my academic study rather than an endpoint. I am very thankful to have the opportunity to be supervised by Professor Rajkumar Buyya during my PhD candidature in the recent three years. I met him 5 years ago and then applied his PhD, it just looks like what happened yesterday. I would like to express my deepest gratitude for his continuous insights and supports for my PhD life. I thank Dr. Amir Vahid Dastjerdi for his guidance during the first year of my PhD. I would also like to thank the members of my PhD committee: Prof. Liz Sonenberg and Dr. Adel Nadjaran Toosi. Prof. Liz has provided constructive comments and advice on my research. Dr. Adel has collaborated with me and helped me to improve my work. Besides, I thank Dr. Marcos Assuncao, who kindly helped to provide the Grid’5000 infrastructure resources to support my experiments. I thank the visiting scholars to CLOUDS laboratory, Prof. Wenhong Tian and Prof. Satish Narayana Srirama. They have provided impressive views on enhancing my research. I express my thanks to the other post-docs in CLOUDS laboratory, Dr. Rodrigo Calheiros, Dr. Maria Rodriguez, Dr. Sukphal Singh Gill, Dr. Jungming Jay Son, who have also provided insights and suggestions on the technical side. I would like to thank all members of the CLOUDS Laboratory. To my friends, Dr. Chenhao Qu, Dr. Bowen Zhou, Dr. Xunyun Liu, Safiollah Heidari, Caesar Wu, Sara Kar- dani Moghaddam, Muhammad H. Hilman, Redowan Mahmud, Muhammed Tawfiqul Islam, Shashikant Ilager for their friendship and support. I thank them to proof-reading my work and they gave me their useful comments. I acknowledge the China Scholarship Council, University of Melbourne and ARC Discovery Project for providing me with scholarships to pursue my doctoral study. My deepest gratitude goes to my family. I thank my parents and parents-in-law, with- out their encouragement, I cannot obtain current achievements. My special gratitude goes to my wife, Ms. Mengyun Liu, her sacrifice and love support me to reach here. Minxian Xu Melbourne, Australia October 2018 ix Contents 1 Introduction1 1.1 Motivations....................................4 1.2 Research Problems and Objectives.......................6 1.3 Methodology...................................7 1.4 Contributions...................................9 1.5 Thesis Organization................................ 11 2 Taxonomy and Literature Review 15 2.1 Introduction.................................... 15 2.1.1 Need for Adaptive Management in Cloud Computing Systems.. 16 2.1.2 Motivation of Research......................... 18 2.1.3 Our Contributions............................ 18 2.1.4 Related Surveys.............................. 19 2.2 Background.................................... 19 2.2.1 Cloud Computing............................ 20 2.2.2 Adaptive Management.......................... 20 2.3 Article Selection Methodology......................... 21 2.3.1 Source of Articles............................. 21 2.3.2 Search Method.............................. 21 2.3.3 Outcome.................................. 21 2.4 Brownout Approach............................... 22 2.4.1 Overview of Brownout Approach................... 22 2.4.2 Evolution of Brownout Approaches in Cloud Computing...... 24 2.5 Phases and Taxonomy of Brownout-based Adaptive Management..... 27 2.5.1 Application Design............................ 28 2.5.2 Workload Scheduling.......................... 29 2.5.3 Monitoring................................ 31 2.5.4 Brownout Controller/Dimmer Design................. 32 2.5.5 Metrics................................... 34