Thesis Presented to the University of Waterloo in Fulfillment of the Thesis Requirement for the Degree of Doctor of Philosophy in Electrical and Computer Engineering
Total Page:16
File Type:pdf, Size:1020Kb
A Game-Theoretic Decision-Making Framework for Engineering Self-Protecting Software Systems by Mahsa Emami-Taba A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Doctor of Philosophy in Electrical and Computer Engineering Waterloo, Ontario, Canada, 2018 c Mahsa Emami-Taba 2018 Examining Committee Membership The following served on the Examining Committee for this thesis. The decision of the Examining Committee is by majority vote. External Examiner: Dr. Mohammad Zulkernine Professor and Canada Research Chair, School of Computing, Queen’s University Supervisor: Dr. Ladan Tahvildari Professor, Dept. of Electrical and Computer Engineering, University of Waterloo Internal Member: Dr. Otman Basir Professor, Dept. of Electrical and Computer Engineering, University of Waterloo Dr. Vijay Ganesh Associate Professor, Dept. of Electrical and Computer Engineering, University of Waterloo Internal-External Member: Dr. Reid Holmes Adjunct Associate Professor, Cheriton School of Computer Science, University of Waterloo ii Author’s Declaration I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. iii Abstract Targeted and destructive nature of strategies used by attackers to break down a software system require mitigation approaches with dynamic awareness. Making a right decision, when facing today’s sophisticated and dynamic attacks, is one of the most challenging aspects of engineering self-protecting software systems. The challenge is due to: (i) the consideration of the satisfaction of various security and non-security quality goals and their inherit conflicts with each other when selecting a countermeasure, (ii) the proactive and dynamic nature of these security attacks which make their detection and consequently their mitigation challenging, and (iii) the incorporation of uncertainties such as the intention and strategy of the adversary to attack the software system. These factors motivated the need for a decision-making engine that facilitates adaptive security from a holistic view of the software system and the attacker. In- spired by game theory, in this research work, we model the interactions between the attacker and the software system as a two-player game. Using game-theoretic techniques, the self-protecting software systems is able to: (i) fuse the strategies of attackers into the decision-making model, and (ii) refine the strategies in dynamic attack scenarios by utilizing what has learned from the system’s and adversary’s interactions. This PhD research devises a novel framework with three phases: (i) modeling quality/malicious goals aiming at quantifying them into the decision-making engine, (ii) designing game-theoretic techniques which build the decision model based on the satisfaction level of quality/malicious goals, and (iii) realizing the decision-making engine in a working software system. The framework aims at exhibiting a plug- and-play capability to adapt a game-theoretic technique that suite security goals and requirements of the software. In order to illustrate the plug-and-play capability of our proposed framework, we have designed and developed three decision-making engines. Each engine aims at addressing a different challenge in adaptive secu- rity. Hence, three distinct techniques are designed: (i) incentive-based (“IBSP”), (ii) learning-based (“MARGIN”), and (iii) uncertainty-based (“UBSP”). For each engine a game-theoretic approach is taken considering the security requirements and the input information. IBSP maps the quality goals and the incentives of the at- tacker to the interdependencies among defense and attack strategies. MARGIN, protects the software system against dynamic strategies of attacker. UBSP, han- dles adversary-type uncertainty. The evaluations of these game-theoretic approaches show the benefits of the proposed framework in terms of satisfaction of security and non-security goals of the software system. iv Acknowledgements First and foremost I would like to thank my enthusiastic supervisor Professor Ladan Tahvildari. It was under Dr. Tahvildari’s supervision that I prospered in this compelling research area, and ultimately, was able to meet and pass all the milestones one after another. My Ph.D. journey has been an amazing experience, not only for her tremendous academic support, but also for having her as a mentor and a sister. In addition to our academic collaboration, I greatly value the close personal rapport that Ladan and I have forged over the years. Words cannot express my gratitude for all her sweet gestures, encouragements and kind support during my ups and downs of starting a new life in Waterloo. I would like to thank the members of my dissertation committee. I was so for- tunate to have such an intellectual committee with me along the path: Professor Mohammad Zulkernine my external examiner, for having accepted to take the time out of his busy schedule to read my thesis and provide me with his insightful sug- gestions, Dr. Otman Basir for his encouraging comments and indispensable support, Dr. Vijay Ganesh for his inspiring remarks which broadened my perspectives in re- search, Dr. Reid Holmes, my internal-external examiner for his invaluable comments on my thesis, and Dr. Adam Kolkiewicz for serving as the thesis examination chair. I am grateful to have had the opportunity to meet Professor Hausi Muller. I cannot thank him enough for all his valuable feedback and kind support. I am par- ticularly indebted to Dr. Marin Litoiu for exposing me to their testbed infrastructure at CERAS lab. I was fortunate to be part of the collaboration project with Profes- sor Ettore Merlo and Dr. Marios-Eleftherios Fokaefs from Ecole Polytechnique de Montreal, Viorel Onut from IBM Markham, and CERAS lab member Nasim Beigi Mohammadi. Also, I would like to express my gratitude to Dr. Patrick Martin for his valuable feedback and remarks during the ORF-RE project on Ultra-Large-Scale Services meetings. I would like to express my sincere appreciation to Communications Security Es- tablishment Canada (CSEC) to open my horizon to do this line of research. I would like to take time and thank the Doctoral Symposium committee, and the ACM stu- dent research competition committee, at the International Conference on Software Engineering (ICSE)-2017, whose constructive comments, and influential criticisms, paved the way for accomplishing my research goals. I am very lucky to have been given the opportunity to be a part of Lovell Corpo- ration. I am grateful for their contribution in my academic life as a start up company to see where some techniques used in my thesis can have a direct application in a v young company. Many thanks to Kelly Lovell for providing a pleasant and friendly working and research environment. I would like to thank my colleagues in the Software Technologies and Applied Research (STAR) lab, Dr. Mohammad Hamdaqa and Soheil Soltani for their moral support and valuable feedback. In particular, I would like to thank Dr. Mehdi Amoui for all the brainstorming sessions and discussions. I gratefully acknowledge the funding sources that made my Ph.D. work possi- ble. I was funded by Natural Sciences and Engineering Research Council (NSERC) Engage and Ontario Centres of Excellence (OCE) Voucher for Innovation and Pro- ductivity (VIP), University of Waterloo Provost Doctoral Entrance Award, Ontario Graduate Scholarship (OGS), University of Waterloo Presidents Graduate Scholar- ship (PGS), and Lovell Corporation. Finally, but by no means least, I owe an immense debt of gratitude to my awesome Mom and Dad, who raised me with love and supported me in all my pursuits, and to my sisters, Leila and Parisa, who were always next to me. Thank you for all the love and support and sacrifice you have made through bad times and for being a part of my happiness in the good times. This last word of acknowledgment I have saved for my caring and encouraging friend, Abbas, whose faithful support during the final stages of this Ph.D. is so appreciated. vi Dedication To my beloved parents and sisters. vii Table of Contents List of Tables xiii List of Figures xv 1 Introduction1 1.1 Motivation.................................1 1.2 Problem Description and Research Focus................3 1.3 Approach and Research Questions....................4 1.3.1 RQ1: How should the framework incorporate the incentives of the attacker?...........................4 1.3.2 RQ2: How should the framework learn from previous action selections?.............................5 1.3.3 RQ3: How should the framework support adversary-type un- certainty?.............................5 1.4 Research Contributions..........................5 1.5 Document Organization.........................8 2 Background and Related Works9 2.1 Self-Protecting Software Principles and Requirements......... 11 2.2 Related Research Projects........................ 14 2.2.1 Adaptive Application Security.................. 14 2.2.2 Modeling and Analyzing Security in Self-Protecting Software. 15 viii 2.2.3 Decision Making in Self-Protecting Software.......... 16 2.2.4 Application of Game Theory in Cybersecurity......... 18 2.2.5 Attack-Type Uncertainty in Self-Protecting Software..... 19 2.3 Summary................................. 22 3 A Framework for Decision Making in Self-Protecting Software Sys- tems 23 3.1 A Motivating Scenario.........................