A Decision Support Framework for Software Test Managers Deane Robert Larkman Submitted in partial fulfilment of the requirements for the degree of Doctor of Information Technology University of Canberra February 2012 Copyright This thesis may be freely copied and distributed for private use and study. However, no part of this thesis, or the information contained therein, may be included in or referred to in a publication without prior written permission of the author. Any references must be fully acknowledged. Signature: Date: 10/2/2012 © 2012 Deane Robert Larkman - ii - To my wife Vicki - iii - To get through the hardest journey we need take only one step at a time, but we must keep on stepping – Chinese proverb (ThinkExist.com, 2011, para. 4) It is good to have an end to journey toward; but it is the journey that matters, in the end – Ursula K. LeGuin (ThinkExist.com, 2011, para. 5) When you have completed 95 percent of your journey, you are only halfway there – Japanese proverb (ThinkExist.com, 2011, para. 8) - v - ABSTRACT Software is an indispensable part of our modern lifestyle. Users of software in business and society rely on software to support the tasks they need to, or want to achieve. Unfortunately, software fails, and the human and economic consequences can be devastating. High quality software can help minimise the consequences of software failures. Software testing is one way to improve software quality, and thereby provide high quality software. A comprehensive review of the literature has shown that, within the parameters of the boundary and focus of this research, support for test managers’ decision making in software testing is scarce. In particular, research addressing decision support for the software test manager is especially lacking in planning and risk management of successful software testing. This research investigates the need for a decision support framework for software test managers: with a focus on the planning and risk management done by software test managers for successful software testing. The research findings developed during this project are based on concepts, issues and ideas, grounded in evidence, reasons, and experience. The major contribution of this research is the development and application of a decision support framework. Software test managers can use the decision support framework in assisting them to plan and provide risk management for successful software testing. The research contribution falls into two parts. The first part is the design and development of a decision support framework, which is described and discussed. The components of the framework are detailed, and the topology of those components is illustrated graphically. The second part of the research contribution is the application of the decision support framework by software test managers, establishing a basis for risk management. Application of the decision support framework is a two-step procedure. The software test manager first assigns relevant input to the framework, and this input allows the test manager to use the framework to model their particular software testing situation, set in the context of their organisation. Next, the software test manager using one or more analytical techniques, developed from two different evaluation perspectives, evaluates the model built in the first application step. The analytical techniques are presented and their potential use by software test managers is illustrated. - vii - In summary, this research has produced a decision support framework, a practical and useful tool for the software test manager to plan for and manage the risk in successful software testing. It is envisaged that software test managers can apply the decision support framework to their specific organisational and software testing circumstances. - viii - CERTIFICATE OF AUTHORSHIP OF THESIS Except where clearly acknowledged in footnotes, quotations and the bibliography, I certify that I am the sole author of the thesis submitted today entitled – A Decision Support Framework for Software Test Managers I further certify that to the best of my knowledge the thesis contains no material previously published or written by another person except where due reference is made in the text of the thesis. The material in the thesis has not been the basis of an award of any other degree or diploma except where due reference is made in the text of the thesis. The thesis complies with University requirements for a thesis as set out in Gold Book Part 7: Examination of Higher Degree by Research Theses Policy, Schedule Two (S2). Refer to http://www.canberra.edu.au/research-students/goldbook Signature of candidate Signature of chair of the supervisory panel Date: 10/2/2012 - ix - KEYWORDS Software testing, risk management, test planning, frameworks, models, software test manager, software test management, decision making, and decision support. - xi - PUBLICATIONS Some of the material in this thesis has been peer reviewed and published. Larkman, D., Jentzsch, R. & Mohammadian, M. (2011). “Software Testing - Factor Contribution Analysis in a Decision Support Framework”. In J. Watada, G. Phillips-Wren, L. C. Jain & R. J. Howlett (Eds.), Smart Innovation, Systems and Technologies: Vol 10. Intelligent Decision Technologies: Proceedings of the 3rd International Conference on Intelligent Decision Technologies (IDT' 2011), Piraeus, Greece, July 20-22, 2011, (pp. 897-905). Berlin: Springer-Verlag. doi: 10.1007/978-3-642-22194-1_88 Larkman, D., Mohammadian, M., Balachandran, B. & Jentzsch, R. (2010). “Fuzzy Cognitive Map for Software Testing Using Artificial Intelligence Techniques”. In H. Papadopoulos, A. S. Andreou & M. Bramer (Eds.), IFIP Advances in Information and Communication Technology: Vol 339. Artificial Intelligence Applications and Innovations: Proceedings of the 6th IFIP WG 12.5 International Conference, AIAI 2010, Larnaca, Cyprus, October 6-7, 2010, (pp. 328-335). Berlin: Springer. doi: 10.1007/978-3-642-16239-8_43 Larkman, D., Mohammadian, M., Balachandran, B. & Jentzsch, R. (2010). “General Application of a Decision Support Framework for Software Testing Using Artificial Intelligence Techniques”. In G. Phillips-Wren, L. C. Jain, K. Nakamatsu & R. J. Howlett (Eds.), Smart Innovation, Systems and Technologies: Vol 4. Advances in Intelligent Decision Technologies: Proceedings of the Second KES International Symposium IDT 2010, Baltimore, Maryland, July 28-30, 2010, (pp. 53-63). Berlin: Springer-Verlag. doi: 10.1007/978-3-642-14616-9_5 - xiii - TABLE OF CONTENTS ABSTRACT ................................................................................................................................ vii CERTIFICATE OF AUTHORSHIP OF THESIS .............................................................................. ix KEYWORDS ............................................................................................................................... xi PUBLICATIONS ........................................................................................................................ xiii LIST OF FIGURES..................................................................................................................... xix LIST OF TABLES ...................................................................................................................... xxi ACKNOWLEDGEMENTS ......................................................................................................... xxiii CHAPTER 1. OVERVIEW OF THE RESEARCH ............................................................................ 1 1.1 Introduction to the Research ............................................................................................ 1 1.2 Research Problem and the Strategy to Tackle that Problem ............................................ 5 1.2.1 Research Problem ..................................................................................................... 5 1.2.2 Research Goal ........................................................................................................... 6 1.2.3 Research Objectives .................................................................................................. 7 1.2.4 Research Questions ................................................................................................... 7 1.3 Approach to the Research ................................................................................................ 7 1.4 Boundary of the Research ................................................................................................ 8 1.5 Research Focus ................................................................................................................. 9 1.6 Research Significance ...................................................................................................... 9 1.7 Outline of the Thesis ...................................................................................................... 10 CHAPTER 2. LITERATURE REVIEW ........................................................................................ 15 2.1 Introduction .................................................................................................................... 15 2.2 The Literature Review and Absence of Evidence .......................................................... 16 2.3 Criteria Applied to the Literature Review ...................................................................... 17 2.4 Decision Support Issues ................................................................................................