University of Calgary PRISM: University of Calgary's Digital Repository Graduate Studies The Vault: Electronic Theses and Dissertations 2014-09-30 RELREA-An Analytical Approach Supporting Continuous Release Readiness Evaluation Shahnewaz, S. M. Shahnewaz, S. M. (2014). RELREA-An Analytical Approach Supporting Continuous Release Readiness Evaluation (Unpublished master's thesis). University of Calgary, Calgary, AB. doi:10.11575/PRISM/27619 http://hdl.handle.net/11023/1867 master thesis University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission. Downloaded from PRISM: https://prism.ucalgary.ca UNIVERSITY OF CALGARY RELREA-An Analytical Approach Supporting Continuous Release Readiness Evaluation by S. M. Shahnewaz A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE GRADUATE PROGRAM IN ELECTRICAL AND COMPUTER ENGINEERING CALGARY, ALBERTA SEPTEMBER, 2014 © S. M. Shahnewaz 2014 Abstract As part of iterative development, decision about “Is the software product ready to be released at some given release date?” have to be made at the end of each release, sprint or iteration. While this decision is critically important, so far it is largely done either informally or in a simplistic manner relying on a small set of isolated metrics. In addition, continuity in release readiness evaluation is not achieved and any problems related to release issues cannot be addressed proactively. In this thesis, an analytical approach called RELREA is proposed to address these gaps. The main characteristics of RELREA are: i) project specific release readiness attributes and metrics are selected in a systematic way, ii) overall release readiness is determined by aggregating and evaluating the degree of satisfaction of the most relevant readiness attributes, iii) continuous visibility on release readiness status, and iv) projection of release readiness at release time. Case studies with proprietary and open source software projects are performed to evaluate the proposed approach and its implementation. The case study results show that the RELREA approach and its implementation are able to assist one product manager in evaluating, monitoring, and analyzing release readiness at any point of time in the release cycle. ii Publications Some of the ideas, materials, tables, figures used in this thesis have appeared previously in the following referred publications: Referred published and accepted papers: 1. S. M. Shahnewaz and G. Ruhe, "RELREA - An Analytical Approach for Evaluating Release Readiness," in Proceedings of the 26thInternational Conference on Software Engineering and Knowledge Engineering (SEKE), pp. 437 – 442, 2014. 2. S. M. D. A. Alam, S. M. Shahnewaz, D. Pfahl, and G. Ruhe, "Monitoring Bottlenecks in Achieving Release Readiness – A Retrospective Case Study across Ten OSS Projects”, Accepted in "8th International Symposium on Empirical Software Engineering and Measurement (ESEM)," 2014. 3. Jason Ho, S. M. Shahnewaz and G. Ruhe, "A Prototype Tool Supporting When-to- release Decisions in Iterative Development," in Proceedings of the 2nd International Workshop on Release Engineering (RELENG), 2014. Papers under review: 4. S. M. D. A. Alam, S. M. Shahnewaz, D. Pfahl, and G. Ruhe, "Analysis and Improvement of Release Readiness – A Genetic Optimization Approach, Submitted in "15th International Conference on Product-Focused Software Development and Process Improvement (PROFES)", 2014. 5. S. M. D. A. Alam, Jason Ho, D. Pfahl, and G. Ruhe, and S. M. Shahnewaz "Are We Ready to Release? Analyzing Factors that Limit Release Readiness, Submitted in "IEEE Software Special Issue", 2014. iii Acknowledgements This thesis would not have taken place without the continuous guidance and the help of several individuals who in one way or other contributed and extended their valuable assistance in the preparation and completion of this thesis. I am mostly grateful to my supervisor Dr. Guenther Ruhe for his valuable guidance, continuous support, and supervision. Without his inspiring enthusiasm and encouragement this thesis could not be successfully completed. I would also like to thank my former supervisor Dr. Vahid Garousi for introducing me in the world of software engineering research. My heartfelt thanks and appreciation go to my colleagues in the Software Decision Support Laboratory for their benevolent effort to put valuable comments on this thesis. Discussing with them, from time to time, has really sharpened my thoughts’ process for the research. Finally, this work is really indebted to the endless support from my parents, relatives, and one very special person. The distance among us was never a hindrance to their way of encouraging me. Last but not the least, the friends from my house, who are also from my home country, gave me a perfect milieu to study and flavor of my country to make me reminiscent of my family. iv This thesis is dedicated to my father and mother for all their continuous support, love and inspiration. v Table of Contents Abstract ........................................................................................................................................... ii Publications .................................................................................................................................... iii Acknowledgements ........................................................................................................................ iv Table of Contents ........................................................................................................................... vi List of Tables ................................................................................................................................. ix List of Figures and Illustrations ..................................................................................................... xi List of Abbreviations ................................................................................................................... xiii CHAPTER ONE: INTRODUCTION ..............................................................................................1 1.1 Motivation ..............................................................................................................................1 1.2 Difficulties in Evaluating Software Release Readiness .........................................................2 1.3 Research Objectives ...............................................................................................................4 1.4 Overview of the Solution Approach ......................................................................................5 1.5 Thesis Structure .....................................................................................................................7 CHAPTER TWO: BACKGROUND AND LITERATURE REVIEW ...........................................8 2.1 Background ............................................................................................................................8 2.1.1 Iterative Software Development ....................................................................................8 2.1.2 Definition of Release in Iterative Development ............................................................9 2.1.3 Software Product Management ...................................................................................10 2.1.4 Definition of Release Readiness ..................................................................................11 2.1.5 Value of Continuous RR Evaluation in the Context of SPM ......................................12 2.1.6 Concept of Fuzzy Set Theory ......................................................................................13 2.1.6.1 The Use of Fuzzy Set Theory in Software Engineering ....................................15 2.1.7 Overview of the Aggregation Operators .....................................................................18 2.1.7.1 Mathematical and Behavioural Properties of Aggregation Operators ...............18 2.1.7.2 Some Popular Aggregation Operators ...............................................................21 2.1.8 Overview of the Time Series Projection Methods ......................................................23 2.1.8.1 Simple Regression .............................................................................................24 2.1.8.2 Exponential Smoothing......................................................................................24 2.1.8.3 Time Series Decomposition ...............................................................................25 2.2 A Glossary of Terms Related to Software Release Readiness ............................................26 2.3 Literature Review ................................................................................................................27 2.3.1 Existing Release Readiness Approaches .....................................................................27 2.3.1.1 Checklist-based Approaches ..............................................................................27 2.3.1.2 Multi-dimensional Metrics Aggregation-based Approaches .............................29 2.3.1.3 Testing Metrics-based Approach