A survey on software testability Vahid Garousi Michael Felderer Feyza Nur Kılıçaslan Information Technology Group Blekinge Institute of Technology, Sweden & Department of Computer Engineering Wageningen University, Netherlands University of Innsbruck, Austria Hacettepe University, Ankara, Turkey [email protected] [email protected] [email protected] Abstract: Context: Software testability is the degree to which a software system or a unit under test supports its own testing. To predict and improve software testability, a large number of techniques and metrics have been proposed by both practitioners and researchers in the last several decades. Reviewing and getting an overview of the entire state-of-the-art and state-of-the- practice in this area is often challenging for a practitioner or a new researcher. Objective: Our objective is to summarize the body of knowledge in this area and to benefit the readers (both practitioners and researchers) in preparing, measuring and improving software testability. Method: To address the above need, the authors conducted a survey in the form of a systematic literature mapping (classification) to find out what we as a community know about this topic. After compiling an initial pool of 303 papers, and applying a set of inclusion/exclusion criteria, our final pool included 208 papers. Results: The area of software testability has been comprehensively studied by researchers and practitioners. Approaches for measurement of testability and improvement of testability are the most-frequently addressed in the papers. The two most often mentioned factors affecting testability are observability and controllability. Common ways to improve testability are testability transformation, improving observability, adding assertions, and improving controllability. Conclusion: This paper serves for both researchers and practitioners as an “index” to the vast body of knowledge in the area of testability. The results could help practitioners measure and improve software testability in their projects. To assess potential benefits of this review paper, we shared its draft version with two of our industrial collaborators. They stated that they found the review useful and beneficial in their testing activities. Our results can also benefit researchers in observing the trends in this area and identify the topics that require further investigation. Keywords: Software testing; software testability; survey; systematic literature mapping; systematic literature review; systematic mapping 1 TABLE OF CONTENTS 1 INTRODUCTION ....................................................................................................................................................................... 2 2 BACKGROUND AND RELATED WORK ........................................................................................................................................ 3 2.1 A brief overview of the concept of software testability ...................................................................................................................... 3 2.2 Related works: other survey (review) papers on software testability ................................................................................................ 5 3 RESEARCH METHOD AND SLM PLANNING ................................................................................................................................ 6 3.1 Overview .......................................................................................................................................................................................... 6 3.2 Goal and review questions ............................................................................................................................................................... 7 4 SEARCHING FOR AND SELECTION OF SOURCES ........................................................................................................................ 8 4.1 Selecting the source engines and search keywords ......................................................................................................................... 8 4.2 Application of inclusion/exclusion criteria and voting ........................................................................................................................ 9 4.3 Final pool of the primary studies ..................................................................................................................................................... 10 5 DEVELOPMENT OF THE SYSTEMATIC MAP AND DATA-EXTRACTION PLAN .................................................................................. 10 5.1 Development of the systematic map............................................................................................................................................... 11 5.2 Data extraction process and data synthesis ................................................................................................................................... 11 6 RESULTS .............................................................................................................................................................................. 13 6.1 Group 1- Common to all SLM studies............................................................................................................................................. 13 6.1.1 RQ 1.1: Classification of studies by contribution types ................................................................................................................................... 13 6.1.2 RQ 1.2: Classification of studies by research method types ........................................................................................................................... 14 6.2 Group 2-Specific to the domain (testability) .................................................................................................................................... 16 6.2.1 RQ 2.1- Types of approaches for treating testability ....................................................................................................................................... 16 6.2.2 RQ 2.2- Factors affecting testability ............................................................................................................................................................... 19 6.2.3 RQ 2.3- Techniques for improving testability .................................................................................................................................................. 23 6.3 Group 3-Specific to empirical studies and also the case-study of each paper ................................................................................ 24 6.3.1 RQ 3.1-Research questions investigated in the empirical studies .................................................................................................................. 24 6.3.2 RQ 3.2-Number and sizes of SUTs (examples) in each paper ....................................................................................................................... 25 6.3.3 RQ 3.3-Domains of SUTs .............................................................................................................................................................................. 26 6.4 Group 4- Demographic information ................................................................................................................................................ 26 6.4.1 RQ 4.1-Affiliation types of the study authors .................................................................................................................................................. 26 6.4.2 RQ 4.2-Highly cited papers ............................................................................................................................................................................ 27 7 DISCUSSIONS ....................................................................................................................................................................... 28 7.1 Summary of research findings ........................................................................................................................................................ 28 7.2 Potential benefits of this review ...................................................................................................................................................... 29 7.2.1 Benefits for practitioners ................................................................................................................................................................................ 29 7.2.2 Benefits for researchers ................................................................................................................................................................................. 30 7.3 Potential threats to validity .............................................................................................................................................................. 30 8 CONCLUSIONS AND FUTURE WORK ........................................................................................................................................ 31 ACKNOWLEDGEMENTS ............................................................................................................................................................ 32 9 REFERENCES ......................................................................................................................................................................