A Study on Software Testability and the Quality of Testing In Object-Oriented Systems By Amjed Abdalhamed Abbas Tahir A THESIS SUBMITTED IN FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN THE UNIVERSITY OF OTAGO This thesis is dedicated to the loving memory of my father, Abdalhmed, who taught me to always believe in myself. Gone but not forgotten! my mother, Amnah, for her constant love and support. I Abstract Software testing is known to be important to the delivery of high-quality systems, but it is also challenging, expensive and time-consuming. This has motivated academic and industrial researchers to seek ways to improve the testability of software. Software testability is the ease with which a software artefact can be effectively tested. The first step towards building testable software components is to understand the factors – of software processes, products and people – that are related to and can influence software testability. In particular, the goal of this thesis is to provide researchers and practitioners with a comprehensive understanding of design and source code factors that can affect the testability of a class in object oriented systems. This thesis considers three different views on software testability that address three related aspects: 1) the distribution of unit tests in relation to the dynamic coupling and centrality of software production classes, 2) the relationship between dynamic (i.e., runtime) software properties and class testability, and 3) the relationship between code smells, test smells and the factors related to smells distribution. The thesis utilises a combination of source code analysis techniques (both static and dynamic), software metrics, software visualisation techniques and graph-based metrics (from complex networks theory) to address its goals and objectives. A systematic mapping study was first conducted to thoroughly investigate the body of research on dynamic software metrics and to identify issues associated with their selection, design and implementation. This mapping study identified, evaluated and classified 62 research works based on a pre-tested protocol and a set of classification criteria. Based on the findings of this study, a number of dynamic metrics were selected and used in the experiments that were then conducted. The thesis demonstrates that by using a combination of visualisation, dynamic analysis, static analysis and graph-based metrics it is feasible to identify central classes and to diagrammatically depict testing coverage information. Experimental results show that, even in projects with high test coverage, some classes appear to be left without any direct unit testing, even though they play a central role during a typical execution profile. It is II contended that the proposed visualisation techniques could be particularly helpful when developers need to maintain and reengineer existing test suites. Another important finding of this thesis is that frequently executed and tightly coupled classes are correlated with the testability of the class – such classes require larger unit tests and more test cases. This information could inform estimates of the effort required to test classes when developing new unit tests or when maintaining and refactoring existing tests. An additional key finding of this thesis is that test and code smells, in general, can have a negative impact on class testability. Increasing levels of size and complexity in code are associated with the increased presence of test smells. In addition, production classes that contain smells generally require larger unit tests, and are also likely to be associated with test smells in their associated unit tests. There are some particular smells that are more significantly associated with class testability than other smells. Furthermore, some particular code smells can be seen as a sign for the presence of test smells, as some test and code smells are found to co-occur in the test and production code. These results suggest that code smells, and specifically certain types of smells, as well as measures of size and complexity, can be used to provide a more comprehensive indication of smells likely to emerge in test code produced subsequently (or vice versa in a test-first context). Such findings should contribute positively to the work of testers and maintainers when writing unit tests and when refactoring and maintaining existing tests. III Acknowledgements I would like to express my gratitude to a number of people who supported me throughout my PhD journey. Firstly, I would like to express my sincere gratitude to my primary supervisor Prof Stephen MacDonell for his guidance and support throughout the course of my PhD (and other related) research. Steve has been a constant source of motivation during every stage of my research. I would like also to thank my secondary supervisor Prof Michael Winikoff for his advice, guidance and for reading drafts and providing feedback during later stages of this thesis. I am extremely grateful to my mother, Amnah, and all my siblings (Adil, Adnan, Adlah, Abbas, and Ammar) and their young families for their encouragement and unlimited support (especially spiritually) throughout my PhD journey and my life in general. I could not have done it without your support, so thank you all! I would like to especially thank Abbas for his inspiration and for the many technical discussions which have improved the outcomes of this thesis. I thank my fellow colleagues and lab-mates at the University of Otago (and previously at AUT): Sherlock Licorish, Michael Bosu and Aftab Mughal. They have certainly helped me in making this PhD experience an enjoyable one. I also thank all fellow PhD students at the Department of Information Science for the lively discussions we had whenever we get together. In particular, I would like to thank M. Alghamdi, A. Alshaer, M. Farhangian and H. Lin. I would like also to thank my fellow colleagues and members of the Software Engineering Research Laboratory at Auckland University of Technology, whom I shared space with during my time at AUT: Frederik Schmidt, Bilal Raza, Nadia Kasto, Minjuan Tong, Waqar Husin, Da Zhang and Dr Jacqui Whalley. I would like to especially thank Jim Buchan for his feedback and collaboration during various stages of this PhD journey. I am grateful for the financial support I received from University of Otago (Otago’s Business School Dean Scholarship) and previously from AUT (SERL special PhD Scholarship). IV Last but not least, I would like to thank all my friends (around the world!) for their moral help and support, and for their understanding during my PhD journey. I am lucky to have such friends! Thank you all… Amjed Tahir July 2015 V Table of Contents Abstract ............................................................................................................................... II Acknowledgements ......................................................................................................... IV List of Tables .................................................................................................................... IX List of Figures ................................................................................................................... XI Acronyms ....................................................................................................................... XIII Chapter 1 Introduction ..................................................................................................... 1 1.1. Background and Motivation ................................................................................. 1 1.2. Research Goal and Objectives ............................................................................... 2 1.3. Main Outcomes ....................................................................................................... 4 1.4. List of Publications ................................................................................................. 5 1.5. Structure of the Thesis ............................................................................................ 6 Chapter 2 Literature Review ........................................................................................ 8 2.1 Introduction ............................................................................................................. 8 2.2 Software Testability .............................................................................................. 10 2.2.1 Testability Definitions ................................................................................... 11 2.2.2 Testability Factors .......................................................................................... 13 2.3 Software Measurement and Metrics .................................................................. 14 2.3.1 Static Metrics .................................................................................................. 15 2.3.2 Dynamic Metrics ............................................................................................ 16 2.4 Program Comprehension .................................................................................... 22 2.4.1 Test Comprehension...................................................................................... 23 2.4.2 Test and Code Smells .................................................................................... 25 Chapter 3 Research Methodology ............................................................................