An Empirical Study Investigating the Predictors of Software Metric Correlation in Application Code and Test Code. by Daniel Kwame Dapaah Afriyie A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Carleton University Ottawa, Ontario © 2019, Daniel Kwame Dapaah Afriyie Abstract On non-trivial software, the large test code base needs adequate maintenance similarly to the application code. It has often been argued that test code should be simple, some authors even arguing that a test script should be reduced to a single control flow. If this is indeed what happens in practice, then we believe test code maintenance will be very different from application code maintenance, despite the fact that both are source code. Using a number of large open source software, we compare application code and test code by using a series of well-known source code complexity metrics. The results reveal that application code is, as expected, more complex than test code but not necessarily so much more complex. The study also confirms the assertion that test code is not as simple as it should be, at least as advocated in textbooks, and may therefore be very complex to maintain. Using complexity metrics to determine the difference, our findings also reveal that, the kind of code determines the extent of monotonicity between the number of lines of code (LOC) and cyclomatic complexity (CC). While a number of authors hypothesize and experimentally confirm that CC has a very strong correlation with LOC, justifying the use of LOC in place of CC (and Halstead Effort), we believe that this strong correlation is prevalent only in production code as results from test code prove otherwise. In test code, there exist a very weak (or almost no) correlation between (a) LOC and CC, (b) Halstead Effort and CC, and (c) LOC and Halstead Effort. We therefore argue that the level of correlation depends on at least three factors namely: the kind of code, the kind of software and the kind of metric. Therefore, we believe it would be inappropriate, without considering these factors to substitute one metric with another using some generalized correlation hypothesis. The results gained thus far contradict the notion that some metrics are correlated (that is if the right factors are not considered). Given the weak monotonicity between CC and LOC, CC and Halstead Effort for test code, we disagree that CC and Halstead metrics are redundant as some studies suggest. Rather, we advocate for the use of CC over LOC (or both or cylomatic density) due to numerous advantages CC has over LOC such as the fact that CC is perceived to better reflect cognitive complexity, numerical complexity, adequacy of testing, interdependency and code refactoring that cannot be accounted for simply by LOC. ii Acknowledgements One great discovery of climbing mountains and winning battles is the realization of even greater mountains to climb and tougher battles to win. In doing so, we are faced with hard choices; one of which is flinching back, to the very least. The true test of our resilience is in our ability to ultimately win and endure climbing higher mountains. Without which many have lived their lives not having ever lived. I am therefore grateful to Almighty God that this project is finally completed. A few years ago, a bleary-eyed stare at what lies in the future was unassuring, but the joining of the dots as time went on gives us this delight to be forever grateful and an assurance to be hopeful. I owe special thanks most of all, to my supervisor, Prof. Yvan Labiche. Such a great inspiration. Everything he does, from what is believed to be conceptually complex to the least significant, are tackled with a great deal of perfection and a high level of academic proficiency. A mentor, a father, and a friend. You only get better by heeding to his advice. Very helpful, insightful and full of brilliance. I am privileged to work with him. It was a great pleasure to collaborate with Prof. Natalia Stepanova at the School of Mathematics and Statistics for sharing ideas on complicated topics when organizing my thoughts on statistical principles and methodologies. I feel so favored to enjoy special treatment from Jennifer Poll, Jenna McConnell and Darlene Hebert. Many thanks to them and members of the Department and Faculty. I am indebted to the many individuals who generously and without hesitation, supported in various ways. The last stages of my masters had turmoil moments, to say the least. Thankfully, I had my fair share of great and reliable people who stood with me. A mentor Mr. George Cole, my sweetheart and good friend Abigail Oduro and a brother Eric Obeng. I am grateful for the support and encouragement from my family, Samuel Afriyie, Mercy Afriyie and Isaac Afriyie. Where water fails, blood sticks better. I love you. iii Table of Contents Chapter 1 INTRODUCTION .......................................................................................... 1 1.1 Contribution of this Study .............................................................................................. 3 1.2 Summary of This Research ............................................................................................ 4 1.3 Organization of this Document....................................................................................... 5 Chapter 2 RELATED WORK ......................................................................................... 6 Chapter 3 BACKGROUND INFORMATION ............................................................ 11 3.1 Cyclomatic Complexity, Lines of Code and Halstead Metrics. ................................... 11 3.2 On the Use of Maintainability Index ............................................................................ 12 3.3 Measuring Correlation .................................................................................................. 13 3.3.1 Spearman’s Correlation Coefficient ......................................................................... 14 3.3.2 On Pearson’s Correlation Coefficient ...................................................................... 15 3.3.3 Kendall’s Correlation Coefficient ............................................................................ 15 3.3.4 Conclusion on correlation tests ................................................................................ 16 3.4 Test for Normality: Anderson Darling Test, Shapiro-Wilk, D’Agostino K-squared, Kolmogorov Smirnov ................................................................................................................ 17 3.5 Homogeneity of Variance: T-Test and ANOVA .......................................................... 18 3.6 Statistical Analysis – Wilcoxon Rank Sum Test .......................................................... 19 3.6.1 Kolmogorov Smirnov Tests ..................................................................................... 20 Chapter 4 EXPERIMENTAL DESIGN ....................................................................... 21 4.1 Selected Applications and Rationale ............................................................................ 21 4.2 Measurement tool ......................................................................................................... 24 4.3 Selected metrics ............................................................................................................ 25 4.4 Selection of Statistical Methods ................................................................................... 26 Chapter 5 RESULTS (ANALYSIS OF DATA) ........................................................... 28 5.1 Statistical Analysis of Extracted Data .......................................................................... 28 5.1.1 Descriptive statistics on Lines of Code .................................................................... 28 5.1.2 Descriptive statistics on Cyclomatic Complexity .................................................... 35 5.1.3 Descriptive statistics on Using Distributions, Graphs and Average Values for Cyclomatic Complexity......................................................................................................... 38 5.2 Inferential Statistics ...................................................................................................... 54 5.3 On the Use of Non-Parametric Statistical Tests ........................................................... 57 iv 5.4 On Correlation among Software Metric ....................................................................... 59 Chapter 6 THREATS TO VALIDITY ......................................................................... 65 Chapter 7 CONCLUSIONS ........................................................................................... 68 REFERENCES ................................................................................................................ 71 APPENDICES ................................................................................................................. 76 Appendix A Distributions of cyclomatic complexity values for objects. .................................. 76 A.1 Sample representative distributions ......................................................................... 76 A.2 Boxplots for all objects ............................................................................................ 78 Appendix B : Linear Scale Zoomed In On The