THE EFFECTIVENESS OF TEST-DRIVEN DEVELOPMENT AND REFACTORING TECHNIQUES IN COMPUTATIONAL SCIENCE AND ENGINEERING SOFTWARE DEVELOPMENT by AZIZ NANTHAAMORNPHONG JEFFREY CARVER, COMMITTEE CHAIR JEFF GRAY KARLA MORRIS RANDY SMITH XIAOYAN HONG A DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of The University of Alabama TUSCALOOSA, ALABAMA 2014 Copyright Aziz Nanthaamornphong 2014 ALL RIGHTS RESERVED ABSTRACT Many computational science and engineering (CSE) software developers have applied soft- ware engineering practices to their work in recent years. CSE software development has unique characteristics that differ from those of traditional software development. Some existing soft- ware engineering practices may not be suitable for CSE software development. Agile methods are gaining more interest in both industry and research fields, including the CSE application do- main. Agile methods focus on an incremental and iterative development process in which the requirements, specifications, design, implementation, and testing evolve throughout the project lifecycle. Test-driven development (TDD) and refactoring practices are critical for the success of Agile methods. Although many CSE projects employ Agile practices, the effect of TDD on CSE software development remains unknown and should thus be investigated. The empirical study is the primary research method by which the choice of a given soft- ware development technique should be justified. The empirical evidence obtained from software engineering research can be used by practitioners who are working to improve software develop- ment. The overall goal of this work is to provide the empirical evidence of the effects of TDD on CSE software development. Thus, I will perform a series of case studies and other empirical studies. I also propose a reverse engineering tool to assist developers in performing refactoring activities during project development. The details and results of these studies will be described in this dissertation. ii DEDICATION To the memory of my beloved father, his love and encouragement are always with me. iii ACKNOWLEDGMENTS I would like to thank my advisor, Dr. Jeffrey Carver, for his overwhelming support, direc- tion, and feedback throughout my Ph.D. program. His guidance provided me with the opportunity to learn from a recognized expert in the field of empirical software engineering. I would also like to thank committee members Dr. Jeff Gray, Dr. Karla Morris, Dr. Randy Smith, and Dr. Xi- aoyan Hong for their guidance in helping me complete the dissertation. I also gratefully thank Dr. Damian Rouson at Stanford University, Hope Michelsen member of the Combustion Chem- istry Department at Sandia National Laboratories, and Dr. David Hudak at the Ohio Supercomputer Center; their comments and helpful discussions were extremely valuable. I would like to thank the staff and developers at the Combustion Research Facility, Sandia National Laboratories and Ohio Supercomputer Center for their contributions to this research. My completion of the doctoral program also required the assistance of my classmates and school colleagues. I would like to thank the members of Software Engineering Research Group at University of Alabama for providing valuable feedback and suggestions and for offering me plentiful opportunities to present my research. I would especially like to thank my friends in the Empirical Software Engineering research group for their support and good friendship - we had great and brilliant times together. Finally, I thank Royal Thai Government Scholarship Program for generous financial support toward my Ph.D. program. iv CONTENTS ABSTRACT . ii DEDICATION . iii ACKNOWLEDGMENTS . iv LIST OF TABLES . ix LIST OF FIGURES . .x 1 INTRODUCTION . .1 1.1 Problem Statement . .3 1.2 Study Rationale . .7 1.3 Research Questions . .8 1.4 Outline of Dissertation . .9 2 BACKGROUND RESEARCH . 10 2.1 Empirical Study . 10 2.2 eXtreme Programming (XP) . 13 2.3 Test-Driven Development (TDD) . 18 2.4 Refactoring . 19 2.5 Design Patterns . 21 2.6 Object-Oriented Fortran . 24 3 METHODOLOGY . 27 3.1 Building the software tools for use in the case study . 28 v 3.1.1 The automated data collection tool . 29 3.1.2 Reverse engineering tool for object-oriented Fortran: ForUML . 30 3.2 Case Study: Test-Driven Development in the Community Laser-Induced Incandes- cence Modeling Environment (CLiiME) . 43 3.2.1 Study Rationale . 43 3.2.2 Research Questions . 44 3.2.3 CLiiME Description . 44 3.2.4 Procedure . 47 3.2.5 Data Collection . 49 3.2.6 Data Analysis . 50 3.3 Case Study: TDD in the Microscopy Imaging Processing project. 51 3.3.1 Study Rationale . 51 3.3.2 Research Questions . 52 3.3.3 Data Collection . 52 3.3.4 Data Analysis . 54 3.4 The survey of the effectiveness of TDD in the CSE community . 57 3.4.1 Study Rationale . 57 3.4.2 Research Questions . 57 3.4.3 Participants . 57 3.4.4 Procedure . 58 3.4.5 Data Collection . 59 3.4.6 Data Analysis . 60 4 RESULTS . 62 vi 4.1 ForUML . 62 4.1.1 Evaluation Results . 62 4.1.2 Limitations . 63 4.1.3 Lessons Learned . 66 4.2 A Case Study: Agile Development in the Community Laser-Induced Incandes- cence Modeling Environment (CLiiME) . 67 4.2.1 Findings and Results . 67 4.2.2 Lessons Learned . 76 4.3 The survey of the effectiveness of TDD in the CSE community . 81 4.3.1 Demographics . 83 4.3.2 Test-Driven Development . 86 4.3.3 Software Quality Characteristics . 88 4.3.4 Difficulty of employing TDD . 91 4.3.5 Testing . 94 4.3.6 Refactoring . 106 4.3.7 Benefits and Challenges of TDD . 120 4.3.8 Summary . 127 4.4 Case Study: TDD in the Microscopy Imaging Processing project. 131 4.4.1 The effectiveness of TDD . 131 4.4.2 Testing . 132 4.4.3 Refactoring . 133 4.4.4 Benefits and Disadvantages of TDD . 135 4.4.5 Summary . 135 vii 5 FINDINGS . 137 6 SUMMARY . 141 6.1 Summary . 141 6.2 Contribution . 144 6.3 Future studies . 145 6.4 Publications . 146 REFERENCES . 148 APPENDICES . 157 A FORUML - SCREENSHOTS . 158 B WEEKLY SURVEY . 160 C MONTHLY SURVEY . 165 D BACKGROUND QUESTIONNAIRE . 168 E SURVEY: TEST-DRIVEN DEVELOPMENT IN COMPUTATIONAL SCIENCE AND ENGINEERING . 171 F INSTITUTIONAL REVIEW BOARD CERTIFICATIONS . 180 viii LIST OF TABLES 2.1 Empirical study methods (adapted from [Pfleeger, 1995]) . 14 2.2 The characteristics of Agile development ([Miller, 2001]) and CSE software de- velopment . 15 2.3 The 29 rules of XP ([Don, 2009b]).......................... 17 2.4 Object-Oriented Fortran terms (adapted from [Rouson, Adalsteinsson, and Xia, 2010])......................................... 25 3.1 Fortran to XMI Conversion Rules . 39 3.2 The list of weekly questions . 55 3.3 The list of monthly questions . 56 3.4 The Survey Questions . 61 4.1 Evaluation of ForUML : recall (extracted data / actual data) . ..