ABSTRACT GEORGE, BOBY, Analysis and Quantification of Test Driven Development Approach. (Under the direction of Dr. Laurie Ann Williams.) Software industry is increasingly becoming more demanding on development schedules and resources. Often, software production deals with ever-changing requirements and with development cycles measured in weeks or months. To respond to these demands and still produce high quality software, over years, software practitioners have developed a number of strategies. One of the more recent one is Test Driven Development (TDD). This is an emerging object-oriented development practice that purports to aid in producing high quality software quickly. TDD has been popularized through the Extreme Programming (XP) methodology. TDD proponents profess that, for small to mid- size software, the technique leads to quicker development of higher quality code. Anecdotal evidence supports this. However, until now there has been little quantitative empirical support for this TDD claim. The work presented in this thesis is concerned with a set of structured TDD experiments on very small programs with pair programmers. Programmers were both students and professionals. In each programmer category (students and professionals), one group used TDD and the other (control group) a waterfall-like software development approach. The experiments provide some interesting observations regarding TDD. When TDD was used, both student and professional TDD developers appear to achieve higher code quality, as measured using functional black box testing. The TDD student pairs passed 16% more test cases while TDD professional pair passed 18% more test cases than the their corresponding control groups. However, professional TDD developer pairs did spent about 16% more time on development. It was not established whether the increase in the quality was due to extra development time, or due to the TDD development process itself. On the other hand, the student experiments were time-limited. Both the TDD and the non-TDD student programmers had to complete the assignment in 75 minutes. Professional programmers took about 285 minutes on the average, to complete the same assignment. Consequently, the development cycle of the student-developed software was severely constrained and the resulting code was underdeveloped and of much poorer quality than the professional code. Still, it is interesting to note that the code developed using the TDD approach under these severe restrictions appears to be less faulty than the one developed using the more classical waterfall-like approach. It is conjectured that this may be due to the granularity of the TDD process, one to two test cases per feedback loop, which may encourage more frequent and tighter verification and validation episodes. These tighter cycles may result in a code that is better when compared to that developed by a coarser granularity waterfall-like model. As part of the study, a survey was conducted of the participating programmers. The majority of the programmers thought that TDD was an effective approach, which improved their productivity. Analysis and Quantification of Test Driven Development Approach By Boby George A thesis submitted to the Graduate Faculty of North Carolina State University in partial fulfillment of the requirements for the Degree of Master of Science COMPUTER SCIENCE Raleigh 2002 APPROVED BY: ___________________________________ Chair of Advisory Committee PERSONAL BIOGRAPHY Boby George is a graduate student in the Department of Computer Science, North Carolina State University. In 1999, he got his Bachelor of Technology in Computer Science and Engineering from University of Kerala, India. After his graduation, he worked as a system executive at eFunds International India Private Limited. His responsibilities included the automation of the testing process and preparation of test cases. In his graduate study, from August 2001 at NC State, Boby George focused on agile software development methodologies, in particular the Test Driven Development approach. ii ACKNOWLEDGEMENTS It is the hard work and contribution of many and not one that made this work possible. First and foremost, thanks to my research committee members, in particular the chair, Dr. Laurie Williams for proposing the topic and for all your persistent effort and guidance. A special thanks to Dr. Mladen Vouk for all your detailed review and suggestions that made this thesis work more accurate and to Dr. Aldo Dagnino, thank you for your keen interest in my research work. To the Fall 2001 undergraduate software engineering students of North Carolina State University, the John Deere, RoleModel Software, and Ericsson developers, thank you for participating in the long and strenuous experiments. Also, I express my gratitude to Ken Auer for arranging the RoleModel Software experiment and giving valuable insight, to Doug Taylor for arranging John Deere experiment, and to Lisa Woodring for arranging the Ericsson experiment. Lastly, I express my appreciation to AT&T who provided the research funding for this work. iii TABLE OF CONTENTS LIST OF TABLES ................................................................................................ vi LIST OF FIGURES ............................................................................................. vii 1. INTRODUCTION ..............................................................................................1 1.1 Research motivation ....................................................................................1 1.2 Historical perspective...................................................................................2 1.3 Research approach......................................................................................5 1.4 Thesis layout................................................................................................7 2. RELATED WORK .............................................................................................8 2.1 Extreme Programming (XP).........................................................................8 2.2 Unit Testing..................................................................................................9 2.3 Software Models ........................................................................................11 2.4 Test Driven Development (TDD)................................................................12 2.5 Differences between TDD and other models .............................................14 2.6 Refactoring ................................................................................................16 2.7 Pair programming ......................................................................................17 2.8 Object Oriented Metrics .............................................................................18 3. TESTING METHODOLOGIES........................................................................22 3.1 Testing methodology in OO Development .................................................22 3.2 Unit testing.................................................................................................24 3.3 Various Types of Testing ...........................................................................27 3.3 Testability...................................................................................................28 3.4 Mean Time Between Failure ......................................................................30 3.5 Code coverage...........................................................................................30 3.6 Limitations of testing process.....................................................................32 4. TEST DRIVEN DEVELOPMENT ....................................................................33 4.1 Traditional OOP approaches .....................................................................33 4.1.1 Limitations of traditional OO development approaches........................36 4.2 TDD Explained...........................................................................................37 4.2.1 TDD without High/Low Level Design....................................................40 4.2.2 Evolutionary software process models and TDD .................................41 4.2.3 Reusability, Design Patterns and TDD.................................................43 5. EXPERIMENT DESIGN..................................................................................45 5.1 Basics of Software Engineering Experimentation ......................................45 5.2 Basics of Statistics.....................................................................................46 5.2.1 Measures of Central Tendency............................................................47 5.2.2 Measures of Variability.........................................................................47 5.2.3 Box Plots..............................................................................................48 5.3 Statistical significant analysis.....................................................................48 5.3.1 Normal approximation..........................................................................49 5.3.2 Spearman’s Rho Test ..........................................................................50 6. EXPERIMENTAL RESULTS...........................................................................51 6.1 Experiment Details.....................................................................................51 6.2 External Validity .........................................................................................52