University of South Florida Scholar Commons Graduate Theses and Dissertations Graduate School 2005 Understanding ROI Metrics for Software Test Automation Naveen Jayachandran University of South Florida Follow this and additional works at: https://scholarcommons.usf.edu/etd Part of the American Studies Commons Scholar Commons Citation Jayachandran, Naveen, "Understanding ROI Metrics for Software Test Automation" (2005). Graduate Theses and Dissertations. https://scholarcommons.usf.edu/etd/2938 This Thesis is brought to you for free and open access by the Graduate School at Scholar Commons. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please contact [email protected]. Understanding ROI Metrics for Software Test Automation by Naveen Jayachandran A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science Department of Computer Science and Engineering College of Engineering University of South Florida Co-Major Professor: Dewey Rundus, Ph.D. Co-Major Professor: Alan Hevner, Ph.D. Member: Rafael Perez, Ph.D. Date of Approval: June 2, 2005 Keywords: Return on investment metrics, automated testing feasibility, manual vs. automated testing, software quality assurance, regression testing, repetitive testing © Copyright 2005, Naveen Jayachandran Dedication To my Parents, Suja, Rocky and Reemo. Acknowledgements I would like to thank my major professors Dr. Alan Hevner and Dr. Dewey Rundus for recognizing and encouraging my interest, giving me this opportunity and providing constant support for my research. I have gained immensely from this experience enabling me to truly appreciate the saying – ‘The journey is more important than the destination’. I would like to acknowledge my faculty committee member, Dr. Rafael Perez for his time and valuable comments. This work was supported in part by the National Institute for Systems Test and Productivity (NISTP) at the University of South Florida under the US Space and Naval Warfare Systems Command Contract No. N00039-02-C-3244. I would like to thank Tom Barr, Judy Hyde and the rest of the NISTP team for their useful insights and thoughts. I also acknowledge IBM Rational for providing the Rational Test Suite that has been used extensively in this research study. Finally, I thank my parents and my sister for being my source of motivation and hope. Table of Contents List of Tables ..................................................................................................................... iv List of Figures..................................................................................................................... v Abstract.............................................................................................................................. vi Chapter 1: Introduction....................................................................................................... 1 1.1 Prelude ................................................................................................................ 1 1.2 The Problem........................................................................................................ 2 1.3 The Goals............................................................................................................ 2 1.4 The Questions ..................................................................................................... 3 1.5 The Metrics ......................................................................................................... 3 1.6 The Experiment................................................................................................... 4 Chapter 2: Literature Review.............................................................................................. 5 2.1 Software Testing ................................................................................................. 5 2.1.1 Manual Testing ............................................................................................ 5 2.1.2 Automated Testing....................................................................................... 6 2.2 Metrics ................................................................................................................ 6 2.2.1 Metrics for Software Testing ....................................................................... 6 2.3 Return on Investment (ROI) ............................................................................... 7 2.4 Related Work ...................................................................................................... 8 i 2.4.1 COCOMO II ................................................................................................ 8 2.4.2 Rational Suite Economic Value Model........................................................ 9 Chapter 3: Experimental Design....................................................................................... 10 3.1 The Experiment................................................................................................. 10 3.1.1 Print Dialog Testing (P1)........................................................................... 12 3.1.2 File Dialog Testing (P2)............................................................................. 12 3.2 The Metrics ....................................................................................................... 13 3.3 Data Collection ................................................................................................. 14 3.4 Other Details ..................................................................................................... 15 Chapter 4: Results & Discussion ...................................................................................... 16 4.1 Demographic Data ............................................................................................ 16 4.2 Schedule............................................................................................................ 18 4.2.1 Regression Testing..................................................................................... 21 4.2.2 Pure Repetitive Testing.............................................................................. 24 4.3 Cost ................................................................................................................... 31 4.3.1 Fixed Costs................................................................................................. 31 4.3.2 Variable Costs............................................................................................ 32 4.3.3 Regression Testing..................................................................................... 35 4.3.4 Pure Repetitive Testing.............................................................................. 35 4.3.5 Accounting for Fixed Costs ....................................................................... 35 4.4 Effectiveness..................................................................................................... 37 4.4.1 Lessons Learned......................................................................................... 41 4.5 Other Metrics .................................................................................................... 41 ii 4.5.1 Scope of Testing ........................................................................................ 42 4.5.2 Depth of Testing ........................................................................................ 43 4.5.3 Test Tool Adoption.................................................................................... 45 4.5.4 User Ratings for Effectiveness & Productivity.......................................... 48 4.5.5 Documentation and Traceability................................................................ 49 4.5.6 Automation – Individual Vs. Group Effort................................................ 50 4.5.7 Test Coverage ............................................................................................ 50 4.6 Tripod SDLC Model......................................................................................... 51 4.6.1 Automation Requirements ......................................................................... 52 4.6.2 Automation Feasibility............................................................................... 53 4.6.3 Framework Design..................................................................................... 53 4.7 Contributions..................................................................................................... 53 4.8 Limitations ........................................................................................................ 54 4.8.1 Project Scope ............................................................................................. 54 4.8.2 Error Seeding ............................................................................................. 55 4.8.3 Automation Tool........................................................................................ 55 Chapter 5: Conclusion....................................................................................................... 56 5.1 Future Research ................................................................................................ 57 References......................................................................................................................... 59 Appendices.......................................................................................................................