Walden University ScholarWorks Walden Dissertations and Doctoral Studies Walden Dissertations and Doctoral Studies Collection 2015 The ffecE t of Applying Design of Experiments Techniques to Software Performance Testing Gloria Johnson Walden University Follow this and additional works at: https://scholarworks.waldenu.edu/dissertations Part of the Business Administration, Management, and Operations Commons, and the Management Sciences and Quantitative Methods Commons This Dissertation is brought to you for free and open access by the Walden Dissertations and Doctoral Studies Collection at ScholarWorks. It has been accepted for inclusion in Walden Dissertations and Doctoral Studies by an authorized administrator of ScholarWorks. For more information, please contact [email protected]. Walden University College of Management and Technology This is to certify that the doctoral dissertation by Gloria Johnson has been found to be complete and satisfactory in all respects, and that any and all revisions required by the review committee have been made. Review Committee Dr. Branford McAllister, Committee Chairperson, Applied Management and Decision Sciences Faculty Dr. Robert Kilmer, Committee Member, Applied Management and Decision Sciences Faculty Dr. Christos Makrigeorgis, University Reviewer Applied Management and Decision Sciences Faculty Chief Academic Officer Eric Riedel, Ph.D. Walden University 2015 Abstract The Effect of Applying Design of Experiments Techniques to Software Performance Testing by Gloria Johnson MBA, University of Dallas, 1995 BS, Jackson State University, 1975 Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Applied Management and Decision Sciences Walden University January 2015 Abstract Effective software performance testing is essential to the development and delivery of quality software products. Many software testing investigations have reported software performance testing improvements, but few have quantitatively validated measurable software testing performance improvements across an aggregate of studies. This study addressed that gap by conducting a meta-analysis to assess the relationship between applying Design of Experiments (DOE) techniques in the software testing process and the reported software performance testing improvements. Software performance testing theories and DOE techniques composed the theoretical framework for this study. Software testing studies (n = 96) were analyzed, where half had DOE techniques applied and the other half did not. Five research hypotheses were tested, where findings were measured in (a) the number of detected defects, (b) the rate of defect detection, (c) the phase in which the defect was detected, (d) the total number of hours it took to complete the testing, and (e) an overall hypothesis which included all measurements for all findings. The data were analyzed by first computing standard difference in means effect sizes, then through the Z test, the Q test, and the t test in statistical comparisons. Results of the meta-analysis showed that applying DOE techniques in the software testing process improved software performance testing (p < 05). These results have social implications for the software testing industry and software testing professionals, providing another empirically-validated testing methodology. Software organizations can use this methodology to differentiate their software testing process, to create more quality products, and to benefit the consumer and society in general. The Effect of Applying Design of Experiments Techniques to Software Performance Testing by Gloria Johnson MBA, University of Dallas, 1995 BS, Jackson State University, 1975 Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Applied Management and Decision Sciences Walden University January 2015 Dedication I dedicate this study to my faithful champion, my son, Kevin. I am deeply grateful for the tenacity and consistency of your encouragement and support. Acknowledgments First and foremost, all the glory, honor, and praise go to God for getting me through this program. I would like to acknowledge and thank my committee for all their guidance and patience throughout this process. Thank you to Dr. Branford McAllister, Dr. Robert Kilmer, and Dr. Christos Makrigeorgis. Thank you Dr. Teresa Bittner for you encouragement as I started this journey. I am deeply grateful to Cheryl Lockett for the help, encouragement, and unwavering friendship throughout this doctoral journey. To my Walden colleagues, thank you for your encouragement, advice, and support during our time together on this doctoral journey. To my son, my number one supporter, my extended family, friends, and church family, my deepest gratitude for your prayers and unfailing support. Finally, thank you Walden faculty and staff for your expertise, advice, and guidance. Table of Contents List of Tables ..................................................................................................................... vi List of Figures .................................................................................................................. viii Chapter 1: Introduction to the Study ....................................................................................1 Background ....................................................................................................................1 Purpose of the Study ......................................................................................................8 Nature of the Study ........................................................................................................9 Research Questions and Hypotheses ...........................................................................10 Research Questions ............................................................................................... 11 Research Hypotheses ............................................................................................ 12 Theoretical Bases for the Study ...................................................................................14 Definitions of Terms and Acronyms............................................................................15 Assumptions, Limitations, Scope, and Delimitations ..................................................19 Assumptions .......................................................................................................... 19 Limitations ............................................................................................................ 19 Scope ................................................................................................................... 19 Delimitations ......................................................................................................... 20 Significance of the Study .............................................................................................20 Gap in Current Literature ...................................................................................... 21 Professional Application ....................................................................................... 21 Implications for Positive Social Change ............................................................... 22 Summary ......................................................................................................................22 i Chapter 2: Literature Review .............................................................................................24 Introduction ..................................................................................................................24 Literature Search ..........................................................................................................25 Search Strategy ..................................................................................................... 25 General Theortical Concepts........................................................................................27 Software Testing ................................................................................................... 27 DOE Strategies...................................................................................................... 32 Meta-Analysis ....................................................................................................... 38 Literature Examination and Analysis...........................................................................49 Early Prior Research ............................................................................................. 49 Current Research ................................................................................................... 50 Variables of Interest .............................................................................................. 55 Summary ......................................................................................................................56 Chapter 3: Methodology and Design .................................................................................59 Introduction ..................................................................................................................59 Target Population .........................................................................................................60 Sampling Design ..................................................................................................