Master Thesis Software Engineering Thesis no: MSE-2007:02 January 2007 Metrics in Software Test Planning and Test Design Processes Wasif Afzal School of Engineering Blekinge Institute of Technology Box 520 SE – 372 25 Ronneby Sweden This thesis is submitted to the School of Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 20 weeks of full time studies. Contact Information: Author(s): Wasif Afzal Address: Folkparksvägen 16: 14, 37240, Ronneby, Sweden. E-mail: [email protected] University advisor(s): Dr. Richard Torkar Assistant Professor Department of Systems and Software Engineering Blekinge Institute of Technology School of Engineering PO Box 520, SE – 372 25 Ronneby Sweden School of Engineering Internet : www.bth.se/tek Blekinge Institute of Technology Phone : +46 457 38 50 00 Box 520 Fax : + 46 457 271 25 SE – 372 25 Ronneby Sweden ii ABSTRACT Software metrics plays an important role in measuring attributes that are critical to the success of a software project. Measurement of these attributes helps to make the characteristics and relationships between the attributes clearer. This in turn supports informed decision making. The field of software engineering is affected by infrequent, incomplete and inconsistent measurements. Software testing is an integral part of software development, providing opportunities for measurement of process attributes. The measurement of software testing process attributes enables the management to have better insight in to the software testing process. The aim of this thesis is to investigate the metric support for software test planning and test design processes. The study comprises of an extensive literature study and follows a methodical approach. This approach consists of two steps. The first step comprises of analyzing key phases in software testing life cycle, inputs required for starting the software test planning and design processes and metrics indicating the end of software test planning and test design processes. After establishing a basic understanding of the related concepts, the second step identifies the attributes of software test planning and test design processes including metric support for each of the identified attributes. The results of the literature survey showed that there are a number of different measurable attributes for software test planning and test design processes. The study partitioned these attributes in multiple categories for software test planning and test design processes. For each of these attributes, different existing measurements are studied. A consolidation of these measurements is presented in this thesis which is intended to provide an opportunity for management to consider improvement in these processes. Keywords: Metrics, Attributes, Software Test Planning and Test Design. CONTENTS ABSTRACT .......................................................................................................................................... I CONTENTS .........................................................................................................................................II 1 INTRODUCTION .......................................................................................................................1 1.1 BACKGROUND........................................................................................................................1 1.2 PURPOSE ................................................................................................................................1 1.3 AIMS AND OBJECTIVES ..........................................................................................................1 1.4 RESEARCH QUESTIONS ..........................................................................................................2 1.4.1 Relationship between Research Questions and Objectives...............................................2 1.5 RESEARCH METHODOLOGY ...................................................................................................2 1.5.1 Threats to Validity ............................................................................................................3 1.6 THESIS OUTLINE ....................................................................................................................3 2 SOFTWARE TESTING..............................................................................................................5 2.1 THE NOTION OF SOFTWARE TESTING.....................................................................................5 2.2 TEST LEVELS .........................................................................................................................5 3 SOFTWARE TESTING LIFECYCLE......................................................................................8 3.1 THE NEED FOR A SOFTWARE TESTING LIFECYCLE.................................................................8 3.2 EXPECTATIONS OF A SOFTWARE TESTING LIFECYCLE ...........................................................9 3.3 SOFTWARE TESTING LIFECYCLE PHASES ...............................................................................9 3.4 CONSOLIDATED VIEW OF SOFTWARE TESTING LIFECYCLE..................................................11 3.5 TEST PLANNING ...................................................................................................................11 3.6 TEST DESIGN .......................................................................................................................14 3.7 TEST EXECUTION .................................................................................................................17 3.8 TEST REVIEW.......................................................................................................................18 3.9 STARTING/ENDING CRITERIA AND INPUT REQUIREMENTS FOR SOFTWARE TEST PLANNING AND TEST DESIGN PROCESSES ..........................................................................................................19 4 SOFTWARE MEASUREMENT..............................................................................................23 4.1 MEASUREMENT IN SOFTWARE ENGINEERING ......................................................................23 4.2 BENEFITS OF MEASUREMENT IN SOFTWARE TESTING..........................................................25 4.3 PROCESS MEASURES............................................................................................................25 4.4 A GENERIC PREDICTION PROCESS .......................................................................................27 5 ATTRIBUTES FOR SOFTWARE TEST PLANNING PROCESS......................................28 5.1 PROGRESS ............................................................................................................................28 5.1.1 The Suspension Criteria for Testing ...............................................................................29 5.1.2 The Exit Criteria .............................................................................................................29 5.1.3 Scope of Testing .............................................................................................................29 5.1.4 Monitoring of Testing Status ..........................................................................................29 5.1.5 Staff Productivity............................................................................................................29 5.1.6 Tracking of Planned and Unplanned Submittals.............................................................29 5.2 COST....................................................................................................................................29 5.2.1 Testing Cost Estimation..................................................................................................30 5.2.2 Duration of Testing.........................................................................................................30 5.2.3 Resource Requirements ..................................................................................................30 5.2.4 Training Needs of Testing Group and Tool Requirement...............................................30 5.3 QUALITY..............................................................................................................................30 5.3.1 Test Coverage .................................................................................................................30 5.3.2 Effectiveness of Smoke Tests .........................................................................................30 5.3.3 The Quality of Test Plan.................................................................................................31 5.3.4 Fulfillment of Process Goals...........................................................................................31 5.4 IMPROVEMENT TRENDS .......................................................................................................31 5.4.1 Count of Faults Prior to Testing......................................................................................31 ii 5.4.2 Expected Number of Faults ............................................................................................31 5.4.3 Bug Classification...........................................................................................................31 6 ATTRIBUTES FOR SOFTWARE TEST DESIGN PROCESS............................................33