PRACTICAL SOFTWARE TESTING Springer New York Berlin Heidelberg Hong Kong London Milan Paris Tokyo PRACTICAL SOFTWARE TESTING A PROCESS-ORIENTED APPROACH ILENE BURNSTEIN Ilene Burnstein Department of Computer Science Illinois Institute of Technology 10 West 31 Street Chicago, IL 60616 USA [email protected] Library of Congress Cataloging-in-Publication Data Burnstein, Ilene. Practical software testing : a process-oriented approach / Ilene Burnstein. p. cm. Includes bibliographical references and index. ISBN 0-387-95131-8 (hc : alk. paper) 1. Computer software—Testing. I. Title. QA76.76.T48 B87 2002 2002024164 005.1Ј4–dc21 ISBN 0-387-95131-8 Printed on acid-free paper. ᭧ 2003 Springer-Verlag New York, Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly anal- ysis. Use in connection with any form of information storage and retrieval, electronic adapta- tion, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Capability Maturity Model and CMM are registered trademarks of the Software Engineering Institute and Carnegie Mellon University. Testing Maturity Model and TMM are service marks of Illinois Institute of Technology. Printed in the United States of America. 987654321 SPIN 10779083 www.springer-ny.com Springer-Verlag New York Berlin Heidelberg Business Media GmbHםA member of BertelsmannSpringer Science CONTENTS Preface xv INTRODUCTION TO TESTING 1 AS AN ENGINEERING ACTIVITY 1.0 The Evolving Profession of Software Engineering 1 1.1 The Role of Process in Software Quality 4 1.2 Testing as a Process 6 1.3 Overview of the Testing Maturity Model (TMM) 8 1.3.1 TMM Levels 10 List of Key Terms 16 Exercises 16 References 17 2 TESTING FUNDAMENTALS 2.0 Introduction 19 2.1 Basic Definitions 19 vi | Contents 2.2 Software Testing Principles 26 2.3 The Tester’s Role in a Software Development Organization 34 List of Key Terms 36 Exercises 36 References 37 3 DEFECTS, HYPOTHESES, AND TESTS 3.0 Origins of Defects 39 3.1 Defect Classes, the Defect Repository, and Test Design 43 3.1.1 Requirements and Specification Defects 44 3.1.2 Design Defects 46 3.1.3 Coding Defects 48 3.1.4 Testing Defects 51 3.2 Defect Examples: The Coin Problem 51 3.3 Developer/Tester Support for Developing a Defect Repository 57 List of Key Terms 58 Exercises 58 References 59 4 STRATEGIES AND METHODS FOR TEST CASE DESIGN I 4.0 Introduction to Testing Design Strategies 61 4.1 The Smart Tester 62 4.2 Test Case Design Strategies 63 4.3 Using the Black Box Approach to Test Case Design 66 4.4 Random Testing 66 4.5 Equivalence Class Partitioning 67 4.6 Boundary Value Analysis 72 4.7 An Example of the Application of Equivalence Class Partitioning and Boundary Value Analysis 73 4.8 Other Black Box Test Design Approaches 76 4.8.1 Cause-and-Effect Graphing 78 4.8.2 State Transition Testing 82 4.8.3 Error Guessing 85 Contents | vii 4.9 Black Box Testing and Commercial Off-the-Shelf Components (COTS) 86 4.10 Black Box Methods and TMM Level 2 Maturity Goals 88 List of Key Terms 91 Exercises 92 References 95 5 STRATEGIES AND METHODS FOR TEST CASE DESIGN II 5.0 Using the White Box Approach to Test Design 97 5.1 Test Adequacy Criteria 98 5.2 Coverage and Control Flow Graphs 101 5.3 Covering Code Logic 103 5.4 Paths: Their Role in White Box–Based Test Design 108 5.5 Additional White Box Test Design Approaches 111 5.5.1 Data Flow and White Box Test Design 111 5.5.2 Loop Testing 115 5.5.3 Mutation Testing 116 5.6 Evaluating Test Adequacy Criteria 118 5.7 White Box Testing Methods and the TMM 124 List of Key Terms 127 Exercises 127 References 130 6 LEVELS OF TESTING 6.0 The Need for Levels of Testing 133 6.0.1 Levels of Testing and Software Development Paradigms 135 6.1 Unit Test: Functions, Procedures, Classes, and Methods as Units 137 6.2 Unit Test: The Need for Preparation 138 6.3 Unit Test Planning 139 6.4 Designing the Unit Tests 141 6.5 The Class as a Testable Unit: Special Considerations 142 6.6 The Test Harness 148 6.7 Running the Unit Tests and Recording Results 150 viii | Contents 6.8 Integration Test: Goals 152 6.9 Integration Strategies for Procedures and Functions 153 6.10 Integration Strategies for Classes 158 6.11 Designing Integration Tests 159 6.12 Integration Test Planning 162 6.13 System Test: The Different Types 163 6.13.1 Functional Testing 166 6.13.2 Performance Testing 167 6.13.3 Stress Testing 169 6.13.4 Configuration Testing 171 6.13.5 Security Testing 172 6.13.6 Recovery Testing 175 6.14 Regression Testing 176 6.15 Alpha, Beta, and Acceptance Tests 176 6.16 Summary Statement on Testing Levels 178 6.17 The Special Role of Use Cases 179 6.18 Levels of Testing and the TMM 181 List of Key Terms 184 Exercises 184 References 186 7 TEST GOALS, POLICIES, PLANS, AND DOCUMENTATION 7.0 Introductory Concepts 189 7.1 Testing and Debugging Goals and Policies 191 7.2 Test Planning 197 7.3 Test Plan Components 200 7.4 Test Plan Attachments 216 7.4.1 Test Design Specifications 217 7.4.2 Test Case Specifications 218 7.4.3 Test Procedure Specifications 220 7.5 Locating Test Items: The Test Transmittal Report 221 7.6 Reporting Test Results 221 7.7 The Role of the Three Critical Groups in Test Planning and Policy Development 226 Contents | ix 7.8 Process and the Engineering Disciplines: The Role of the Individual as a Process Facilitator 230 List of Key Terms 231 Exercises 231 References 232 8 THE TEST ORGANIZATION 8.0 Introducing the Test Specialist 235 8.1 Skills Needed by a Test Specialist 237 8.2 Building a Testing Group 240 8.3 The Structure of the Testing Group 242 8.4 The Technical Training Program 247 8.5 Career Paths for Testers: An Industry Example 250 8.6 Tester Certification 252 8.7 Integrating Testing Activities in the Software Life Cycle 253 8.8 The Test Organization, Technical Training Program, and Test Integration: Support from the Three Critical Views 257 Exercises 261 References 262 CONTROLLING AND MONITORING 9 THE TESTING PROCESS 9.0 Defining Terms 263 9.1 Measurements and Milestones for Controlling and Monitoring 266 9.1.1 Measurements for Monitoring Testing Status 271 9.1.2 Measurements for Monitoring Tester Productivity 275 9.1.3 Measurements for Monitoring Testing Costs 276 9.1.4 Measurements for Monitoring Errors, Faults and Failures 277 9.1.5 Monitoring Test Effectiveness 279 9.2 Status Meetings, Reports, and Control Issues 283 9.3 Criteria for Test Completion 289 9.4 Software Configuration Management 292 x | Contents 9.5 Controlling and Monitoring: Three Critical Views 296 List of Key Terms 300 Exercises 300 References 302 10 REVIEWS AS A TESTING ACTIVITY 10.0 Expanding the Testing Activity Umbrella 303 10.1 Types of Reviews 307 10.1.1 Inspections as a Type of Technical Review 308 10.1.2 Walkthroughs as a Type of Technical Review 310 10.2 Developing a Review Program 311 10.3 The Need for Review Policies 313 10.4 Components of Review Plans 314 10.4.1 Review Goals 315 10.4.2 Preconditions and Items to be Reviewed 315 10.4.3 Roles, Participants, Team Size, and Time Requirements 317 10.4.4 Review Procedures 320 10.4.5 Review Training 320 10.4.6 Review Checklists 324 10.5 Reporting Review Results 333 10.6 Review, Rework, and Follow-Up 337 10.7 Review Metrics 337 10.8 Support from the Extended/Modified V-Model 340 10.9 The Self-Check or Personal Review 340 10.10 Reviews and the TMM Critical Views 343 List of Key Terms 345 Exercises 345 References 347 A MEASUREMENT PROGRAM TO SUPPORT 11 PRODUCT AND PROCESS QUALITY 11.0 The Need for a Formal Test Measurement Program 349 11.1 Some Measurement-Related Definitions 353 Contents | xi 11.2 Initiating a Measurement Program 354 11.3 Software Quality Evaluation 364 11.4 Measurement and TMM Levels 372 11.4.1 Measurements for TMM Level 1 373 11.4.2 Measurements for TMM Level 2 375 11.4.3 Measurements for TMM Level 3 377 11.4.4 Measurements for TMM Level 4 381 11.4.5 Measurements for TMM Level 5 383 11.5 A Test Measurement Program, Software Quality Valuations and the Three Critical Views 386 List of Key Terms 389 Exercises 389 References 391 EVALUATING SOFTWARE QUALITY: 12 A QUANTITATIVE APPROACH 12.0 Review of Quality Concepts 393 12.1 Quality Costs 395 12.2 What Is Quality Control? 397 12.3 The Role of Operational Profiles and Usage Models in Quality Control 399 12.4 Support for Quality Control: Statistical Testing 407 12.5 Software Reliability 410 12.5.1 Measurements for Software Reliability 413 12.6 Reliability, Quality Control, and Stop-Test Decisions 414 12.6.1 Applying Reliability Models 417 12.7 Confidence Levels and Quality Control 422 12.8 Usability Testing and Quality Control 424 12.9 An Approach to Usability Testing 425 12.9.1 Exploratory Usability Testing 426 12.9.2 Assessment Usability Testing 427 12.9.3 Validation Usability Testing 427 12.9.4 Comparison Test 429 12.9.5 Usability Testing: Resource Requirements 429 12.9.6 Usability Tests and Measurements 430 xii | Contents 12.10 Software Quality Control and the Three Critical Views 433 List of Key Terms 436 Exercises 436 References 437 13 DEFECT ANALYSIS AND PREVENTION 13.0 Processes and Defects 439 13.1 History of Defect Analysis and Prevention