ISSN:1353-7776 DESMET: A method for evaluating Software Engineering methods and tools Barbara Kitchenham Technical Report TR96-09 August 1996 Department of Computer Science University of Keele Keele Staffordshire ST5 5BG U.K. TEL: 01782 583075 FAX: 01782 713082 Abstract This report describes the results of the DESMET project. The DESMET project was a collaborative project part funded by the U.K. DTI. Its goal was develop and validate a method for evaluating software engineering methods and tools. This report describes the guidelines developed by DESMET for • selecting an appropriate evaluation method; • performing a feature analysis; • performing a quantitative case study. It also describes some of the management and sociological issues that can influence evaluation exercise. Finally, the report describes briefly the attempt the project made to evaluate the DESMET method itself. CONTENTS 1. INTRODUCTION..............................................................................................................................................1 2. THE DESMET METHOD: SCOPE, TERMINOLOGY AND LIMITATIONS..........................................2 2.1 DESMET USERS.................................................................................................................................................2 2.2 EVALUATION CONTEXT.......................................................................................................................................2 2.3 EVALUATION OBJECTS........................................................................................................................................2 2.4 EVALUATION TYPES ............................................................................................................................................2 2.5 EVALUATION PROCEDURE...................................................................................................................................3 2.6 LIMITATIONS OF THE DESMET METHOD............................................................................................................3 3. EVALUATION METHODS .............................................................................................................................4 3.1 QUANTITATIVE EVALUATION METHODS..............................................................................................................4 3.1.1 Case Studies ...............................................................................................................................................4 3.1.1.1 Advantages of case studies....................................................................................................................................4 3.1.1.2 Disadvantages of case studies ...............................................................................................................................4 3.1.2 Surveys .......................................................................................................................................................5 3.1.2.1 Advantages of surveys ..........................................................................................................................................5 3.1.2.2 Disadvantages of surveys......................................................................................................................................5 3.2 QUALITATIVE METHODS.....................................................................................................................................5 3.3 HYBRID EVALUATION METHODS ........................................................................................................................6 3.3.1 Collated expert opinion - Qualitative Effects Analysis...............................................................................6 3.3.2 Benchmarking.............................................................................................................................................6 3.4 SUMMARY OF EVALUATION METHODS ................................................................................................................6 4. SELECTING AN APPROPRIATE EVALUATION METHOD...................................................................7 4.1 TECHNICAL SELECTION CRITERIA........................................................................................................................7 4.1.1 Introduction................................................................................................................................................7 4.1.2 The evaluation context................................................................................................................................7 4.1.3 The nature of the impact.............................................................................................................................7 4.1.4 Nature of evaluation object ........................................................................................................................8 4.1.5 The scope of impact....................................................................................................................................9 4.1.5.1 Product granularity................................................................................................................................................9 4.1.5.2 Extent of impact....................................................................................................................................................9 4.1.6 The maturity of the item............................................................................................................................10 4.1.7 Learning time ...........................................................................................................................................10 4.1.8 Evaluation maturity of an organisation....................................................................................................10 4.1.9 Summary of Evaluation criteria ...............................................................................................................11 4.2 PRACTICAL ISSUES ............................................................................................................................................12 4.2.1 Evaluation Timescales..............................................................................................................................12 4.2.2 Confidence in results ................................................................................................................................13 4.2.3 Costs of an evaluation ..............................................................................................................................14 4.2.3.1 Costs of Feature Analysis....................................................................................................................................14 4.2.3.2 Costs of Qualitative Effects Analysis..................................................................................................................15 4.2.3.3 Costs of Benchmarking.......................................................................................................................................15 4.2.3.4 Costs of a Quantitative Case Study .....................................................................................................................15 4.2.3.5 Costs of a Quantitative Formal Experiment ........................................................................................................15 4.2.3.6 Costs of a Quantitative Survey............................................................................................................................16 4.2.4 External Evaluation..................................................................................................................................16 5. THE INFLUENCE OF HUMAN FACTORS................................................................................................16 5.1 INTRODUCTION .................................................................................................................................................16 5.2 IMPACT OF HUMAN FACTORS.............................................................................................................................16 5.3 SOCIOLOGICAL EFFECTS....................................................................................................................................17 5.3.1 Novelty Effects..........................................................................................................................................17 5.3.1.1 Learning curve effect ..........................................................................................................................................17 5.3.1.2 The Hawthorne effect..........................................................................................................................................17 5.3.2 Expectation effects....................................................................................................................................18 5.3.2.1 The placebo effect...............................................................................................................................................18 5.3.2.2 The doctor effect.................................................................................................................................................18 5.3.3 Summary...................................................................................................................................................19