Performance Measurements Designing a Generic Measure and Performance Indicator Model Master Thesis submitted by Othmar Heini for the degree of Master of Science in Communication and Information Systems Orientation Information System Technologies Under the supervision of Prof. D. Konstantas, M. Pawlak and K. Wac Geneva, 2007 ii Acknowledgement First of all, I wish to thank Professor Dimitri Konstantas and Professor Michel Léonard for their precious guidance and advice over the years that I have studied at the Depart- ment of Information Systems of the University of Geneva. Special thanks are due to Michel Pawlak, who kept me going in this research, and to Katarzyna Wac for her valuable comments and advice. You are a true researcher! I would like to thank all my fellow students, especially Wanda and Guillaume, as well as all the members of the ASG and MATIS team for their joyful company. I would also like to thank my family and all my friends who have both encouraged and supported me throughout this project. Finally, I wish to thank Fabienne, for her patience and understanding. iii iv Abstract Performance measurement has become a major issue in recent years. The ability of an organisation to measure its vital activities, and this at all organisational levels, has indeed become critical to its success in today’s fast-paced world. However, current knowledge in the field of performance measurement is still insufficiently translated into practice. For instance, performance indicators are rarely linked to the objectives and the overall vision of the organisation. Time-critical activities are often not supported by current measurement systems, and data from external sources is not sufficiently taken into consideration. Such issues are to some extend due to the absence of a comprehen- sive, overall model. Only few models have been proposed in literature, most of which lack detail, or do not consider all relevant aspects. In this research, we present a generic measure and performance indicator model that incorporates the essential aspects of the performance measurement process in a single, comprehensive model. Based on an extensive literature review in the fields of mea- surement and performance measurement, we identify key concepts and discuss major theories and frameworks. From our findings, we derive a set of requirements, which we translate into our model. Finally, we propose a generic architecture for performance measurement systems, and present a prototype application which builds upon our model proposal. v vi Contents Acknowledgement iii Abstract v 1. Introduction 1 1.1. Introduction to Measurement and Performance Measurement . 1 1.2. Motivation . 2 1.3. Objective . 6 1.4. Structure . 6 2. Measurement 9 2.1. The Nature of Measurement . 9 2.1.1. What is a Measurement? . 9 2.1.2. Absolute and Relative Measurement . 11 2.1.3. Methods of Measurement . 12 2.1.4. Measurement Theory . 13 2.2. Fundamental Measurement . 13 2.2.1. Formalisation . 14 2.2.2. Relational Systems and Homomorphisms . 15 2.3. Scales . 16 2.3.1. Regular Scales . 16 2.3.2. Scale Types . 17 2.3.3. Meaningful and Meaningless Statements . 20 2.4. Derived Measurement . 21 2.4.1. An Approach to Derived Measurement . 21 2.4.2. Summary Operations . 23 2.5. Quality of Measurement . 25 2.6. Limits of Measurement . 27 2.6.1. Abstract Concepts . 27 2.6.2. Measurement and Human Behaviour . 28 2.6.3. Fundamental Uncertainty . 29 3. Performance Measurement 31 3.1. Measuring Performance? . 31 3.2. From Management Accounting to Performance Measurement . 33 3.3. Performance Indicators . 34 3.3.1. Classifications . 35 vii 3.3.2. Structures and Relationships . 37 3.3.3. Aggregation of Performance Indicators . 39 3.4. Targets and Ratings . 41 3.4.1. Static and Dynamic Targets . 41 3.4.2. Ratings . 41 3.5. Strategy, Objectives and Initiatives . 43 3.5.1. From Vision to Performance Indicators . 43 3.5.2. Structures and Levels . 44 3.5.3. Quantitative and Qualitative Objectives . 46 3.5.4. Initiatives . 46 3.6. Further Issues in Performance Measurement . 47 3.7. The Balanced Scorecard Framework . 49 3.7.1. Four Perspectives . 50 3.7.2. Cause-and-Effect Relationships . 52 3.7.3. Four Processes . 53 3.7.4. Extensions and Adaptations . 54 4. Existing Measure and Performance Indicator Models 55 4.1. Performance Measure Record Sheet by Neely et al. 55 4.2. Metric Definition Template by Lohman et al. 57 4.3. Security-metric Description of the ISM3 . 58 4.4. KPI Profiler by Bauer . 59 4.5. Balanced Scorecard XML Draft Standard . 60 4.6. Measurement Specification Template of the PSM . 62 4.7. Software Measurement Metamodel of the FMESP Framework . 63 5. Requirements on a Measure and Performance Indicator Model 67 5.1. Requirements and Requirements Elicitation . 67 5.2. The Requirements . 68 5.2.1. Requirements related to Measures . 68 5.2.2. Requirements related to Performance Indicators . 69 5.2.3. Requirements related to Strategy and Performance . 71 6. Measure and Performance Indicator Model 73 6.1. Models and Model-driven Engineering . 73 6.1.1. Abstraction and Models . 73 6.1.2. Model-Driven Engineering and MDA . 75 6.1.3. Class Discovery Techniques . 76 6.2. The Model . 77 6.2.1. Model Proposal . 78 6.2.2. Class Descriptions . 82 6.2.3. Design Choices and Patterns . 89 6.2.4. Rules and Constraints . 92 viii 7. Performance Measurement System Prototype Application 95 7.1. Architecture Proposal . 95 7.1.1. Overview . 96 7.1.2. Client-Tier . 97 7.1.3. Middle-Tier . 98 7.1.4. Resource-Tier . 100 7.2. The Prototype Application . 101 7.2.1. Description and Use Cases . 101 7.2.2. Technical Aspects . 104 7.2.3. Scenario . 107 7.2.4. Evaluation . 109 8. Conclusion and Outlook 111 Bibliography 122 A. Measure Samples 123 B. Performance Measurement Frameworks 129 C. Performance Measurement Systems 131 C.1. Academic Systems . 131 C.2. Commercial and Open-Source Systems . 132 D. Prototype Artefacts 135 ix x List of Figures 2.1. The relational model [103] . 15 2.2. An analogy to validity and reliability [61] . 26 3.1. The performance measurement matrix (Keegan et al. in [90]) . 36 3.2. The performance pyramid (Lynch and Cross in [116]) . 37 3.3. Factors affecting ‘total cost per unit’ and their relationships [112] . 38 3.4. Hierarchy of performance indicators [76] . 40 3.5. Performance indicator control chart [9] . 42 3.6. Strategic alignment pyramid [7] . 44 3.7. Success map [88] . 45 3.8. Hierarchical decompositions of an objective (based on [12]) . 46 3.9. The four perspectives of the balanced scorecard [63] . 51 3.10. Cause-and-effect relationships [63] . 52 3.11. The four processes of the balanced scorecard [63] . 53 4.1. KPI profiler template [6] . 60 4.2. Balanced Scorecard XML schema [3] . 61 4.3. Software measurement metamodel [45] . 64 6.1. Model taxonomy [42] . 74 6.2. General view of the MDA approach [17] . 76 6.3. Measure and Performance Indicator Model . 79 7.1. Architecture proposal for a performance measurement system . 96 7.2. A sample sales dashboard [38] . 98 7.3. Screenshots of the mobile phone client application . 103 7.4. Architecture of the prototype application . 105 7.5. Sequence diagram of an example scenario . 108 D.1. Database schema of the metadata repository . 136 xi xii List of Tables 2.1. Definitions of the term ‘measurement’ . 10 2.2. Definitions of the term ‘metric’ . 11 2.3. Common scale types (based on [105]) . 18 3.1. Definitions of the term ‘performance measurement’ . 32 3.2. Definitions of the term ‘performance indicator’ . 32 3.3. Definitions of the term ‘performance measurement system’ . 33 4.1. Performance measure record sheet [91] . 56 4.2. Metric definition template [76] (based on [91]) . 58 4.3. Security-metric description of the ISM3 [25] . 59 A.1. Measure samples . 123 B.1. Performance measurement frameworks . 129 C.1. Academic performance measurement systems . 131 C.2. Commercial and open-source performance measurement systems . 132 xiii xiv Chapter 1. Introduction 1.1. Introduction to Measurement and Performance Measurement Measurements are processes that we accomplish on a regular basis, and in a variety of domains and circumstances. When leaving for work for example, we may glance at the thermometer in order to decide whether or not we should take our coat; a craftsman constantly controls the size of the piece he is working on to make sure it fits together with the others; a sales manager periodically checks the sales-volume of a product or service in order to compare it with forecasts. The central idea in measurement is the notion of representation, and more precisely ‘the translation of “qualitative” concepts such as relations and operations into appropriate “quantitative” [. ] relations and operations enjoying known properties’ [105]. In mea- suring weight for example, we seek to assign numbers to the weight property of objects in a way that relations such as ‘heavier than’ and ‘lighter than’, and operations such as addition or subtraction remain preserved. Measurement of objects and phenomena is an important aspect in all sciences, for it is through observation, generalisation and mea- surement that theories can be derived. For instance, where.