01 Software Measurement Body of Knowledge 57 02 58
03 59
04 60 Alain Abran 05 61 Alain April 06 62 Software Engineering and IT Department, ETS (E´cole de technologie supe´rieure) University, 07 63 Montreal, Quebec, Canada 08 64 09 Luigi Buglione 65 10 66 Industry and Services Business Unit, Engineering.IT SPA, Rome, Italy 11 67
12 68
13 69
14 Abstract 70 15 Measurement is fundamental to sciences and to the engineering disciplines. In the 2004 version of the Guide 71 16 to the Software Engineering Body of Knowledge—the SWEBOK Guide—the software measurement topic is 72 17 dispersed throughout the Guide and discussed in every knowledge area. To facilitate and improve teaching 73 18 and use of measurement in software engineering, an integrated and more comprehensive view of software 74 19 measurement has been built in the form of a software measurement body of knowledge. This entry presents 75
20 this integrated view on software measurement. In the 2010 version of the SWEBOK Guide, it has been 76
21 proposed that software measurement be assigned its own knowledge area. 77
22 78
23 79
24 INTRODUCTION associated with it. Appropriate references are also given 80
25 for each of the topics. Fig. 1 presents a graphical represen- 81
26 Measurement is fundamental to sciences and to the engi- tation of the top-level decomposition of the breakdown of 82
27 neering disciplines. Similarly, the importance of measure- measurement topics. 83
28 ment and its role in better management practices is widely 84
29 acknowledged. Effective measurement has become one of 85
30 the cornerstones of increasing organizational maturity. BASIC CONCEPTS 86
31 Measurement can be performed to support the initiation 87
32 of process implementation and change or to evaluate the This section presents the foundations of software measure- 88 33 consequences of process implementation and change, or it ment, the main definitions, concepts, as well as software 89 34 can be performed on the product itself. information models. 90
35 At the inception of the Software Engineering Body of 91 [1] 36 Knowledge project, measurement activities had been Foundations 92
37 assigned to all the Knowledge Areas (KA) associate editors 93
38 as a criterion for identifying relevant measurement-related In sciences and engineering disciplines, it is the domain of 94 39 knowledge in their respective KA. It is therefore a common knowledge referred to as “metrology” that focuses on the 95 40 theme present across most of the software engineering activ- development and use of measurement instruments and 96 41 ities. However, no attempt had been done to ensure that the measurement processes. The international consensus on 97 42 measurement-related topic had had a full coverage. This entry metrology is documented in the ISO vocabulary of basic 98 [2] 43 presents both an integrated view of the measurement-related and general terms in metrology. The ISO vocabulary 99 44 knowledge spread throughout all other chapters of the defines 144 measurement-related terms. They are orga- 100 45 SWEBOK as well as an improved coverage of the information nized into the following five categories: 101
46 about measurement, as it is actually understood in software 102
47 engineering. The information presented here on software 1. Quantities and units 103 48 measurement has been proposed for inclusion in the 2010 2. Measurements 104 49 version of the Software Engineering Body of Knowledge to 3. Devices for measurement 105 50 be published jointly by the Computer Society and IEEE. 4. Properties of measurement devices 106 51 5. Measurement standards (etalons) 107 52 BREAKDOWN OF TOPICS FOR 108 53 SOFTWARE MEASUREMENT To represent the relationships across the elements of these 109 54 five categories, the classical representation of a production 110 55 The breakdown for software measurement is presented in process is selected, e.g., input, output, and control vari- 111 56 Fig. 1, together with brief descriptions of the major topics ables, as well as the process itself inside. In Fig. 2, the 112
Encyclopedia of Software Engineering DOI: 10.1081/E-ESE-120044182 Copyright # 2011 by Taylor & Francis. All rights reserved. 1 2 Software Measurement Body of Knowledge
01 57
02 58
03 59
04 60
05 61
06 62
07 63
08 64
09 65
10 66
11 67
12 68
13 69
14 70
15 71
16 72
17 73
18 74
19 75
20 76
21 77
22 78
23 79
24 80
25 81
26 82
27 83
28 84
29 85
30 86
31 87
32 88
33 89
34 90
35 91
36 92
37 93
38 94
39 95
40 Fig. 1 Breakdown of topics for the software measurement KA. 96
41 97
42 98
43 output is represented by the “measurement results” and the 99
44 process itself by the “measurement” in the sense of mea- 100
45 surement operations, while the control variables are the 101
46 “etalons” and the “quantities and units.” This set of con- 102
47 cepts then represents the “measuring devices,” and the 103
48 measurement operations are themselves influenced as 104
49 well by the “properties of measurement devices.” 105
50 As in any new domain of application, empirical trials 106
51 represent the starting point to develop a measure, not 107
52 necessarily following a rigorous process. But after a com- 108
53 munity of interest accepts a series of measures to quantify a 109
54 concept (or a set of concepts through a modeling of the 110 55 Fig. 2 Model of the categories of metrology terms in the ISO relationships across the concepts), it is usual to ascertain 111 56 terminology. that the proposed measures are verified and validated. 112 Software Measurement Body of Knowledge 3
01 Definitions and Concepts operations performed on it. It can also be defined 57 02 through the selection of one scale type (ordinal, 58 03 In sciences and engineering disciplines, it is the domain of interval, or ratio). This also includes the defini- 59 04 knowledge referred to as “metrology” that focuses on the tion of units and other allowed composition 60 05 development and use of measurement instruments and operations on the mathematical structure. 61 06 measurement processes. The international consensus on 62 07 metrology is documented in the ISO vocabulary of basic 2. The measurement method: This is a description of the 63 [2] 08 and general terms in metrology. mapping making it possible to obtain a value for a 64 09 The quality of the measurement results (accuracy, given entity. It involves some general properties of 65 10 reproducibility, repeatability, convertibility, random mea- the mapping (mathematical view), together with a col- 66 11 surement errors) is essential for the measurement programs lection of assignment rules (operational description): 67 12 to provide effective and bounded results. Key characteris- 68 13 tics of measurement results and related quality of measur- a. Mapping properties: In addition to the homo- 69 14 ing instruments are defined in the ISO international morphism of the mapping, this can also include 70 15 vocabulary on metrology previously cited. The theory of a description of other mapping properties; for 71 16 measurement establishes the foundation on which mean- instance, a unit axiom (the mandatory associa- 72 17 ingful measurements can be made. The theory of measure- tion of the number 1 with an entity of the empiri- 73 18 ment and scale types is discussed in Ref. [3]. cal set) or, more generally, an adequate selection 74 19 Measurement is defined in the theory as “the assign- of a small finite representative set of elements 75 20 ment of numbers to objects in a systematic way to represent ranked by domain practitioners. 76 21 properties of the object.” An appreciation of software mea- b. Numerical assignment rules: These correspond 77 22 surement scales and the implications of each scale type in to an operational description of the mapping, i.e., 78 23 relation to the subsequent selection of data analysis meth- how to map empirical entities to numerical 79 24 ods are important. values: identification rules, aggregation rules, 80 25 Meaningful scales are related to a classification of procedural modeling of a measurement instru- 81 26 scales: ments family, usage rules, etc. 82
27 83 1. If the numbers assigned are merely to provide labels 28 3. The measurement procedure: This corresponds to a 84 to classify the objects, they are called nominal. 29 complete technical description of the modus operandi 85 2. If they are assigned in a way that ranks the objects 30 of the measurement method in a particular context 86 (e.g., good, better, best), they are called ordinal. 31 (goal, precision, constraints, etc.) and with a particu- 87 3. If they deal with magnitudes of the property relative to 32 lar measuring instrument. 88 a defined measurement unit, they are called interval 33 89 (and the intervals are uniform between the numbers 34 90 unless otherwise specified, and are therefore additive). Verification activities of the design of a software measure 35 91 4. Measurements are at the ratio level if they have an refer to the verification of each of the above design pro- 36 92 absolute zero point, so ratios of distances to the zero ducts of a software measurement method and of its appli- 37 93 point are meaningful. cation in a specific measurement context. The concept of 38 measurement information model (MIM) is presented and 94 39 Software Measurement Methods and Models discussed in the Appendix A of ISO 15939 (see Fig. 3) to 95 40 help in determining what must be specified during mea- 96 41 Software engineers that specialize in measurement should surement planning, performance, and evaluation: 97 42 be able to design a software measurement method when 98 43 required. This knowledge entails that the following con- 99 44 cepts are known and can be used: A specific measurement method is used to collect a 100 45 base measure for a specific attribute. 101 46 1. The measurement principle: This is a precise defini- The values of two or more base measures can then be 102 47 tion of the entities concerned and the attribute to be used in a computational formula (by means of a mea- 103 48 measured. It involves a description of the empirical surement function) to produce and construct a specific 104 49 world and the numerical world to which the entities derived measure. 105 50 are to be mapped: These derived measures are in turn used in an analysis 106 51 model in order to produce an indicator, which is a 107 52 a. The empirical world can be described through value, and to interpret the indicator’s value in order to 108 53 conceptual modeling techniques or through explain the relationship between it and the information 109 54 mathematical axioms, or both. needed, doing so in the language of the measurement 110 55 b. The numerical world can be, in the general case, user, to produce an Information Product in order to 111 56 any mathematical set, together with the accomplish with the user’s Information Needs. 112 4 Software Measurement Body of Knowledge
01 57
02 58
03 59
04 60
05 61
06 62
07 63
08 64
09 65
10 66
11 67
12 68
13 69
14 70
15 71
16 72
17 73
18 74
19 Fig. 3 ISO 15939 measurement model. 75
20 Source: From Systems and software engineering— 76 Measurement process.[4] 21 77
22 78 The bottom section of Fig. 4 represents the “Data THE MEASUREMENT PROCESS 23 79 Collection” (e.g., the measurement methods and the base 24 80 measures) in this MIM; the middle section the “Data This topic follows the international standard ISO/IEC 25 81 Preparation,” using agreed-upon mathematical formulas 15939, which describes a process that defines the activities 26 82 and related labels (e.g., measurement functions and derived and tasks necessary to implement a software measurement 27 83 process and includes, as well, an MIM (Fig. 5). 28 measures); and the top section the “Data Analysis” 84
29 (e.g., analysis model, indicator, and interpretation). 85
30 Establish and Sustain Measurement Commitment 86
31 87
32 Each software measurement endeavor should be guided by 88
33 organizational objectives and driven by a set of measure- 89
34 ment requirements established by the organization and the 90
35 project. An organizational objective might be “first-to- 91
36 market with new products.” 92
37 This in turn might engender a requirement that factors 93
38 contributing to this objective be measured so that projects 94
39 might be managed to meet this objective. The scope of 95
40 measurement may consist of a functional area, a single 96
41 project, a single site, or even the whole enterprise. 97
42 All subsequent measurement tasks related to this require- 98
43 ment should be within the defined scope. In addition, the 99
44 stakeholders should be identified. The commitment must be 100
45 formally established, communicated, and supported by 101
46 resources. The organization’s commitment to measurement 102
47 is an essential factor for success, as evidenced by assignment 103
48 of resources for implementing the measurement process. 104
49 105
50 Plan the Measurement Process 106
51 107
52 The organizational unit provides the context for measure- 108 53 ment, so it is important to make this context explicit and to 109 54 articulate the assumptions that it embodies and the con- 110 55 straints that it imposes. Characterization can be in terms of 111 56 Fig. 4 Measurement steps and main phases within the ISO 15939. organizational processes, application domains, technology, 112 Software Measurement Body of Knowledge 5
01 57
02 58
03 59
04 60
05 61
06 62
07 63
08 64
09 65
10 66
11 67
12 68
13 69
14 70
15 71
16 72
17 73
18 74
19 75 Fig. 5 The ISO 15939 process model. 20 76 Source: From Systems and software engineering—Measurement process.[4] 21 77 22 and organizational interfaces. An organizational process Resources should be made available for implementing 78 23 79 model is also typically an element of the organizational the planned and approved measurement tasks. Resource 24 80 unit characterization.[4] availability may be staged in cases where changes are to 25 81 Information needs are based on the goals, constraints, be piloted before widespread deployment. Consideration 26 82 risks, and problems of the organizational unit. Information should be paid to the resources necessary for successful 27 83 needs may be derived from business, organizational, reg- deployment of new procedures or measures.[4] This includes 28 84 ulatory, and/or product objectives. These information evaluation of available supporting technologies, selection of 29 85 needs must be identified and prioritized. Then, a subset to the most appropriate technologies, acquisition of those tech- 30 86 be addressed must be selected and the results documented, nologies, and deployment of those technologies. The next 31 87 communicated, and reviewed by stakeholders.[4] two processes of Fig. 5 (i.e., “Perform the measurement 32 88 Candidate measures must be selected, with clear links to process” and “Evaluate measurement”) are described in 33 89 the information needs. Measures must then be selected more details in the balance of this text. 34 90 based on the priorities of the information needs and other 35 91 criteria such as cost of collection; degree of process disrup- 36 THE MEASUREMENT BY SOFTWARE LIFE 92 37 tion during collection; ease of analysis; ease of obtaining 93 [4] CYCLE PHASE 38 accurate, consistent data; and so on. This encompasses 94 collection procedures and schedules, storage, verification, 39 This topic follows the international standard ISO/IEC 95 analysis, reporting, and configuration management of 40 15939, which describes a process that defines the activities 96 data.[4] Criteria for evaluation are influenced by the techni- 41 and tasks necessary to implement a software measurement 97 cal and business objectives of the organizational unit. 42 process and includes, as well, an MIM. 98 43 Information products include those associated with the pro- 99
44 duct being produced, as well as those associated with the 100 [4] Primary Processes 45 processes being used to manage and measure the project. 101 46 The measurement plan must be reviewed and approved by Software has a number of primary processes. These pri- 102 47 the appropriate stakeholders. This includes all data collec- mary processes are described in great detail in the ISO 103 [5] 48 tion procedures, storage, analysis, and reporting procedures; 12207 standard. This topic describes the typical software 104 49 evaluation criteria; schedules; and responsibilities. measurement activities that a software engineer engaged in 105 50 Criteria for reviewing these artifacts should have been measurement should master. 106 51 established at the organizational unit level or higher and 107
52 should be used as the basis for these reviews. Such criteria Software requirements 108
53 should take into consideration previous experience, availabil- 109
54 ity of resources, and potential disruptions to projects when As a practical matter, it is typically useful to have some 110 55 changes from current practices are proposed. Approval concept of the “volume” of the requirements for a particu- 111 56 demonstrates commitment to the measurement process. lar software product. This number is useful in evaluating 112 6 Software Measurement Body of Knowledge
01 the “size” of a change in requirements, in estimating the Fault density: A program under test can be assessed by 57 02 cost of a development or maintenance task, or simply for counting and classifying the discovered faults by their 58 03 use as the denominator in other measurements. types. For each fault class, fault density is measured as 59 04 Functional size measurement (FSM) is a technique for the ratio between the number of faults found and the size 60 05 evaluating the size of a body of functional requirements; ISO of the program[11] (see Chapter 20[12];Chapter9[13]). 61 06 has adopted the ISO 14143 family of international standards 62 Life test, reliability evaluation: A statistical estimate of 07 for the measurement of the functional size of the software. 63 software reliability, which can be obtained by reliabil- 08 ity achievement and evaluation (see Chapter 5, sub- 64 Software design 09 topic 2.2.5[1]), can be used to evaluate a product and 65 10 decide whether or not testing can be stopped (see 66 Measures can be used to assess or to quantitatively estimate 11 Chapter 9[14]; pp. 146–154[15]). 67 12 various aspects of a software design’s size, structure, or 68 Reliability growth models: Reliability growth models 13 quality. Most measures that have been proposed generally 69 provide a prediction of reliability based on the failures 14 depend on the approach used for producing the design. 70 [6,7] observed under reliability achievement and evaluation 15 These measures are classified into two broad categories: 71 (see Chapter 5, sub-topic 2.2.5[1]). They assume, in gen- 16 72 eral, that the faults that caused the observed failures have 17 Function-oriented (structured) design measures: the 73 been fixed (although some models also accept imperfect 18 design’s structure, obtained mostly through functional 74 fixes), and thus, on average, the product’s reliability 19 decomposition; generally represented as a structure 75 exhibits an increasing trend. There now exist dozens of 20 chart (sometimes called a hierarchical diagram) on 76 published models. Many are laid down on some common 21 which various measures can be computed. 77 assumptions, while others differ. Notably, these models 22 Object-oriented design measures: the design’s overall 78 are divided into failure-count and time-between failure 23 structure is often represented as a class diagram on 79 models (see Chapters 3, 4, and 7[13]; Chapter 9[14]). 24 which various measures can be computed. Measures 80 Coverage/thoroughness measures: Several test adequacy 25 on the properties of each class’s internal content can 81 criteria require that the test cases systematically exercise a 26 also be computed. 82 set of elements identified in the program or in the speci- 27 83 fications (see Chapter 5, sub-area 3[1]). To evaluate the 28 84 Software construction thoroughness of the executed tests, testers can monitor the 29 85 elements covered, so that they can dynamically measure 30 86 Numerous construction activities and artifacts can be mea- the ratio between covered elements and their total num- 31 87 sured, including code developed, code modified, code ber. For example, it is possible to measure the percentage 32 88 reused, code destroyed, code complexity, code inspection of covered branches in the program flow-graph, or that of 33 89 statistics, fault-fix and fault-find rates, effort, and scheduling. the functional requirements exercised among those listed 34 90 These measurements can be useful for purposes of managing in the specifications document. Code-based adequacy 35 91 construction, ensuring quality during construction, improv- criteria require appropriate instrumentation of the pro- 36 92 ing the construction process, as well as for other reasons. gram under test[11] (see Chapter 9[9];Chapter8[14]). 37 Even if measures applied to software source code 93 Fault seeding: Some faults are artificially introduced into 38 evolve with new programming languages and styles, 94 the program before test. When the tests are executed, 39 some “older” measures continue to be discussed and revis- 95 some of these seeded faults will be revealed, and possibly 40 ited with new technologies (for instance, the cyclomatic 96 some faults that were already there will be as well. In 41 complexity number). 97 42 theory, depending on which of the artificial faults are 98 discovered, and how many, testing effectiveness can be 43 Software testing 99
44 evaluated, and the remaining number of genuine faults 100
45 Numerous testing measures have been proposed in soft- can be estimated. In practice, statisticians question the 101 46 ware engineering. The most important relate to the distribution and representativeness of seeded faults rela- 102 47 following: tive to genuine faults and the small sample size on which 103
48 any extrapolations are based. Some also argue that this 104 technique should be used with great care, since inserting 49 Program measurements to aid in planning and design- 105 faults into software involves the obvious risk of leaving 50 ing testing: Measures based on program size (e.g., 106 them there. (see Chapter 8[14]; Section 3.1[16]). 51 source lines of code or function points) or on program 107 52 structure (like complexity) are used to guide testing. Mutation score: In mutation testing (see Chapter 5, sub- 108 [1] 53 Structural measures can also include measurements topic 3.4.2 ), the ratio of killed mutants to the total 109 54 among program modules in terms of the frequency number of generated mutants can be a measure of the 110 55 with which modules call each other (see Chapter 7, effectiveness of the executed test set (see Sections 111 [8] [9,10,11] [16] 56 Section 4.2 ; Chapter 9 ). 3.2–3.3 ). 112 Software Measurement Body of Knowledge 7
01 Cost/effort estimation and other test process measures: measurements may produce insights leading to process 57 [17] 02 Several measures related to the resources spent on changes and corresponding updates to the SCM process. 58 03 testing, as well as to the relative fault-finding effective- Software libraries and the various SCM tool capabilities 59 04 ness of the various test phases, are used by managers to provide sources for extracting information about the char- 60 05 control and improve the test process. These test mea- acteristics of the SCM process (as well as providing project 61 06 sures may cover aspects such as number of test cases and management information). For example, information 62 07 specified, number of test cases executed, number of test about the time required to accomplish various types of 63 08 cases passed, and number of test cases failed, among changes would be useful in an evaluation of the criteria 64 09 others. Evaluation of test phase reports can be combined for determining what levels of authority are optimal for 65 10 with root-cause analysis (RCA) to evaluate test process authorizing certain types of changes. Care must be taken to 66 11 effectiveness in finding faults as early as possible. Such keep the focus of the surveillance on the insights that can 67 12 an evaluation could be associated with the analysis of be gained from the measurements, not on the measure- 68 13 risks. Moreover, the resources that are worth spending ments themselves. 69 14 on testing should be commensurate with the use/criti- 70 15 cality of the application: different techniques have dif- Software engineering process 71 16 ferent costs and yield different levels of confidence in 72 17 product reliability[11] (see Chapter 4, Chapter 21, Evaluate the measurement process against specified eva- 73 18 Appendix B[12]; pp.139–145[15]). luation criteria and determine the strengths and weaknesses 74 19 Termination: A decision must be made as to how much of the process. This may be performed by an internal 75 20 testing is enough and when a test stage can be termi- process or an external audit and should include feedback 76 21 nated. Thoroughness measures, such as achieved code from measurement users. Record lessons learned in an 77 [4] 22 coverage or functional completeness, as well as esti- appropriate database. When incomplete or insufficient 78 23 mates of fault density or of operational reliability, data are available, appropriate techniques should be used 79 [18] 24 provide useful support, but are not sufficient in them- for analysis of historical databases. 80 25 selves. The decision also involves considerations about 81 26 the costs and risks incurred by the potential for remain- Software engineering tools and methods 82 27 ing failures, as opposed to the costs implied by con- 83 28 tinuing to test. See also Chapter 5, sub-topic 1.2.1[1] Software engineering tools evolve continuously and it is 84 [14,19,20] 29 Test selection criteria/Test adequacy criteria (see necessary to evaluate them. The ISO JTC1/SC7/ 85 30 Chapter 2, Section 2.4[8]; Chapter 2[12]). WG4 is developing a series of standards related to the 86 31 evaluation of software tools (including criteria and mea- 87 32 sures). On software methods, three groups are discussed in 88 33 SWEBOK, Chapter 10 (heuristic, formal, and prototyp- 89 34 Supporting Processes ing), each one dealing with a series of approaches. 90 35 In particular, an added value to measurement that these 91 36 Software has a number of supporting processes. These methods can bring is, respectively, about proposing meth- 92 37 supporting processes are described in great detail in the ods emphasizing the non-functional view on the project, 93 38 ISO 12207 standard. A supporting process “supports refinement of specifications, and prototyping evaluation 94 39 another process as an integral part with a distinct purpose techniques. 95
40 and contributes to the success and quality of the software 96
41 project. A supporting process is employed and executed, as Software quality 97 [5] 42 needed, by another process.” 98
43 As a multidimensional attribute, quality measurement is less 99 44 Software configuration management straightforward to define than those above. Furthermore, 100 45 some of the dimensions of quality are likely to require 101 46 Software configuration measures can be designed to pro- measurement in qualitative rather than quantitative form. 102 47 vide specific information on the evolving product or to A more detailed discussion of software quality measurement 103 48 provide insight into the functioning of the software config- is provided in the Software Quality KA, topic 3.4. 104 49 uration management (SCM) process. A related goal of ISO models for software product quality and related 105 50 monitoring the SCM process is to discover opportunities measurements are described in ISO 9126, parts 1 to 4 106 [7] [21] [22] 51 for process improvement. Measurements of SCM pro- (see Chapter 15 ; Chapters 9, 10 ; Chapter 24 ) and 107 52 cesses provide a good means for monitoring the effective- often include measures to determine the degree of each 108 [23] 53 ness of SCM activities on an ongoing basis. quality characteristic attained by the product. When 109 54 These measurements are useful in characterizing the selected properly, measures can support software quality 110 55 current state of the process, as well as in providing a (among other aspects of the software life cycle processes) 111 56 basis for making comparisons over time. Analysis of the in multiple ways. They can help 112 8 Software Measurement Body of Knowledge
01 in the management decision-making process; types of defects that occur most frequently, that is, answer- 57 [3,14,21] 02 find problematic areas and bottlenecks in the related ing questions in order to assess their density. Defect 58 03 software process; and analysis also aid in understanding the trends and how well 59 04 the software engineers assess the quality of their work detection techniques are working, and how well the devel- 60 05 for SQA (Software Quality Assurance) purposes and opment and maintenance processes are progressing. 61 06 for longer-term process quality improvement. Measurement of test coverage helps to estimate how 62 07 much test effort remains to be done, and to predict possible 63 08 remaining defects. From these measurement methods, 64 With the increasing sophistication of software, questions of 09 defect profiles can be developed for a specific application 65 quality go beyond whether or not the software works to how 10 domain. Then, for the next software system within that 66 well it achieves measurable quality goals. There are a few 11 organization, the profiles can be used to guide the SQM 67 more topics where measurement supports software quality 12 processes, that is, to expend the effort where the problems 68 management (SQM) directly. These include assistance in 13 are most likely to occur. 69 deciding when to stop testing. For this, reliability models 14 Similarly, benchmarks, or defect counts typical of that 70 and benchmarks, both using fault and failure data, are useful. 15 domain, may serve as one aid in determining when the 71 The cost of software quality is an issue that is almost 16 product is ready for delivery. 72 always raised in deciding how a project quality assurance 17 should be organized. Often, generic models of cost of 73 18 quality (CoQ) are used, which are based on when a defect 74 19 TECHNIQUES AND TOOLS 75 is found and how much effort it takes to fix the defect 20 relative to finding the defect earlier in the development 76 21 Measurement techniques and tools are useful to support 77 process. 22 decision makers in the use of quantitative information. 78 Project data may give a better picture of cost. 23 Additional techniques can be found in the total quality 79 Discussion on this topic can be found in Ref. [24, pp. 24 management (TQM) literature. 80 39–50]. Related information can be found in the 25 SWEBOK chapters on Software Engineering Process 81 26 Measurement Techniques 82 (Chapter 8) and Software Engineering Management 27 (Chapter 9). 83 28 Software measurement techniques may be used to analyze 84 Finally, the software quality reports provide valuable 29 software engineering processes and to identify strengths and 85 information not only on the executed processes but also on 30 weaknesses. This can be performed to initiate process imple- 86 how all the software life cycle processes can be improved. 31 mentation and change, or to evaluate the consequences of 87 Discussions on these topics can be found in Ref. [25]. 32 process implementation and change. The qualities of mea- 88 Measures of product quality can be represented using 33 surement results, such as accuracy, repeatability, and repro- 89 mathematical and graphical techniques. This aids in the 34 ducibility, are issues in the measurement of software 90 interpretation of the measures. Authors assign the mea- 35 engineering processes, since there are both instrument- 91 sures to the following categories:[3,13,21,26] 36 based and judgmental measurements, as, for example, 92 37 when assessors assign scores to a particular process. 93 38 Statistically based (e.g., Pareto analysis, run charts, A discussion and method for achieving quality of mea- 94 39 scatter plots, normal distribution) surement are presented in Ref. [27]. Process measurement 95 40 Statistical tests (e.g., the binomial test, chi-squared techniques have been classified into two general types: 96 41 test) analytic and benchmarking. 97 [28] 42 Trend analysis The two types of techniques can be used together since 98 43 Prediction (e.g., reliability models) they are based on different types of information. First, the 99 44 analytical techniques are characterized as relying on 100 45 The statistically based and statistical tests categories pro- “quantitative evidence to determine where improvements 101 46 vide a snapshot of the more troublesome areas of the soft- are needed and whether an improvement initiative has been 102 47 ware product under examination. The resulting charts and successful.” The analytical type is exemplified by the Quality 103 48 graphs are visualization aids, which the decision makers Improvement Paradigm (QIP) consisting of a cycle of under- 104 [29] 49 can use to focus resources where they appear most needed. standing, assessing, and packaging. The techniques pre- 105 50 Results from trend analysis may indicate that a schedule sented next are intended as other examples of analytical 106 [13,21,27,30,31] 51 has not been respected, such as in testing, or that certain techniques, and reflect what is done in practice. 107 52 classes of faults will become more intense unless some Whether or not a specific organization uses all these techni- 108 53 corrective action is taken. The predictive techniques can ques will depend, at least partially, on its maturity: 109 54 assist in planning test time and in predicting failure. 110 55 Defect analysis consists of measuring defect occurrences Experimental studies: Experimentation involves setting 111 56 and then applying statistical methods to understanding the up controlled or quasi experiments in the organization to 112 Software Measurement Body of Knowledge 9
[32] 01 evaluate processes. Usually, a new process is com- practices of the excellent organization, it will also 57 02 pared with the current process to determine whether or become excellent. Benchmarking involves assessing the 58 03 not the former has better process outcomes. maturity of an organization or the capability of its pro- 59 04 Process simulation: This type of study can be used to cesses. It is exemplified by the software process assess- 60 05 analyze process behavior, explore process improve- ment work. A general introductory overview of process 61 06 ment potentials, predict process outcomes if the current assessments and their application is provided in Ref. [42]. 62 07 process is changed in a certain way, and control process 63 08 execution. Initial data about the performance of the Measurement Tools 64 09 current process need to be collected, however, as a 65 10 basis for the simulation. Software engineering management tools are subdivided 66 [43] 11 Process definition: Process definition review is a means into three categories: 67 12 by which a process definition (either a descriptive or a 68 13 prescriptive one, or both) is reviewed, and deficiencies Project planning and tracking tools. These tools are 69 14 and potential process improvements identified. Typical used in software project effort measurement and cost 70 15 examples of this are presented in Refs. [33, 34]. An easy estimation, as well as project scheduling. 71 16 operational way to analyze a process is to compare it to Risk management tools. These tools are used in identi- 72 17 an existing standard (national, international, or profes- fying, estimating, and monitoring risks. 73 [5] 18 sional body), such as ISO 12207.0. With this approach, Measurement tools. The measurement tools assist in 74 19 quantitative data are not collected on the process, or, if performing the activities related to the software mea- 75 20 they are, they play a supportive role. The individuals surement program. 76 21 performing the analysis of the process definition use 77 22 their knowledge and capabilities to decide what process 78 23 changes would potentially lead to desirable process out- QUANTITATIVE DATA 79 24 comes. Observational studies can also provide useful 80 [35] 25 feedback for identifying process improvements. Quantitative data is identified by Vincenti as one of the 81 [44] 26 Root-cause analysis: It is another common analytical engineering knowledge types. According to Vincenti, 82 27 technique that is used in practice. This involves tracing quantitative data should be represented in tables and 83 28 back from detected problems (e.g., faults) to identify graphs, proposing measurement references and codified 84 29 the process causes, with the aim of changing the pro- experimental data, in order to show to those interested 85 30 cess to avoid these problems in the future, removing the practical results from the consistent application of a shared 86 31 inner, original cause instead of the solely back-sided, practice in the domain. Such quantitative data is currently 87 32 temporary effect. Examples for different types of pro- not referenced in any of the SWEBOK chapters. 88 33 cesses are described in Refs. [36–38]. 89 34 Orthogonal defect classification (ODC): It is a techni- Types of Entities 90 35 que that can be used to link faults found with potential 91 36 causes. It relies on a mapping between fault types and Quantitative data are typically collected based on the fol- 92 [39,40] 37 fault triggers. The IEEE standard on the classifi- lowing entities being measured. 93 38 cation of faults (or anomalies) may be useful in this 94 39 context (IEEE Standard for the Classification of Organization 95 40 Software Anomalies—IEEE1044-02). ODC is thus a 96 41 technique used to make a quantitative selection for Data on organizations include, for example, the semiann- 97 42 where to apply RCA. ual results on Sw-CMM (Software Capability Maturity 98 43 Statistical process control (SPC): It is an effective way Model) and CMMI (Capability Maturity Model 99 [45,46] 44 to identify stability, or the lack of it, in the process Integration) appraisals, as well as the results from 100 45 through the use of control charts and their interpreta- prizes associated to Performance Management Models as 101 46 tions. A good introduction to SPC in the context of Malcolm Baldrige in the USA and EFQM (European 102 47 software engineering is presented in Ref. [41]. Foundation for Quality Management) in Europe. 103 48 Personal software process (PSP): It defines a series of 104 49 improvements to an individual’s development prac- Project 105 50 tices in a specified order. It is “bottom-up” in the 106 51 sense that it stipulates personal data collection and Data on projects are available from project benchmarking 107 52 improvements based on the data interpretations. repositories such as the ISBSG (International Software 108 [47] 53 Benchmarking techniques: Benchmarking depends on Benchmarking Standards Group), with over more than 109 54 identifying an “excellent” organization in a field and þ5000 projects gathered worldwide as of 2009. A character- 110 55 documenting its practices and tools. Benchmarking istic of this repository is that the size of most of these projects 111 56 assumes that if a less-proficient organization adopts the is measured with one of the ISO-recognized FSM methods. 112 10 Software Measurement Body of Knowledge
01 Resource De Jure Standards 57
02 58 [48] 03 About the “resource” entity, P-CMM appraisal data Software product quality 59 04 could be a good source of information about people man- 60 [49] 05 agement. Other information on resources can be Software engineers should be able to specify product qual- 61 [47] 06 retrieved from the ISBSG repository. ity objectives and measure them throughout the software 62 07 life cycle. ISO defines a standard model for assessing the 63 08 quality of a software product (ISO 9126-01). This standard 64 Process 09 presents three perspectives of software quality: internal (as 65 10 viewed by the software engineer), external (as viewed by 66 Process data are typically available from project reposi- 11 the client), and quality in use (as viewed by the end user). 67 tories or from process assessment appraisals.[45,46] 12 While ISO 9126 proposes candidate measures for each 68 Repositories of process assessment appraisal data are typi- 13 of these perspectives, none of these measures are yet con- 69 cally proprietary. 14 sidered by ISO as measurement standards. The ISO 9126 70 15 standard is currently under revision and will be renamed 71 16 Product and extended into the ISO 25000 series. This new version 72 17 will progressively include measurement standards. In addi- 73 18 The data collected on software products are based on either tion, ISO currently proposes a software usability model 74 19 individually defined measures and models, or on measures (ISO 9241-11) that can be used to measure the usability 75 20 and models proposed by standards organizations: ISO 9126 of a software. 76 21 (that are going to be moved into the new upcoming ISO 77 22 25000 series) contains both examples of measures and Software functional size 78 23 model to quantify software product quality. There are 79 24 some publications on data based on such measures and Software engineers should be able to measure the size of a 80 25 models, but with limited scope. An example of the use of software. FSM is an important measure of software product 81 26 the ISO 9126 approach is documented in Ref. [50] for Web functional size. The key concepts of this measure have 82 27 environment and software package selection. A mapping been specified in ISO 14143-1 for a method to be recog- 83 28 of ISO 9126 to ISBSG is provided in Ref. [51]. nized as an FSM method. The ISO 14143 series includes 84 29 five additional parts (two international standards and three 85 30 technical reports) on various aspects related to FSM meth- 86 Software Measurement Repositories 31 ods, such as verification criteria and functional domains. 87 32 Five FSM methods have been adopted as ISO standards in 88 In 2009, two publicly available data repositories were 33 conformity with ISO 14143-1: 89 34 available to the software engineering community: the 90 [47] 35 ISBSG and the PROMISE repository for quality stu- 91 [52] COSMIC-FFP: ISO 19761 36 dies. The ISBSG repository had, as of the beginning of 92 IFPUG FPA: ISO 20926 37 2007, more than 4100 projects of several types. These 93 Mk-II FPA: ISO 20968 38 repositories are sources of data for external benchmark 94 NESMA FPA: ISO 24570 39 analysis as well as examples of data collection procedures, 95 FISMA FPA: ISO 29881 40 including standardized definitions for the data collected. 96
41 97 42 Software measurement process 98 43 MEASUREMENT STANDARDS 99 44 Software engineers should be able to apply a formal mea- 100 45 Software engineers should be aware of measurement stan- surement process. ISO 15939 documents the software mea- 101 46 dards to be used and referred when developing, maintain- surement process to be followed by software engineers, as 102 47 ing, and operating software. A de jure standard is typically previously discussed. 103 48 104 defined and adopted by a (independent) standardization 49 105 body such as ISO and IEC, ANSI (United States), 50 De Facto Standards 106 AFNOR (France), etc. A de facto standard is typically 51 107 defined and adopted by a community of interests, such as 52 ISBSG 108 IEEE, ISBSG, SPIN (Software Process Improvement 53 109 Network), itSMF (Information Technology Service 54 The ISBSG has put into place a data collection standard on 110 Management Forum), PMI (Project Management 55 software projects. The ISBSG data collection question- 111 Institute), etc. [53] 56 naire includes seven sections: 112 Software Measurement Body of Knowledge 11
01 Submitter information: information about the organiza- Knowledge—Guide to the SWEBOK. This proposal is 57 02 tion and the people filling out the questionnaire. designed to build, over time, consensus in industry, profes- 58 03 Project process: information about the project process. sional societies, and standards-setting bodies, among prac- 59 04 The information collected can be detailed along the ticing software developers and in academia. 60 05 various phases of a software life cycle. 61 06 Technology: information about the tools used for 62 ACKNOWLEDGMENT 07 developing and carrying out the project and for each 63 08 stage of the software life cycle. 64 Figures 2 and 5 were adapted from Fig. 1 (p. 9) of ISO/IEC 09 People and work effort: three groups of people are 65 15939:2007 [or] Fig. A.1 (p. 21) ISO/IEC 15939:2007. 10 considered: development team, customers and end 66 These figures are not considered official ISO figures nor 11 users, and IT operators. Collected information includes 67 were they authorized by ISO. Copies of ISO/IEC 12 information about the various people working on the 68 15939:2007 can be purchased from ANSI at http:// 13 project, their roles and their expertise, and the effort 69 webstore.ansi.org. 14 expended for each phase of the software life cycle. 70 15 Product: information about the software product itself 71 16 (e.g., software application type and deployment plat- REFERENCES 72 17 form, such as client/server, etc.). 73 18 Project functional size: information about the software 1. SWEBOK. Guide to the Software Engineering Body of 74 19 product functional size, according to the adopted count Knowledge; IEEE Computer Society: Los Alamitos, CA, 75 20 type, the measurement context, and the experience of http://www.swebok.org (accessed May 2010). 76 21 the people doing the count for such system. 2. ISO/IEC Guide 99:2007, International vocabulary of 77 22 Project completion: information about an overall pic- metrology—Basic and general concepts and associated 78 23 ture of the project, including project duration, defects, terms (VIM), 2007. 79 24 number of lines of code, user satisfaction, and project 3. Kan, S.H. Metrics and Models in Software Quality 80 Engineering, 2nd Ed.; Addison-Wesley: Boston, MA, 2002. 25 costs, including cost validation. 81 26 4. ISO/IEC 15939:2007, Systems and software engineering— 82 Measurement process, 2007, http://webstore.ansi.org 27 83 (accessed May 2010). 28 84 SPIN 5. ISO/IEC 12207:2008, Systems and software engineering— 29 Software life cycle processes, 2008. 85 30 86 A SPIN is a community of interest that supports and promotes 6. Jalote, P. An Integrated Approach to Software Engineering, 31 2nd Ed.; Springer-Verlag: Secaucus, NJ, 1997. 87 the use of software and systems engineering maturity models 32 7. Pressman, R. S. Software Engineering: A Practitioner’s 88 33 (i.e., ISO 15504, Capability Maturity Model Integration, 89 [54] Approach, 6th Ed.; McGraw-Hill: New York, 2004. 34 Software Maintenance Maturity Model, etc.) to improve 8. Beizer, B. Software Testing Techniques; International 90
35 software engineering processes. Measurement exemplary Thomson Press: Boston, MA, 1990. 91
36 practices are described in each maturity model. 9. Jorgensen, P.C. Software Testing: A Craftsman’s Approach, 92
37 2nd Ed.; CRC Press: Boca Raton, FL, 2004. 93 10. Bertolino, A.; Marre`, M. How many paths are needed for 38 itSMF 94 branch testing? J. Syst. Software 1996, 35 (2), 95–106. 39 95 The itSMF is a community of interest that supports and 11. IEEE 982.1-2005, IEEE Standard Dictionary of Measures 40 of the Software Aspects of Dependability, Institute of 96 41 promotes service management frameworks such as ITIL, 97 BS 15000, and ISO 20000. These exemplary practices Electrical and Electronics Engineers, Jan 1, 2006, 41 pp. 42 12. Perry, W. Effective Methods for Software Testing, Wiley: 98 apply to the IT infrastructure, services, and operations. 43 New York, 1995. 99 Measurement exemplary practices such as key process 44 13. Lyu, M.R. Handbook of Software Reliability Engineering, 100 45 indicators and service level agreements are described in Mc-Graw-Hill/IEEE: New York, 1996. 101
46 these references. 14. Pfleger, S.L. Software Engineering: Theory and Practice, 102
47 2nd Ed.; Prentice-Hall: Englewood Cliffs, NJ, 2001. 103
48 15. Poston, R.M. Automating Specification-Based Software 104 Testing, IEEE Press: Los Alamitos, CA, 1996. 49 CONCLUSION 105 16. Zhu, H.; Hall, P.A.V.; May, J.H.R. Software unit test cover- 50 106 age and adequacy. ACM Comput. Surv. 1997, 29 (4), 51 Achieving consensus by the profession on a software mea- 107 366–427, http://citeseerx.ist.psu.edu/viewdoc/download?doi= 52 surement body of knowledge is a key milestone for the 10.1.1.93.7961&rep=rep1&type=pdf (accessed May 2010). 108 53 evolution of software engineering toward a professional 17. Royce, W. Software Project Management, A United 109 54 status. This entry has presented the content of the new Framework; Addison-Wesley: Boston, MA, 1998. 110 55 knowledge area on software measurement proposed for 18. Strike K.; El-Emam, K.; Madhavji N. Software Cost 111 56 the next version of the Software Engineering Body of Estimation with Incomplete Data, NCR 43618, Technical 112 12 Software Measurement Body of Knowledge
01 Report, National Research Council Canada, January 2000, Proceedings of the International Conference on Software 57
02 http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc Maintenance, Bari, Italy, 1997. 58
03 &an=8913263&article=0&lang=en (accessed May 2010). 38. Nakajo, T.; Kume, H. A case history analysis of software 59 19. Mosley, V. How to assess tools efficiently and quantita- error cause-effect relationship. IEEE Trans. Software Eng. 04 60 tively. IEEE Software 1992, 9 (3), 29–32. 1991, 17 (8), 830–838. 05 61 20. Valaer, L.A.; Babb, R.G. II. Choosing a user interface 39. Chillarege, R. et al. Orthogonal defect classification—a 06 development tool. IEEE Software 1997, 14 (4), 29–39. concept for in-process measurement. IEEE Trans. 62 07 21. Fenton, N.E.; Pfleeger, S. L. Software Metrics: A Rigorous Software Eng. 1992, 18 (11), 943–956. 63 08 & Practical Approach, 2nd Ed.; International Thomson 40. Chillarege, R. Orthogonal defect classification. Handbook 64 09 Computer Press: Boston, MA, 1998. of Software Reliability Engineering, M. Lyu, Ed.; IEEE 65 10 22. Sommerville, I. Software Engineering, 7th Ed.: Addison- Computer Society Press: Los Alamitos, CA, 1996. 66 11 Wesley: Reading, MA, 2005. 41. Florac, W.; Carleton, A. Measuring the Software Process: 67
12 23. Grady, R.B. Practical Software Metrics for project Statistical Process Control for Software Process 68
13 Management and Process Management; Prentice Hall, Improvement; Addison-Wesley: Boston, MA, 1999. 69
14 Englewood Cliffs, NJ, 1992. 42. Zahran, S. Software Process Improvement: Practical Guid- 70 24. Rakitin, S.R. Software Verification and Validation: A elines for Business Success; Addison-Wesley: Reading, 15 71 Practitioner’s Guide; Artech House, Inc.: Boston, MA, 1997. MA, 1998. 16 72 25. McConnell, S. Code Complete: A Practical Handbook of 43. Dorfman, M.; Thayer, R.H. Eds. Software Engineering;IEEE 17 Software Construction; Microsoft Press: Redmond, WA, Computer Society Press: Los Alamitos, CA, 2002; Vol. 1 and 2. 73 18 1993, http://standards.ieee.org/reading/ieee/std_public/new_ 44. Vincenti, W.G. What Engineers Know and How They 74 19 desc/se/1012-2004.html (accessed May 2010). Know It—Analytical Studies from Aeronautical History, 75 20 26. Musa, J. Software Reliability Engineering: More Reliable John Hopkins University Press: Baltimore, MD, 1990. 76 21 Software, Faster Development and Testing; McGraw Hill: 45. Software Engineering Institute, Process Maturity Profile 77 22 New York, 1999. Sw-CMM 2006 Mid-Year Update, Software Engineering 78
23 27. Goldenson, D.; El-Emam, K.; Herbsleb, J.; Deephouse, C. Measurement and Analysis Team, Carnegie Mellon 79
24 Empirical Studies of Software Process Assessment University, August 2006; http://www.sei.cmu.edu/cmmi/ 80
25 Methods, ISERN Technical Report, ISERN-97-09, casestudies/profiles/swcmm.cfm (accessed May 2010). 81 Germany, 1997, http://isern.iese.de/moodle/file.php/1/ 46. Software Engineering Institute, Process Maturity Profile 26 82 Reports/reports97/isern-97-09.ps.gz (accessed May 2010). CMMI SCAMPI Class A Appraisal Results 2007- Year- 27 83 28. Carver, R. H.; Tai, K. C. Replay and testing for concurrent End Update, Software Engineering Measurement and 28 programs. IEEE Software 1991, 8 (2), 66–74. Analysis Team, Carnegie Mellon University, March 2008; 84 29 29. Software Engineering Laboratory, Software Process http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 85 30 Improvement Guidebook, NASA/GSFC, Technical Report (accessed May 2010). 86 31 SEL-95-102, April 1996. 47. ISBSG, Data Repository Release 10 January 2007, http:// 87 32 30. Weinberg, G.M. Measuring Cost and Value, “Quality www.isbsg.org (accessed May 2010). 88 33 Software Management,” First-Order Measurement, Dorset 48. Curtis, B.; Hefley, B.; Miller, S. People Capability Maturity 89
34 House: New York; 1993, vol. 2, Chap. 8. Model (P-CMM) version 2.0, Software Engineering 90
35 31. Zelkowitz, M.V.; Wallace, D. R. Experimental models Institute, Technical Report, CMU/SEI-2009-TR-003, http:// 91
36 for validating technology, IEEE Comput. 1998, 31, www.sei.cmu.edu/library/abstracts/reports/09tr003.cfm 92 23–31, www.idi.ntnu.no/emner/empse/papers/zelkowitz.pdf (accessed May 2010). 37 93 (accessed May 2010). 49. Miller, S., People Capability Maturity Model – Product Suite 38 94 32. McGarry, F. et al. “Software Process Improvement in the Maturity Profile, Software Engineering Institute, January 39 NASA Software Engineering Laboratory,” Software 2008, http://www.sei.cmu.edu/ (accessed May 2010). 95 40 Engineering Institute CMU/SEI-94-TR-22, 1994, http:// 50. Franch X.; Cavallo J.P., Using quality models in software 96 41 www.sei.cmu.edu/library/abstracts/reports/94tr022.cfm (acce- package selection. IEEE Software January/February 2003, 97 42 ssed May 2010). 20 (1), 34–41. 98 43 33. Bandinelli, S. et al. Modeling and improving an industrial soft- 51. Cheikhi, L.; Abran, A.; Buglione, L. The ISBSG Software 99 44 ware process. IEEE Trans. Software Eng. 1995, 21 (5), 440–454. Projects Repository from the ISO 9126 Quality Perspective. 100
45 34. Kellner, M. et al. Process guides: Effective guidance for J. Software Qual., American Society for Quality, March 2007, 101
46 process participants. Presented at the 5th International 9 (2), pp. 4–16. 102
47 Conference on the Software Process, Chicago, IL, 1998, 52. Boetticher, G.; Menzies, T.; Ostrand, T. PROMISE 103 http://users.ece.utexas.edu/~perry/prof/ispa/icsp5/program. Repository of empirical software engineering data repository, 48 104 html (accessed May 2010). West Virginia University, Department of Computer Science, 49 105 35. Agresti, W. The role of design and analysis in process 2007, http://promisedata.org/ (accessed May 2010). 50 improvement. Presented at Elements of Software Process 53. ISBSG, Data Collection Questionnaire—New 106 51 Assessment and Improvement, Los Alamitos, CA, 1999. Development, Redevelopment, or Enhancement sized 107 52 36. Collofello, J.; Gosalia, B. An application of causal analysis using IFPUG or NESMA Function Points, version 5.10, 108 53 to the software production process. Software Pract. Exp. Sept 2007, http://www.isbsg.org (accessed May 2010). 109 54 1993, 23 (10) 1095–1105. 54. April, A.; Abran, A. Software Maintenance Management: 110 55 37. El-Emam, K.; Holtje, D.; Madhavji, N. Causal analysis of Evaluation and Continuous Improvement; John Wiley & 111
56 the requirements change process for a large system. In Sons, Inc.: Hoboken, NJ, 2008. 112