Comparative Analysis of COCOMO II, SEER-SEM and True-S Software Cost Models

USC-CSSE-2008-816

Raymond Madachy, Barry Boehm USC Center for Systems and Software Engineering {madachy, boehm}@usc.edu

1. Abstract

We have been assessing the strengths, limitations, and improvement needs of cost, schedule, quality and risk models for NASA flight projects. The primary cost models used in this domain for critical flight software are COCOMO II, SEER-SEM and True S. A comparative survey and analysis of these models against a common database of NASA projects was undertaken. A major part of this work is defining transformations between the different models by the use of Rosetta Stones that describe the mappings between their cost factors.

With these Rosetta Stones, projects can be represented in all models in a fairly consistent manner and differences in their estimates better understood. Top-level Rosetta Stones map the factors between the models, and the detailed ones map the individual ratings between the corresponding factors. Most of the Rosetta Stone mappings between factors are one to one, but some are one to many.

The Rosetta Stones we have developed so far allow one to convert COCOMO II estimate inputs into corresponding SEER-SEM or True S inputs, or vice-versa. NASA data came in the COCOMO format and was converted to SEER-SEM and True S factors per the Rosetta Stones.

This initial study was largely limited to a COCOMO viewpoint. The current Rosetta Stones need further review and have to deal with incommensurate quantities from model to model. The cost models performed well when assessed against the NASA data despite these drawbacks, the absence of contextual data and potential flaws in the factor transformations.

The current set of Rosetta Stones has provided a usable framework for analysis, but more should be done including developing two-way and/or multiple-way Rosetta Stones, and partial factor-to-factor mappings. Factors unique to some models should be addressed and detailed translations between the size inputs should be developed including COTS and reuse sizing. Remaining work also includes elaborating the detailed Rosetta Stone for the new True S model, and rigorous review of all the top-level and detailed Rosetta Stones.

Conclusions for existing model usage and new model development are provided. In practice no one model should be preferred over all others, and it is best to use a variety of

1 USC-CSSE-2008-816 methods. Future work involves repeating the analysis with the refined Rosetta Stones, updated calibrations, improved models and new data.

2. Introduction

This research is assessing the strengths, limitations, and improvement needs of existing cost, schedule, quality and risk models for critical flight software for the NASA AMES project Software Risk Advisory Tools. This report focuses only on the cost model aspect.

A comparative survey and analysis of cost models used by NASA flight projects is described. The models include COCOMO II, SEER-SEM and True S. We look at evidence of accuracy, the need for calibration, and the use of knowledge bases to reflect specific domain factors. The models are assessed against a common database of relevant NASA projects. The overriding primary focus is on flight projects, but part of the work also looks at related sub-domains for critical NASA software. They are assessed as applicable in some of the following analyses. This report also addresses the critical NASA domain factors of high reliability and high complexity, and how the cost models address them.

Transformations between the models are also developed, so projects can be represented in all models in a consistent manner and to help understand why estimates may vary. There is a more thorough treatment of the USC public domain COCOMO II related models as they planned for usage in the current research, and the current datasets are in the COCOMO format. Conclusions for existing model usage and new model development are provided. The SEER-SEM and True S model vendors provided minor support and have identified areas of improvement as described later in this report.

2.1 Cost Models Used

The most frequently used cost and schedule models for critical flight software being evaluated are the COCOMO II, True S (previously PRICE S) and SEER-SEM parametric models. COCOMO II is a public domain model that USC continually updates and is implemented in several commercial tools. True S and SEER-SEM are both proprietary commercial tools with unique features but also share some aspects with COCOMO. All three have been extensively used and tailored for flight project domains.

Other industry cost models such as SLIM, Checkpoint and Estimacs have not been nearly as frequently used for flight software and are more oriented towards business applications. A previous comparative survey of software cost models can be found in [Boehm et al. 2000b].

A previous study at JPL analyzed the same three models, COCOMO II, SEER-SEM and PRICE S with respect to some of their flight and ground projects [Lum et al. 2001]. In that case each model estimate was a separate data point. This current approach varied since the data used only came in the COCOMO model format and required translation to the other models.

2 USC-CSSE-2008-816

2.1.1 COCOMO II

The COCOMO (COnstructive COst MOdel) cost and schedule estimation model was originally published in [Boehm 1981]. The COCOMO II research effort was started in 1994, and the model continues to be updated at USC, the home institution of research for the COCOMO model family. COCOMO II defined in [Boehm et al. 2000] has three submodels: Applications Composition, Early Design and Post-Architecture. They can be combined in various ways to deal with different software environments. The Application Composition model is used to estimate effort and schedule on projects typically done as rapid application development. The Early Design model involves the exploration of alternative system architectures and concepts of operation. Typically, not enough is known to make a detailed fine-grained estimate. This model is based on function points (or lines of code when available) and a set of five scale factors and seven effort multipliers.

The Post-Architecture model is used when top level design is complete and detailed information about the project is available and the software architecture is well defined. It uses Source Lines of Code and/or Function Points for the sizing parameter, adjusted for reuse and breakage; a set of 17 effort multipliers and a set of five scale factors that determine the economies/diseconomies of scale of the software under development.

USC provides a public domain tool for COCOMO II. The primary vendor tools that offer the COCOMO II model family include the following:  Costar offered by Softstar Systems has a complete COCOMO II implementation with tools for calibration and the Constructive Systems Engineering Model (COSYSMO). See http://www.softstarsystems.com. Softstar Systems provided a COCOMO II calibration spreadsheet used in support of this research (see Acknowledgments).  The Cost Xpert tool offered by the Cost Xpert Group has a superset of the COCOMO II post-architecture submodel. It has additional linear cost drivers and additional constraint factors on effort and schedule. See http://costxpert.com.  The True Planner tool from PRICE Systems has a COCOMO II model that can be invoked in lieu of the True S model.

The remainder of this report only considers the COCOMO II post-architecture submodel.

2.1.2 True S (formerly PRICE S)

True S is the updated product to the PRICE S model offered by PRICE Systems. PRICE S was originally developed at RCA for use internally on software projects such as the Apollo moon program, and was then released in 1977 as a proprietary model. Many of the model’s central algorithms were published in [Park 1988]. See the PRICE Systems website at http://www.pricesystems.com.

The PRICE S model consists of three submodels that enable estimating costs and schedules for the development and support of computer systems. The model covers

3 USC-CSSE-2008-816 business systems, communications, command and control, avionics, and space systems. PRICE S includes features for reengineering, code generation, spiral development, rapid development, rapid prototyping, object-oriented development, and software productivity measurement. Size inputs include SLOC, function points and/or Predictive Object Points (POPs).

The switch to True S is taking place during this work. Hence some of the descriptions retain the old PRICE S terminology (such as the Rosetta Stone) while we move towards a complete True S implementation. All numeric estimate results are for the latest True S model.

2.1.3 SEER-SEM

SEER-SEM is a product offered by Galorath, Inc. This model is based on the original Jensen model [Jensen 1983], and has been on the market for over 15 years. Its parametric modeling equations are proprietary. Descriptive material about the model can be found in [Galorath-Evans 2006].

The scope of the model covers all phases of the project life-cycle, from early specification through design, development, delivery and maintenance. It handles a variety of environmental and application configurations, and models different development methods and languages. Development modes covered include object oriented, reuse, COTS, spiral, waterfall, prototype and incremental development. Languages covered are 3rd and 4th generation languages (C++, FORTRAN, COBOL, Ada, etc.), as well as application generators.

The SEER-SEM cost model allows probability levels of estimates, constraints on staffing, effort or schedule, and it builds estimates upon a knowledge base of existing projects. Estimate outputs include effort, cost, schedule, staffing, and defects. Sensitivity analysis is also provided. Many sizing methods are available including lines of code and function points. See the Galorath Inc. website at http://www.gaseer.com.

3. Model Comparison

This section describes the major similarities and differences between the models. Analyses of their performance against project data, calibration and knowledge bases are addressed in Sections 4 and 5.

3.1 Algorithms

As described in [Lum et al. 2001], all three models essentially boil down to the common effort formula shown in . Size of the software is provided in a number of available units, cost factors describe the overall environment and calibrations may take the form of coefficients adjusted for actual data or other types of factors that account for domain-

4 USC-CSSE-2008-816 specific attributes. The total effort is calculated and then decomposed by phases or activities according to different schemes in the model.

Size Cost Factors Effort = A * Size B * EM Effort Phase and Activity Calibrations Decomposition

. Effort in person-months . A - calibrated constant . B - scale factor . EM - effort multiplier from cost factors

Figure 1: Common Core Effort Formula

All models allow size to be expressed as lines of code, function points, object-oriented metrics and others. Each model has its own respective cost factors for the linear effort multiplier term, and each model specifies the B scale factor in slightly different ways (either directly or through other factors). True S and SEER-SEM models have factors for the project type or domain, which COCOMO II currently does not. The model WBS phases and activities are addressed in section 3.4.

3.2 Size

All models support size inputs for new and adapted software, where adapted software can be modified or reused without change. Automatically translated or generated code is also supported in some of the models. The models differ with respect to their detailed parameters for the categories of software as shown in Table 1.

Commercial Off-The-Shelf (COTS) software is not addressed, but is a future research activity. COCOMO II can treat COTS as reused software or be used in conjunction with the COCOTS model [Boehm et al. 2000]. SEER-SEM and True S have more extensive COTS models.

Table 1: Model Size Inputs

COCOMO II Size Inputs SEER-SEM Size Inputs True S Size Inputs

New Software New Size New Size New Size New Size Non-executable Adapted Software Adapted Size Pre-exists Size 2 Adapted Size % Design Modified (DM) Deleted Size Adapted Size Non-executable % Code Modified (CM) Redesign Required % % of Design Adapted % Integration Required (IM) Reimplementation Required % % of Code Adapted

5 USC-CSSE-2008-816

Assessment and Assimilation (AA) Retest Required % % of Test Adapted Software Understanding (SU) 1 Reused Size Programmer Unfamiliarity (UNFM) 1 Reused Size Non-executable Deleted Size Code Removal Complexity Automatically Translated and Generated Code Adapted SLOC Auto Generated Code Size Automatic Translation Productivity Auto Generated Size Non-executable % of Code Reengineered Auto Translated Code Size Auto Translated Size Non-executable

1 - Not applicable for reused software 2 – Specified separately for Designed for Reuse and Not Designed for Reuse

COCOMO II allows for sizing in SLOC or function points. SEER-SEM and True S provide both of those along with additional size units. User-defined proxy sizes can be developed for any of the models and converted back to SLOC or function points.

Future work can also be undertaken to develop model translations between the size input parameters. These would consist of rules or guidelines to convert size inputs between models, and can be supplemented with knowledge base settings.

3.3 Cost Factor Rosetta Stones

This section describes the mappings, or transformations between cost factors in the different models. With this information COCOMO II estimate inputs can be converted into corresponding SEER-SEM or True S (or PRICE S) inputs, or vice-versa. It also illustrates differences in the models to help understand why estimates may vary. Top- level Rosetta Stones map the factors between the models, and the detailed ones map the individual ratings between the corresponding factors.

An integrated top-level Rosetta Stone for all of the COCOMO II factors is shown in Table 2. Most of the mappings between factors are one to one, but some are one to many (e.g. SEER-SEM has platform factor ratings split into target and host). In the case of True S, many of the COCOMO II factors have direct corollaries to sub-factors in aggregate True S factors. For example the COCOMO personnel factors are represented as sub-factors under the aggregate True S factor for Development Team Complexity. Table 3 and Table 4 show the additional factors in SEER-SEM and True S for which there are no analogs in COCOMO II.

Table 2: Integrated Top-Level Rosetta Stone for COCOMO II Factors

COCOMO II Factor SEER-SEM Factor(s) True S Factor(s) SCALE DRIVERS Precedentedness none none

6 USC-CSSE-2008-816

Development Flexibility none Operating Specification Architecture/Risk Resolution none none Team Cohesion none Development Team Complexity Process Maturity none 1 Organization Productivity - CMM Level PRODUCT ATTRIBUTES Required Software Reliability Specification Level - Operating Specification Reliability Data Base Size none Code Size non Executable Product Complexity - Complexity (Staffing) Functional Complexity - Application Class Complexity Required Reusability - Reusability Level Required Design for Reuse - Software Impacted by Reuse Documentation Match to none Operating Specification Lifecycle Needs PLATFORM ATTRIBUTES Execution Time Constraint Time Constraints Project Constraints - Communications and Timing Main Storage Constraint Memory Constraints Project Constraints - Memory & Performance Platform Volatility - Target System Volatility Hardware Platform Availability 3 - Host System Volatility PERSONNEL ATTRIBUTES Analyst Capability Analyst Capability Development Team Complexity - Capability of Analysts and Designers Programmer Capability Programmer Capability Development Team Complexity - Capability of Programmers Personnel Continuity none Development Team Complexity - Team Continuity Application Experience Application Experience Development Team Complexity - Familiarity with Platform Platform Experience - Development System Development Team Complexity Experience - Familiarity with Product - Target System Experience Language and Toolset Programmer’s Language Development Team Complexity Experience Experience - Experience with Language PROJECT ATTRIBUTES Use of Software Tools Software Tool Use Design Code and Test Tools Multi-site Development Multiple Site Development Multi Site Development Required Development none 2 Start and End Date Schedule

1 - SEER-SEM Process Improvement factor rates the impact of improvement, not the CMM level 2 - Schedule constraints handled differently 3 - A software assembly input factor

7 USC-CSSE-2008-816

Table 3: SEER-SEM Cost Factors with no COCOMO II Mapping

PERSONNEL CAPABILITIES AND EXPERIENCE

Practices and Methods Experience

DEVELOPMENT SUPPORT ENVIRONMENT

Modern Development Practices Logon thru Hardcopy Turnaround Terminal Response Time Resource Dedication Resource and Support Location Process Volatility

PRODUCT DEVELOPMENT REQUIREMENTS

Requirements Volatility (Change) 1 Test Level 2 Quality Assurance Level 2 Rehost from Development to Target

PRODUCT REUSABILITY

Software Impacted by Reuse

DEVELOPMENT ENVIRONMENT COMPLEXITY

Language Type (Complexity) Host Development System Complexity Application Class Complexity 3 Process Improvement

TARGET ENVIRONMENT

Special Display Requirements Real Time Code Security Requirements

1 – COCOMO II uses the Requirements Evolution and Volatility size adjustment factor 2 – Captured in the COCOMO II Required Software Reliability factor 3 – Captured in the COCOMO II Complexity factor

Table 4: True S Cost Factors with no COCOMO II Mapping

To-be-provided

8 USC-CSSE-2008-816

3.3.1 COCOMO II to SEER-SEM

Table 5 shows the detailed correspondence between COCOMO II and SEER-SEM factors with guidelines to convert ratings between the two models for applicable factors. In some cases the SEER-SEM factors cover different ranges than COCOMO and some of the conversions in Table 5 are best approximations. Not all factors have direct corollaries. The settings of the SEER-SEM factors may be defaulted according to project type and domain choices in the knowledge bases.

Table 5: COCOMO II to SEER-SEM Factors

COCOMO II Factor(s) SEER-SEM Factor(s) SCALE DRIVERS Precedentedness none Development Flexibility none Architecture/Risk Resolution none Team Cohesion none Process Maturity none 1 PRODUCT ATTRIBUTES Required Software Reliability Specification Level - Reliability 2 Very Low Very Low- Low Low Nominal Nominal High High Very High High+ Data Base Size none Product Complexity Complexity (Staffing) 3 Very Low Very Low Low Low Nominal Nominal High High Very High Very High Extra High Extra High Required Reusability Reusability Level Required Nominal Nominal High High Very High Very High Extra High Extra High Documentation Match to Lifecycle Needs none PLATFORM ATTRIBUTES Execution Time Constraint Time Constraints Nominal Nominal High Nominal Very High High Extra High Very High Main Storage Constraint Memory Constraints Nominal Nominal

9 USC-CSSE-2008-816

COCOMO II Factor(s) SEER-SEM Factor(s) High High Very High Very High Extra High Extra High Platform Volatility Target System Volatility, Host System Volatility Low Low Nominal High High Very High Very High Extra High PERSONNEL ATTRIBUTES Analyst Capability Analyst Capabilities Very Low Very Low Low Low Nominal Nominal High High Very High Very High Programmer Capability Programmer Capabilities Very Low Very Low Low Low Nominal Nominal High High Very High Very High Personnel Continuity none Application Experience Analyst’s Application Experience Very Low Very Low Low Low+ Nominal Low High Nominal Very High High Platform Experience Development System Experience, Target System Experience Very Low Very Low Low Low Nominal Nominal High Very High Very High Extra High Language and Toolset Experience Programmer’s Language Experience Very Low Very Low Low Low Nominal Nominal High Very High Very High Extra High PROJECT ATTRIBUTES Use of Software Tools Automated Tools Use Very Low Very Low Low Low Nominal Nominal High High Very High Very High

10 USC-CSSE-2008-816

COCOMO II Factor(s) SEER-SEM Factor(s) Multi-site Development Multiple Site Development Very Low Extra High Low Very High Nominal High or Nominal High High or Nominal Very High Nominal Extra High Nominal Required Development Schedule none 4

1 - SEER-SEM Process Improvement factor rates the impact of improvement instead of the CMM level 2 – Related SEER-SEM factors include Test Level and Quality Assurance Level which are also usually driven by reliability requirements 3 – SEER-SEM also has Application Class Complexity to rate at the program level, and other complexity factors for the development environment 4 - Schedule constraints handled differently in models

11 USC-CSSE-2008-816

3.3.2 COCOMO II to PRICE S and True S

Table 2 showed the top-level view of the COCOMO II to True S Rosetta Stone. Due to the new product being currently phased in, the Rosetta Stone will be refined for the next level of True S subfactors and will be provided at a later date.

The more complete Rosetta Stone in Table 6 and Table 7 shows the correspondence to the PRICE S model. The factor names shown are being replaced with the modernized terms in True S in order to elaborate the detailed Rosetta Stone between COCOMO II and True S.

12 USC-CSSE-2008-816

Table 6: COCOMO II to PRICE S Rosetta Stone

COCOMO Factor(s) PRICE S Factor(s)

SCALE DRIVERS

Precedentedness none Development Flexibility none Architecture/Risk Resolution none Team Cohesion none Process Maturity none

PRODUCT ATTRIBUTES

Required Software Reliability PLTFM Very Low 0.65 Low 0.8 Nominal 1 High 1.2 Very High 1.4 Data Base Size PROFAC Product Complexity APPL Very Low 0.86 Low 2.3 Nominal 5.5 High 6.5 Very High 8.5 Extra High 10.95 Required Reusability CPLX1 Nominal +0 High +0.1 Very High +0.3 Extra High +0.5 Documentation Match to Lifecycle none Needs

PLATFORM ATTRIBUTES

Execution Time Constraint UTIL - time Nominal 0.5 High 0.7 Very High 0.85 Extra High 0.95 Main Storage Constraint UTIL - memory Nominal 0.5 High 0.7 Very High 0.85 Extra High 0.95 Platform Volatility CPLX2 Low -0.05 Nominal +0 High +0.05 Very High +0.1

13 USC-CSSE-2008-816

Table 7: COCOMO II to PRICE S Rosetta Stone (Continued)

PERSONNEL ATTRIBUTES

Analyst Capability CPLX1 Very Low +0.1 Low +0.05 Nominal +0 High -0.05 Very High -0.1 Programmer Capability CPLX1 Very Low +0.1 Low +0.05 Nominal +0 High -0.05 Very High -0.1 Personnel Continuity none Application Experience CPLX1 Very Low +0.1 Low +0.05 Nominal +0 High -0.05 Very High -0.1 Platform Experience CPLX1 Very Low +0.1 Low +0.05 Nominal +0 High -0.05 Very High -0.1 Language and Toolset Experience PROFAC

Very Low +0.1 Low +0.05 Nominal +0 High -0.05 Very High -0.1

PROJECT ATTRIBUTES

Use of Software Tools CPLX1 Very Low Very Low Low Low Nominal Nominal High High Very High Very High Multi-site Development None Required Development Schedule Development Start date - mandatory

14 USC-CSSE-2008-816

Another aspect is "normalizing" True S against COCOMO II nominal conditions and matching their diseconomy of scale. A baseline normalization is needed against which factors can be changed to represent the projects already modeled with COCOMO II. Figure 2 shows the normalization between True S and COCOMO II.

1500 COCOMO II (nominal) 1400 True S @ complexity = 5.5 1300 True S @ complexity = 5 True S @ complexity = 6 1200 True S Parameters 1100 Functional Complexity = 5-6 1000 Operating Specification = 1.0 Organizational Productivity = 1.33 ) 900 s Development Team Complexity = 2.5 h

t 800 n

o 700 M - 600 n o

s 500 r e

P 400 (

r 300 o f f 200 E 100 0 0 25 50 75 100 125 150 175 200 225 250 Size (KSLOC)

Figure 2: Example of Normalizing True S and COCOMO II

The final determined values that most closely match True S to all nominal conditions in COOOMO II are listed below:  Functional Complexity is in the range of 5-6 and a value of 5.5 is suggested  Organization Productivity = 1.33  Development Team Complexity = 2.5  Operational Specification = 1.0.

15 USC-CSSE-2008-816

3.4 Phases and Activities

Reconciliation of the effort work breakdown structures (WBS) is necessary for valid comparison between models. If estimates are to be compared they need to cover the same activities. The common estimate baseline consists of the elaboration and construction phases for software activities (per the COCOMO II default), and shown in Figure 3. Additionally the NASA 94 data came in the COCOMO format and is assumed to cover those activities; hence a model that estimates more must have some activities subtracted out for a valid comparison. The correspondence of the common baseline and the core effort coverage of the different models are also shown in Figure 3.

16 USC-CSSE-2008-816

Phases Activities COCOMO II Inception Elaboration Construction Transition Maintenance Management Environment/CM Requirements Design Implementation Assessment Deployment

Hardware / System Software Detailed Code / Unit Integration & Test Software System Integration & Maintenance True S Concept Requirements Requirements Preliminary Design Design Test Integration Field Test Test Design Programming Data SEPGM Q/A CFM

System Software Component Detailed Code and Program System Integrate Requirements Requirements Preliminary Design Integrate and Design Unit Test Test Thru OT&E SEER Design Analysis Test Management SW Reqmnts Design Code Data Prep Test CM QA

Legend core effort coverage per model common estimate baseline

effort add-on as % of core coverage

effort add-on with revised model

Figure 3: Model Phases and Activities Coverage

17 USC-CSSE-2008-816

Due to the differences, the SEER-SEM and True S estimates were refined by subtracting the activities described below.

3.4.1 True S

True S provides a two-tier tree of estimates. The lower tier contains core engineering effort only as shown in Figure 4. The figure shows what elements should be subtracted for the common estimate baseline. The upper tier is a superset line item that includes systems engineering and program management (SEPM) activities. A table of its outputs is shown in Table 5 with the corresponding items to subtract for the common baseline set of activities.

Figure 4: Sample True S Engineering Estimate with Effort Items to Subtract

18 USC-CSSE-2008-816

Table 8: Sample True S SEPM-Level Estimate with Effort Items to Subtract

Labor Requirement Table : Engine Control - [Software Assembly] \ Labor Requirements in Months Total 1 Software Maintenance 0.0 2 Manage Project 141.2 3 Perform Configuration Management 61.8 4 Perform Joint Technical Reviews 0.0 5 Perform Quality Assurance 55.8 6 Plan and Oversee 13.0 7 Plan Software Development 19.3 8 Write Documentation 58.0 9 Analyze System Requirements 9.6 10 Design System 6.4 Perform Assembly Integration and 11 Test 88.1 12 Software Requirements Analysis 23.8 13 Software Design 122.9 14 Code and Unit Test 63.2 15 Software Integration and Test 33.1 16 Software Qualification Test 41.4 17 Perform HW/SW Integration and Test 26.1 Perform Software Product 18 Evaluations 0.0 19 Perform System Qualification Test 11.6 20 Total 775.4

subtracted effort = 119.1 revised total = 656.3

3.4.2 SEER

Error: Reference source not found shows a summarized SEER-SEM estimate and the items subtracted out for this analysis to make equivalent estimates.

19 USC-CSSE-2008-816

1.1 Program - 81

Activity Management SW Reqmnts Design Code Data Prep Test CM QA Total

System Requirements Design 2.30 9.97 2.68 0.00 1.15 2.30 0.38 0.38 19.17 S/W Requirements Analysis 6.75 25.88 7.88 3.38 3.38 6.75 1.13 1.13 56.26

Preliminary Design 12.97 11.79 48.34 14.15 9.43 16.50 2.36 2.36 117.89 Detailed Design 21.33 19.39 79.50 23.27 15.51 27.15 3.88 3.88 193.91 Code and Unit Test 21.27 9.12 18.23 167.15 18.23 45.59 12.16 12.16 303.91

Component Integrate and Test 29.10 7.28 14.55 141.89 29.10 105.51 18.19 18.19 363.81 Program Test 3.35 0.84 1.67 16.32 3.35 12.14 2.09 2.09 41.85 System Integrate Thru OT&E 24.86 6.22 12.43 59.04 3.11 183.35 15.54 6.22 310.76

Development Total 121.94 90.47 185.29 425.19 83.26 399.28 55.72 46.40 1,407.55 Maintenance 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Life Cycle Total 121.94 90.47 185.29 425.19 83.26 399.28 55.72 46.40 1,407.55

Figure 5: Sample SEER-SEM Estimate with Effort Items to Subtract

3.5 Critical Domain Factors

The vendor models provide elaborations of reliability and complexity factors beyond what COCOMO II provides. These are critical domain factors of relevance to NASA flight projects. Table 9 shows how the models address them.

Table 9: Vendor Elaborations of Critical Domain Factors

COCOMO II SEER-SEM 1 True S . Required Software . Specification Level – . Operating Specification Level Reliability Reliability (platform and environment . Test Level settings for required reliability, . Quality Assurance portability, structuring and Level documentation) . Product . Complexity (Staffing) . Functional Complexity Complexity . Language Type – Application Type (Complexity) . Language . Host Development . Language Object-Oriented System Complexity . Application Class Complexity

1 - SEER-SEM factors supplemented with and may be impacted via knowledge base settings for – Platform – Application – Acquisition method – Development method – Development standard – Class – Component type (COTS only)

20 USC-CSSE-2008-816

SEER-SEM has an extensive set of knowledge base choices. Table 10 shows example knowledge bases applicable to NASA flight projects.

Table 10: Example SEER-SEM Knowledge Bases Relevant to NASA Projects

Platform Knowledge Bases  Avionics  Ground System Non Critical  Ground-Based Mission Critical  Manned Space  Unmanned Space

Application Knowledge Bases  Flight Systems

In True S the Operating Specification factor describes the intended operating environment and defines the degree of required reliability, portability, structuring and documentation. Table 11 lists the categories for space software. Additional categories for military environments may also apply.

Table 11: True S Operating Specification Choices for Space Projects

Space Software  Unmanned  Manned

4. Model Analysis Against NASA 94 Project Data

The research team was provided the “NASA 94” set of projects. Of the 95 projects only 13 are listed as flight projects. All the remaining analysis is predicated on those 13 projects alone except noted otherwise where COCOMO II was also applied to five ground embedded projects.

The data came in the COCOMO 81 format. They were converted to COCOMO II per the guidelines in [Reifer et al. 2000] and further converted to SEER-SEM or True S factors per the Rosetta Stones in this report. See Appendix A for examples of the original and transformed data.

The database covers flight and ground projects, and some ground projects are embedded. An analysis of the critical factor distributions for reliability and complexity indicate that flight projects, as expected, exhibit patterns for both higher reliability and complexity. Figure 6 and Figure 7 show the distributions of these factors in the database. These spreads are as expected, which also supports the contention that these projects provided in the COCOMO 81 format are

21 USC-CSSE-2008-816 well conditioned. They are internally consistent and standardized in their reporting of these factors.

100 100 Flight Projects Flight Projects 80 80

s f t o s

c t 60 o 60 t

e c j t n e e o n j r c e

r 40 o 40 P c r e r P P e 20 P 20 0 0 Very Low Nominal High Very Extra Very Low Low Nominal High Very High Low High High Rating Rating 100 100 Ground Embedded Projects Ground Embedded Projects 80 80

s f t o s

c t 60 o 60 t

e c j t n e e o n j r c e r

o 40 40 P c r e r P P e 20 P 20

0 0 Very Low Nominal High Very Extra Very Low Low Nominal High Very High Low High High Rating Rating 100 100 Ground Other Projects 90 Ground Other Projects 80 80

f 70 s f o t s t t c o 60 60

c n e t j

e e 50 n o j c r e o r 40 P

c r 40 e r

P 30 P e

P 20 20 10 0 0 Very Low Nominal High Very High Extra Very Low Low Nominal High Very High Low High Rating Rating Figure 6: Reliability Distribution Figure 7: Complexity Distribution

4.1 Analysis Method

The process flow in Figure 8 shows the sequence of steps used in this analysis. Only the first pass through is described in this overall section. In the next iterations with refined and/or additional data not all of the steps will be performed. For example the need for calibration was amply demonstrated in the first round of analysis, and the uncalibrated model runs are not

22 USC-CSSE-2008-816 necessary in subsequent data iterations. The sequences of the vendor tool runs will also vary slightly to reflect their recommended best practices. Start

NASA 94 database

Apply Not all steps performed COCOMO 81→COCOMO II on iterations 2-n Rosetta Stone Select relevant domain projects

Outlier analysis SEER-SEM COCOMO II Apply True S COCOMO II→SEER Rosetta Stone Apply Uncalibrated COCOMO II→True S COCOMO II analysis Uncalibrated SEER analysis Rosetta Stone w/ additional factors defaulted

COCOMO II Set additional True S factors calibration (via Costar) Apply SEER for application domain knowledge base settings

Calibrated SEER analysis with True S analysis with COCOMO II analysis knowledge base settings application domain settings

SEER analysis with calibration and refined settings

Consolidated analysis

Update Additional data analysis? Y N End

Figure 8: Model Analysis Flow

The model performances were evaluated with standard figures-of-merit per the equations below based on comparisons between actual and estimated effort for n projects in a dataset:

Relative Error (RE) = ( Estimated Effort – Actual Effort ) / Actual Effort Magnitude of Relative Error (MRE) = | Estimated Effort – Actual Effort | / Actual Effort Mean Magnitude of relative error (MMRE) = (MRE) / n Root Mean Square (RMS) = ((1/n)  (Estimated Effort – Actual Effort)2) ½ Prediction Level PRED(L) = k/n, where k = the number projects in a set of n projects whose MRE <=L.

23 USC-CSSE-2008-816

Each run consists of a logical group of projects against with estimates are compared to actuals using the above measures. The progressive effects of calibrations and other adjustments are evaluated this way.

To help interpret some of the results we also looked at the error distribution for any biases. All the numeric estimated values for each model run are contained in the calibration analysis spreadsheets provided with [USC 2006] for further investigation.

4.2 Outlier Analysis

Investigation of the project data and the calibration results indicated a single particular outlier project. Its productivity is way out of bounds, by orders of magnitude, with the rest of the productivity spread. It is also the second smallest project (though the other small projects on its scale are inline with the rest of the productivity distribution). The data reporting is potentially suspect or it may be indicative of a single individual or extremely small team. At very small project sizes the effects of individuals tend to predominate, and this could be an extremely productive and unencumbered individual or small team. On modern NASA projects with increased complexity and size, it is highly unlikely that a single individual will create an entire CSCI.

Because of the giant disparity in productivity and these other reasons, subsequent analyses are performed without the outlier in the dataset except where noted otherwise. It does not seem to be representative of projects we wish to estimate. For additional reference [USC 2006] lists all results with the outlier included.

The dataset lists size as “PSLOC”. If it is physical SLOC then the size is probably overstated with respect to the COCOMO II standard of logical sources statements. USC has not been able to determine the exact unit of measurements or obtain further context information such as the language. If physical lines were counted then a conversion factor can be used to estimate logical statements. Without that information the current calibrations are predicated on the reported size units (i.e. the A constant is relevant when size is estimated using the same measure as the reported results.)

4.3 COCOMO

We calibrated for each embedded domain and the combined embedded flight domains. The results of the COCOMO II calibrations for the combined flight domains are in Figure 9. See [USC 2006] for the corresponding results when the outlier project is included. In order to expand the analysis space, we also assessed COCOMO against embedded ground projects. In that case the calibrated coefficient turned out less than the default COCOMO value. This result can be further explored with additional data or clarifications against the current dataset.

24 USC-CSSE-2008-816

Dataset Name: NASA 94 Category: Avionics + Science Mode: Embedded Number of Projects = 12 A = 5.80

Calibrated Effort Estimates vs. Actuals -

n 10000 o s r e ) P ( s

t h r t o n

f 1000 f o E

m l a u t c

A 100 100 1000 10000 Estimated Effort (Person-months)

Effort Prediction Summary Uncalibrated Calibrated MMRE 43% 29% RMS 1366.4 460.2 PRED(10) 9% 36% PRED(20) 9% 45% PRED(30) 18% 64% PRED(40) 36% 73%

Figure 9: COCOMO Results for Embedded Flight Projects (Avionics + Science)

4.4 SEER

The first SEER-SEM runs for uncalibrated and initial knowledge base settings were a “blind” study whereby actuals were not used. The Rosetta Stone was used to map to SEER-SEM inputs and used for the uncalibrated run. No further factors were touched. In the second round USC chose three knowledge base settings without extensive knowledge of SEER-SEM. The previous COCOMO-derived factors overrode any conflicts in the settings. In the most recent iteration, SEER-SEM experts made further settings in the knowledge bases, with use of actuals for calibration, and including the outlier project.

4.5 True S

25 USC-CSSE-2008-816

These results for True S represent the first round of applying an expert-determined mapping to True S without calibration. Table 12 shows the True S project settings. There are still additional calibrations being done with True S and those results will be provided later.

26 USC-CSSE-2008-816

Table 12: True S Project Settings

Project Organizational Productivity Constraints Development Team Complexity Hardware Platform Design, Design Availability Code and Functional for (Assembly Product Platform Exp. Test Multiple Site Start and Project Operational Specification IPT Use CMM Complexity Reuse Input) Analysts Programmers Team Con Fam Fam Lang. Tools Development End Date SLOC 81 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.86 nominal na 0.95 Unstable Capable Capable 5-10% 2-5 yrs <2 yrs <2 yrs nominal all Not 32,000 82 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 6.2 nominal na 0.88 Unstable Highly Cap Highly Cap 5-10% 5-10 yrs 5-10 yrs 5-10 yrs nominal set Enough 53,000 84 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.2 nominal na 0.88 Very Stable Expert Expert 5-10% > 10 yrs novice novice high at Time 41,000 89 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.2 nominal na 0.88 Very Stable Expert Expert 5-10% > 10 yrs novice novice high Entire for 24,000 91 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.86 nominal na 0.95 Mod. Stable Highly Cap Highly Cap 5-10% 5-10 yrs 5-10 yrs 5-10 yrs nominal Team Analysis 165,000 93 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.86 nominal na 0.95 Mod. Stable Highly Cap Highly Cap 5-10% 5-10 yrs 5-10 yrs 5-10 yrs nominal in of 70,000 94 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.86 nominal na 0.95 Mod. Stable Highly Cap Highly Cap 5-10% 5-10 yrs 5-10 yrs 5-10 yrs nominal Same Schedule 50,000 97 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.2 nominal na 0.88 Very Stable Highly Cap Capable 5-10% <2 yrs novice <2 yrs high Place 7,250 98 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.86 nominal na 0.95 Mod. Stable Highly Cap Highly Cap 5-10% 5-10 yrs 5-10 yrs 5-10 yrs nominal 233,000 99 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.1 nominal na 0.82 Unstable Capable Capable 5-10% 2-5 yrs <2 yrs <2 yrs nominal 16,300 100 Unmanned Spac NA IPT Casual and Team Some Success CMM 1 7.1 nominal na 0.82 Unstable Capable Capable 5-10% 2-5 yrs <2 yrs <2 yrs nominal 6,200 - 92 Military Airpborne NA IPT Casual and Team Some Success CMM 1 7.86 nominal na 0.95 Mod. Stable Highly Cap Highly Cap 5-10% 5-10 yrs 5-10 yrs 5-10 yrs nominal 65,000 101 Military Airpborne NA IPT Casual and Team Some Success CMM 1 7.1 nominal na 0.82 Mod. Stable Capable Capable 5-10% 2-5 yrs 2-5 yrs 2-5 yrs nominal 3,000

27 USC-CSSE-2008-816

Some data uncertainties may impact True S like the other models. The True S results may need adjustment for the PSLOC size definition. There are also unknowns about the Operational Specification used. Some of the projects may be manned space, but without that knowledge subsystems cannot be assigned to manned or unmanned. The level of specification may also not be consistent and homogeneous on a spacecraft.

5. Effects of Calibration and Knowledge Bases

The effects of calibration, knowledge base settings and other adjustments are evaluated with the respective models.

5.1 COCOMO II

The improvement effects of calibration for the different project subgroups are clearly seen in the performance measures. It is evident that MMRE improves in Figure 10 for all cases and that PRED(40) also improves in each case shownCOCOMO in Figure II MMRE 11. Calibration Effect 100 Uncalibrated 90 Calibrated 80

t 50 n e c

r 40 e

0 Flight Avionics Flight Science Flight (All) Ground Embedded Embedded Embedded Project Types

Figure 10: COCOMO II MMRE Calibration Effect

28 USC-CSSE-2008-816

100 Uncalibrated 90 Calibrated 80

t 50 n e c

r 40 e

0 Flight Avionics Flight Science Flight (All) Ground Embedded Embedded Embedded Project Types

Figure 11: COCOMO II PRED(40) Calibration Effect

A summary of the COCOMO II calibrations for the different sub-groups are shown in Table 13. The same information when using the outlier project is provided in [USC 2006].

Table 13: Summary of COCOMO II Linear Effort Calibrations

Project Group (# of Projects) Calibrated Coefficient Flight Avionics Embedded (10) A = 6.13 Flight Science Embedded (2) A = 4.38 Flight (All) (12) A = 5.80 Ground Embedded (5) A = 1.82

5.2 SEER

Similar progressive improvement trends were exhibited with the SEER-SEM model runs in Figure 12 and Figure 13. The first run used the translated COCOMO into SEER-SEM parameter settings with no further adjustments – the uncalibrated case. The next progressive improvement was the setting of knowledge base settings by USC personnel without extensive knowledge of SEER-SEM. This action considerably improved the model performance. The last set was calibrated by SEER-SEM personnel with further adjustments to the knowledge base settings and project specific adjustments.

29 USC-CSSE-2008-816SEER MMRE Progressive Adjustment Effects

Uncalibrated Initial Knowledge Base Settings Calibrated and Project-Adjusted t n e c r e P

Flight Avionics Embedded Flight (All) Project Types

Figure 12: SEER-SEM II MMRE Progressive Adjustment Effects

Uncalibrated Initial Knowledge Base Settings Calibrated and Project-Adjusted t n e c r e P

Flight Avionics Embedded Flight (All) Project Types

Figure 13: SEER-SEM PRED(40) Progressive Adjustment Effects

One conclusion from the SEER-SEM analysis this is that although statistical calibration helps, it is very important to properly characterize the technical and programmatic characteristics of the software being estimated. The SEER-SEM results, both uncalibrated and calibrated, improved significantly with more accurate information about platform, application, effective size and other parameters. It is suspected performance could be even better, without calibration, with a complete technical description of the software modules and for a "full-up" estimate.

5.3 True S

In the True S model the factor Application Type serves as a domain setting, similar to the SEER- SEM Knowledge Base settings. By setting Application Type then other factors are affected including  Functional Complexity

30 USC-CSSE-2008-816

 Operating Specification  Development Team Productivity  Sizing-related parameters.

True S was handled somewhat differently than SEER-SEM. Instead of choosing a particular application type to preset them, other factors were independently adjusted to represent the detailed project characteristics. The results of the factor adjustments used for input in the True S runs are shown in Table 12.

5.4 Model Performance Summaries

A summary of the model performances in their last runs (COCOMO calibrated, SEER-SEM with refined knowledge base settings, True S with application domain settings) is shown in Table 14. A scatterplot of the effort estimates vs. actuals is shown in Figure 14.

Table 14: Model Performance Summaries

Model 1 Model 2 Model 3 MMRE 29% 39% 49% RMS 460.2 642.4 613.5 PRED(10) 36% 20% 17% PRED(20) 45% 50% 42% PRED(30) 64% 50% 50% PRED(40) 73% 60% 58%

Model 1 Model 2 Model 3

t ) r s o h f t f n E

o d m e - t 1000 n a o m s i r t s e E P (

100 100 1000 10000 Actual Effort (Person-months)

31 USC-CSSE-2008-816

Figure 14: Effort Estimates vs. Actuals for All Models

5.5 Additional Analysis Runs

This analysis will be repeating as more data is received on the research project. As described only a subset of the steps will be performed on the next iterations of analysis. The process may also vary if project data is received in multiple model format, e.g. using the full set of SEER- SEM or True S parameters and bypassing the Rosetta Stones between COCOMO II.

A minor revision of the first round of results will be done first. This is necessary since Galorath has provided refined data from the NASA 94 dataset. In particular, the equivalent size of three of the flight projects is lower than the values used in the initial runs. This largely explains why Galorath was able to include the apparent outlier project in their final SEER-SEM run. The other model analyses will be re-performed, but a quick assessment of the differences indicates the overall results will vary negligibly. More importantly the root cause of the data discrepancy will be investigated. Further impacts on the overall results and conclusions, if any, will be reported then.

A data collection initiative is also underway at NASA to collect more modern data for this project. On the NASA MOCA research grant, USC has received actuals on the recent Shuttle Abort and Flight Management (SAFM) project and it will be incorporated into the analysis [Madachy-Boehm 2006].

32 USC-CSSE-2008-816

6. Conclusions and Future Work

This paper has presented an overview of software cost models with a focus on critical flight software. The primary cost models were assessed against a relevant database of NASA and performed well, particularly with the absence of contextual data and potential flaws in the factor transformations. When using the NASA 94 dataset, calibration and knowledge base judgments for the domain improved all model performance versus using default parameter values.

This study was performed by persons highly familiar with COCOMO but not necessarily with SEER-SEM or True S. The vendors of these models provided minor support, but do not yet certify or sanction the data nor information contained in this report. Specific vendor concerns include: . the study limited to a COCOMO viewpoint only . current Rosetta Stones need review and may be weak translators from the original data . results not indicative of model performance due to ignored parameters . risk and uncertainty were ground ruled out . data sanity checking needed.

NASA flight projects are typified by extremely high reliability and complexity. Characterizations of the database projects in terms of these important descriptors provided useful and interesting results. Distributions of factor ratings for complexity and reliability factors showed relevant patterns across the subgroups in the database. It also helped to confirm that the COCOMO factor ratings were done consistently across the projects, and adhere to the COCOMO rating criteria. All models support effort variance due to these factors, but the vendor models provide additional elaborations of these factors and domain-specific defaults.

Within the COCOMO II scope and subset of analyses, it can be concluded that the overall embedded flight domain calibration with this data is a linear coefficient A  6. The value is slightly less for embedded science vs. avionics projects. But the sample size for science projects was extremely small, and more data should be incorporated for a more robust and credible calibration.

Successive experiments with the SEER-SEM model illustrated that the model performance measures markedly improved when incorporating knowledge base information for the domains. Simple educated guesses on the knowledge base choices without extensive SEER-SEM knowledge produced far better estimates than strict uncalibrated estimates.

The initial uncalibrated runs from COCOMO II and SEER-SEM both underestimated the projects by approximately 50% overall. That is also reflected in the calibrated COCOMO II coefficient being about twice the default (A  6 vs. A=2.96).

For all models (COCOMO, SEER, True S), calibration against the different subgroups exhibited nearly equivalent trends for embedded flight projects. The model performance measures for either individual flight groups (avionics or science) or combined together (avionics plus science)

33 USC-CSSE-2008-816 were about the same and the improvement trends between uncalibrated and calibrated were identical when the outlying project was excluded.

Major future thrusts include refining and expanding the project dataset, and updating the COCOMO model(s) for flight applications. The calibrations are all derived with respect to the reported size termed PSLOC (likely to be physical lines). Further investigations to directly capture logical source statements or through the use of conversion factors may yield different calibrations for using COCOMO II with its current size definitions.

The vendor models provide more granular factors for the overall effects captured in the COCOMO II Complexity (CPLX) factor. One consideration is to elaborate the current COCOMO definition with more levels of detail specifically interpreted for critical flight project applications. The COCOMO II Required Software Reliability (RELY) factor is also being elaborated for high dependability and security applications, and that research will be brought to bear on the current effort.

The COCOMO II and COQUALMO models are being updated on this project for new technologies, IV&V techniques and new mission requirements (e.g. increased reliability for security and safety). Additional project data and Delphi studies are both being used. The revised models will undergo the same analysis and be re-calibrated again for flight projects with the additional data.

One option to expand COCOMO II is to embed a knowledge-based capability into the model specifically for NASA projects. An example could be based on the observed reliability and complexity factor distributions. A sub-domain is selected by the user (flight science, flight avionics, ground embedded, etc.) and factor settings are defaulted in the model.

This study has been helpful in reducing sources of misinterpretation across the models, and the current set of Rosetta Stones and other model comparisons have provided a usable framework for analysis, but considerably more should be done including: . developing two-way and/or multiple-way Rosetta Stones . explicit identification of residual sources of uncertainty across models and their estimates not fully addressable by Rosetta Stones . factors unique to some models but not others . develop translations between the size input parameter factors . address COTS estimation and sizing . many-to-many factor mappings . partial factor-to-factor mappings . similar factors that affect estimates in different ways: linear, multiplicative, exponential, other . imperfections in data: subjective rating scales, code counting, counting of other size factors, effort/schedule counting, endpoint definitions and interpretations, WBS element definitions and interpretations. The study participants welcome sponsorship of further joint efforts to pin down sources of uncertainty, and to more explicitly identify the limits to comparing estimates across models.

34 USC-CSSE-2008-816

A more rigorous review of the detailed cost factor Rosetta Stones should be completed. Some of the future work to complete this work rests on the vendors. They will be further reviewing their specific sections and clarifying the USC interpretation of their work breakdown structures, factor interpretations and other aspects. Additional approaches for calibration are also being evaluated. Remaining work on the Rosetta Stones includes elaborating the detailed Rosetta Stone for True S, and rigorous review of all the top-level and detailed Rosetta Stones.

The analysis process can also be improved on several fronts. The recommended sequence for vendor tool usage is to first set knowledge bases before COCOMO translation parameter setting. It is also desirable to capture estimate inputs in all three model formats, and try different translation directionalities.

This analysis has also identified additional information on the projects that could be useful. The vendors are involved in this aspect and the model analyses are likely to be re-iterated for several reasons including additional or refined data assumptions.

In practice no one model should be preferred over all others. The key to arriving at sound estimates is to use a variety of methods and tools and then investigate the reasons why the estimates provided by one might differ significantly from those provided by another. If the practitioner can explain such differences to a reasonable level of satisfaction, then it is likely that he or she has a good grasp of the factors which are driving the costs of the project at hand. He/she will be better equipped to support the project planning and control functions performed by management.

Future work involves repeating the analysis with updated calibrations, revised domain settings, improved models and new data. It is highly desirable to incorporate more recent NASA project data in the cost model analyses. The MOCA project collected actuals on the SAFM project, more data is being solicited and it all will be used to update the analysis and support research demands for current data. Other data concerns include the units of size measurement in the NASA 94 dataset, which should be investigated for reasons previously stated in the analyses.

With modern and more comprehensive data, COCOMO II and the other models can be further improved and tailored as necessary for NASA project usage.

6.1 References

[Boehm 1981] Boehm B., Software Engineering Economics. Englewood Cliffs, NJ, Prentice- Hall, 1981

[Boehm et al. 2000] Boehm B., Abts C., Brown W., Chulani S., Clark B., Horowitz E., Madachy R., Reifer D., Steece B., Software Cost Estimation with COCOMO II, Prentice-Hall, 2000

[Boehm et al. 2000b] Boehm B, Abts C, Chulani S, “Software Development Cost Estimation Approaches – A Survey”, USC-CSE-00-505, 2000

35 USC-CSSE-2008-816

[Boehm et al. 2004] Boehm B, Bhuta J, Garlan D, Gradman E, Huang L, Lam A, Madachy R, Medvidovic N, Meyer K, Meyers S, Perez G, Reinholtz KL, Roshandel R, Rouquette N, “Using Empirical Testbeds to Accelerate Technology Maturity and Transition: The SCRover Experience”, Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004

[Galorath 2005] Galorath Inc., SEER-SEM User Manual, 2005

[Galorath-Evans 2006] Galorath D, Evans M, Software Sizing, Estimation, and Risk Management, Auerbach Publications, 2006

[Jensen 1983] Jensen R, “An Improved Macrolevel Software Development Resource Estimation Model”, Proceedings of 5th ISPA Conference, 1983

[Lum et al. 2001] Lum K, Powell J, Hihn J, “Validation of Spacecraft Software Cost Estimation Models for Flight and Ground Systems”, JPL Report, 2001

[Madachy 1997] Madachy R, Heuristic Risk Assessment Using Cost Factors”, IEEE Software, May 1997

[Madachy-Boehm 2006] Madachy R, Boehm B, “A Model of Options and Costs for Reliable Autonomy (MOCA) Final Report”, reported submitted to NASA for USRA contract #4481, 2006

[Park 1988] Park R, “The Central Equations of the PRICE Software Cost Model”, COCOMO User’s Group Meeting, 1988

[PRICE 2005] PRICE Systems, TRUE S User Manual, 2005

[Reifer et al. 1999] Reifer D, Boehm B, Chulani S, “The Rosetta Stone - Making COCOMO 81 Estimates Work with COCOMO II”, Crosstalk, 1999

[USC-CSE 2006] USC Center for Software Engineering, Model Comparison Report, Report to NASA AMES, Draft Version, July 2006

7. Acknowledgements

This work is supported by the NASA AMES Research Center Cooperative Agreement No. NNA06CB29A for Software Risk Advisory Tools. Our colleagues helping on this research at NASA include Mike Lowry, Julian Richardson and Tim Menzies. It also would not have been possible without the contributions of other colleagues and generous organizations. We particularly thank Galorath Inc. and PRICE Systems for providing us with their tool information and people. Thanks are due to all the people mentioned below.

 Tim Hohmann at Galorath Inc. helped as our primary technical contact for SEER-SEM support. Additional assistance and support from Galorath Inc. came from Dan Galorath, Karen McRitchie and Bob Hunt.

36 USC-CSSE-2008-816

 David Seaver was our primary contact and provided technical support from PRICE Systems, and James Otte also provided early assistance.  Jairus Hihn and Sherry Stukes from NASA JPL supported this analysis.  Dan Ligett from Softstar Systems graciously provided a calibration spreadsheet that was modified for this research.

37 USC-CSSE-2008-816

8. Appendix A: NASA 94 Original and Transformed Data

Table 15: Original COCOMO 1981 Data for NASA 94 Avionics, Embedded Projects recordnum berprojectnam ecat2 forg center year m ode rely data cplx time stor virt turn acap aexp pcap vexp lexp m odp tool sced equivphysklocact_effort 81 hst A vionics f 2 1980 em bedded h vh vh xh xh h h n n n l l n n h 32 1350 82 hst A vionics f 2 1980 em bedded h h h vh xh h h h h h h h h n n 53 480 84 spl A vionics f 3 1977 em bedded h l vh vh xh l n vh vh vh vl vl h h n 41 599 89 spl A vionics f 3 1977 em bedded h l vh vh xh l n vh vh vh vl vl h h n 24 430 91 sts A vionics f 5 1977 em bedded vh h vh xh xh n n h h h h h h n h 165 4178.2 93 sts A vionics f 5 1977 em bedded vh h vh xh xh n l h h h h h h n h 70 1645.9 94 sts A vionics f 5 1977 em bedded vh h xh xh xh n n h h h h h h n h 50 1924.5 97 gal A vionics f 5 1982 em bedded vh l vh vh xh l l h l n vl l l h h 7.25 648 98 sts A vionics f 5 1980 em bedded vh h vh xh xh n n h h h h h h n h 233 8211 99 gro A vionics f 2 1983 em bedded h n vh vh vh h h n n n l l n n h 16.3 480 100 gro A vionics f 2 1983 em bedded h n vh vh vh h h n n n l l n n h 6.2 12

Table 16: COCOMO II Transformed Data for NASA 94 All Embedded Projects

38 USC-CSSE-2008-816