Software Cost Estimation for COVID

2021 NASA Cost and Schedule Symposium Virtual Event April 7, 2021

Sherry Stukes Georgia Bajjalieh Jet Propulsion Laboratory, California Logical-R Joint Venture, LLC Institute of Technology NASA 4800 Oak Grove Drive Moffett Field CA 94035 Pasadena, CA 91109 georgia.bajjalieh@.gov [email protected]

DISCLAIMER The cost information contained in this document is of a budgetary and planning nature and is intended for informational purposes only. It does not constitute a commitment on the part of NASA, JPL, and/or Caltech. © 2021 California Institute of Technology. Government sponsorship acknowledged. Contents vBackground and Overview vHistorical Data vModels and Methodologies vFindings vSummary

2 Background and Overview

3 Background - VIPER vVIPER rover mission to the ’s south pole v100 -days vMission to find water ice vWhere is water ice located? vWhere are the highest concentrations? vHow deep is the ice? vFirst U.S. rover launched to the lunar surface since the Apollo missions in the early 1970’s *VIPER in Formulation Phase when the Rover Software Cost Estimate was conducted in 2020

Reference: VIPER home page, https://www.nasa.gov/viper

4 Key Cost Estimating Challenges (1) v Parametric models are not designed to capture unique aspects of VIPER Rover Software v VIPER is being developed as a research and technology project, rather than as a space flight program v VIPER is a risk tolerant mission similar to NASA “Class D” payloads v Key portions of the rover’s software are being designed and implemented as ground software v Reflecting affects of tailored NASA Procedural Requirement (NPR) requirements v No direct analogies v Draw on bits and pieces from prior projects v Look for systems as similar as possible

5 Key Cost Estimating Challenges (2) v Integration of large open source software products v RGSW – Gazebo (3D dynamic simulator) v RSIM – ROS2 (Robotic Operating System image processing and tools) v Impact of COVID v Models do not directly capture pandemic cost impact v Agile development may be less efficient as Agile principles promote face-to-face meetings and a highly collaborative environment v Personnel productivity may be different than expected

6 Rover Flight Software Architecture v3 CSCIs vRover Ground Software (RGSW) vRover Flight Software (RFSW) vRover SIMulations Software (RSIM) vDecomposed to module level for software lines of code sizing vPlanned for high software reuse vIntegration large open source modules

7 Historical Data

8 Historical Data – Resources v Reviewed Resource BoE document dated 12/19/2013 v Limited information and details for five BoEs v Researched updated information v Data Sources v JPL SMART (Software Measures and Analysis Reporting Tool) – JPL proprietary v CADRe (Cost Analysis Data Requirement) – available on ONCE site https://oncedata.hq.nasa.gov/Main.aspx v ASCoT (Analogy Software Cost Tool) – also available on ONCE site v NASA projects v Used “JPL Mission Software: State of Software Report 2018” as a reference for software size

9 Historical Data Collection Project Data

Project Center Source Why Relevant Reviewed CADRe and ASCoT data with LADEE ARC Lunar orbiter, Class D mission Software Project Manger for LADEE LCROSS ARC Only sizing data available Robotic lunar impactor, low-cost LRO GSFC ASCoT data only Robotic lunar orbiter, 3D mapping Data from CADRe, ASCoT, and SMART GRAIL JPL Lunar orbiter and simulation software (used for RFSW and RSIM) Rover, tech demo, all new software MPF JPL Data from CADRe code MER JPL Data from CADRe, ASCoT, and SMART Rover MSL JPL Data from CADRe, ASCoT, and SMART Rover Preliminary software size data from M2020 JPL Rover, subset analogous to RGSW Project GRO GSFC SMART data only Simulation software STS GRC SMART data only Simulation software Centaur STS 51E JSC SMART data only Simulation software and 61I

10 Historical Data Collection What We Found v Software database content v Variety of sources v Across NASA Centers v Organized by element v Established NASA sources v Easily accessible

VIPER rover software elements compare favorably with historical data.

11 Historical Flight Software Data Analogies

VIPER RFSW size is in family with JPL Reference: JPL State of Software Report, 2018 historical projects.

12 Models and Methodologies

13 Models and Methodologies

SEER-SEM and v Parametric Models COCOMO are in v Primary – SEER-SEM wide use across vDeveloped by Galorath Incorporated NASA. vV8.3 vWidely used within NASA v Cross-check – COCOMO (SCAT) vJPL version of COCOMO II vModel implemented in Excel and enhanced to capture uncertainty with Monte Carlo add-in vAvailable to NASA Centers from JPL v Analogy Software Cost Tool – (ASCoT) v Rules of Thumb (SLOC/WM) v Compare and contrast cost estimates

14 SEER-SEM Details Input Data Summary vSLOC (Logical) Rover Software Size Summary vNew SW Reuse Reuse % % Reuse New Total ESLOC Element “as is” Modified Redesign Recode Source SLOC SLOC SLOC vReused “as is” Units SLOC SLOC % % Project SLOC SLOC SLOC vReused RGSW 26371 2860 25 50 RP 10638 40490 17256 modified RFSW 55858 85885 7 26 LADEE 53530 196096 89218 v % re-design RSIM 303700 21600 15 15 RP, OSRF 26320 351930 30852

v % re-code Notes Terms Projects Logical SLOC SLOC - Source Lines of Code LADEE - Lunar Atmosphere Dust and Environment Explorer v % re-test ESLOC - Equivalent (new) Source Lines of Code RP -Resource Prospector RFSW - Rover Flight Software OSRF - Open Source Robotics Foundation vTotal delivered RGSW - Rover Ground Software vEquivalent new RSIM - Rover Simulation Software vKnowledge Bases vEstablishes Rover Software Knowledge Base Selections default Software Acquistion Development Development Platform Application parameter Element Method Method Standard settings RFSW Unmanned Space Flight Systems New & Reused Agile - Novice Commercial vBased on RGSW Unmanned Space Flight Systems New & Reused Agile - Novice Commercial industry RSIM Unmanned Space Simulation New & Reused Agile - Novice Commercial averages vModified for VIPER 15 SEER-SEM Details Software Sizing vSample of software sizing vSized by module vConducted software reuse study vCounted with PySLiC Logical Logical Rework Rework Retest SLOC Soure SLOC repo/folder Description Category Design Code Code Language Inherited Project Expected Notes Modified rockCraterGen rock and crater generation Legacy 15% 30% 50% Matlab 1000 RP 1200 1000 craterDisplay crater marking GUI Legacy 30% 30% 20% C++ 1200 RP 2200 1200 moonShine synthetic DEM generator Legacy 10% 20% 40% C++ 6400 RP 7000 6400 craterProfile improve crater profile New C++ 500 tile to Gazebo model tile_to_gazebo processing Legacy 50% 40% 20% Python 650 RP 800 650 NSS protocol New C++ 300 RP NSS/NIRVSS Sim behavior Legacy 15% 50% 10% Python 1000 RP 2000 1000 interface to external components New C++ 1000 high uncertainty fault ui New C++ 1500 WheelSlipPlugin Legacy 10% 10% 100% C++ 400 OSRF 400 A lot of testing and tweaking will be done on 400this plugin SDF parameters manual process a LOT of manual effort goes into this CAD to URDF manual process New XM L 2000 a LOT of manual effort goes into this Rover Appearance shaders for rover model New GLSL 400 Effort under represented vipers im_t ools util scripts and tools Legacy 10% 30% 30% C++ 1000 RP 2000 1000 Gazebo compositor rewrite Legacy 0.10% 0.250% 1% C++ 284500 OSRF 284500 deleted from total, reuse as is for sdformat Legacy 0.00% 0.00% 0.00% C++ 32300 OSRF 32300 Gazebo

Inherited Expected New Code 325300 351620 26320 21600 303700 Modified As Is

*Note: These are a subset of modules so do not sum to the totals. 16 SEER-SEM Details Knowledge Base (KB) Adjustments v7 categories, 37 parameters vSet at Least, Likely, and Most to create a distribution vCOCOMO parameters set independently

17 SEER-SEM Details SEER-SEM COVID Parameter Adjustments

white RGSW RFSW RSIM Cost Parameter Definition Notes cells =KB Least Likely Most Least Likely Most Least Likely Most Impact* Overall capability of the software analysts Slight penalty to reflect the difficulty of team Analyst assigned to this development, in terms of member coordination of requirements Low Low+ Nom+ Low Nom- Nom+ Low Low+ Nom+ ~8% Capability their ability to function effectively as a team. analysis and design work resulting in Rate the team, not individuals. miscommunication of information. Average capability of programmers assigned Programmer to the development; includes factors related Slight penalty to reflect reduced programmer Low+ Nom- Hi Low+ Nom- Hi Low+ Nom- Hi 3% Capability to capability, motivation and working interaction in the working environment. environment. Represents the Ames software development Rates the development organization's use of Development organization with slight penalty for reduced software engineering processes and methods Nom Nom+ Hi- Nom Nom+ Hi- Nom Nom+ Hi- ~2% Practices Use capabilities due to COVID. Coordinated with that are considered to be "best practices". the VIPER Mobility software leads. Organizational and site diversity of the project team. This parameter should consider Increased Likely to account for being circumstances that prevent project team Multi Site Nom Hi Hi Nom Hi Hi Nom Hi Hi+ ~5% geographically distributed. RSIM Most could members from being able to collaborate on be worse than RGSW and RFSW. the spur of the moment. Consider physical location, time zone differences, etc. Resource Availability of the host and target machines to Lowered slightly to account for reduced Low Nom- Nom Low- Low+ Nom- Low Low+ Nom- ~6% Dedication the development organization. access to servers.

Resource and Diffuculty of access to development resources Increased slightly, especially on Most setting, Support and support. Physical and political separation Nom+ Hi- Vhi Nom+ Hi- Vhi Nom+ Hi Vhi 9% to account for longer wait times for support. Location must be considered. RSIM could be more severly impacted.

= Knowledge Base parameter settings changed due to COVID impact *Approximate cost Impact is to the SEER-SEM knowledge Base settings for the individual parameters

18 Analogy Software Cost Tool (ASCoT)

The NASA Analogy Software Cost Tool (ASCoT) is a web-based tool that provides a suite of estimation tools to support early lifecycle NASA Flight Software analysis. vClustering - Analogy estimation method that estimates software development effort using a variation of principle components to derive the clusters vKNN - Analogy estimation method using three nearest neighbors algorithm

NASA ASCoT provides Cluster Analysis and KNN for early lifecycle analysis. ASCoT is available in ONCE web site.

19 Analogy Software Cost Tool Results

VIPER Estimates KNN and Cluster Inputs: analysis compare Mission Size: Small Mission Type: Rover favorably with Destination: Inner Planetary Number of Instruments: 4 VIPER VIPER estimate. Number of Deployables: 3 RFSW + Inheritance: High RGSW Redundancy: Single String Estimate

KNN Effort: Redundancy = Single String Nearest Neighbors Neighbor's Effort Neighbor's Distance to VIPER Estimate: MPF 1080 1.08 619 WM 997 WMs Grail 868 1.1 DS1 1042.8 1.12 Cluster placement: Cluster 1 KNN SLOC: Redundancy = Single String Nearest Neighbors Neighbor's SLOC Neighbor's Distance to VIPER Delivered GLORY Estimate: MPF 94.3 0.78 236 GPM Core 103 LOC LRO 62.9 1.07 ESLOC Grail Grail 156.1 1.07 106 LADEE NuStar Cluster Effort: Redundancy = Single String Nearest Neighbors Neighbor's Effort Neighbor's Distance to VIPER OCO Estimate: Grail 868 1.1 619 WM WISE 619 WMs NuStar 493 1.12 LADEE 492 1.12

20 Findings

21 Estimate Results Summary

Note: WM – Work Months vEffort estimates for software development only vSEER-SEM and COCOMO estimates at 50% confidence Software effort from vLower than the KNN models appears to be vNear and within the Cluster range reasonable with KNN vReflects COVID impact and Cluster results

SEER-SEM Aggressive Conservative ASCoT ASCoT ASCoT ASCoT Element Estimate COCOMO COCOMO KNN KNN Range Cluster Cluster Range (WM) Estimate (WM) Estimate (WM) (WM) (WM) (WM) (WM) RGSW 80 78 89 RFSW 587 519 600 Substotal 667 597 689 997 868 - 1080 619 492 - 868 RSIM 266 270 290 - - - -

22 RGSW Development Effort CDF

v SEER-SEM VIPER Rover RGSW Development Effort CDF v Based on adjusted (Requirements through SW I&T) 100% KBs COCOMO 90% v Accounts for COVID Optimistic 78-95 WMs 80% v COCOMO conservative

70% estimate assumes Recommend Budget be between a 50th with COCOMO v Possibility that a 60% reserves and 70th without Conservative reserves 89-108 WMs larger amount of new 50% and modified code 40% will be required 30% v Higher volatility 20% Likelihood of Occurrence SEER v Required Reliability is 10% 80-98 WMs Software Class B- 0% 20 40 60 80 100 120 140 v COCOMO estimates Effort (Work Months) schedule to be between 20-26 Months

23 Using the Estimate v The rover software estimate modeled without COVID impacts compared favorably to the project’s internal rover software grassroots estimate v The COVID impacts modeled resulted in a cost increase of ~6% to the project’s rover software estimate v Monitor rover software actuals against the estimate for variances regularly to detect cost or schedule threats v Refine the estimate based on results of variance analysis or changing conditions

24 Summary

25 Lessons Learned – User Perspective v Advantages of parametric models v Model parameters provide structure to think about and tune for outliers or “black swan" events (e.g., COVID) v Greater flexibility, conducive to scenario analysis/what-if analysis v Use a combination of methodologies and tools to exploit strengths of each when building a life-cycle cost estimate v Cost estimating in a rapidly changing environment will require more frequent updates to the estimates and cost models

26 Lessons Learned – Estimator Perspective v Virtual software cost estimates can be done v Communication with the estimating team v Conduct frequent, regular meetings v WebEx helps keep team focused v Keep all members of the team involved v Current software cost estimating models may be able to capture the impacts of COVID v Appears possible that existing parametric model parameters can be used as proxies for COVID effects v Still need to validate / ground truth estimate with actuals v Need to determine if there are COVID effects that need to be modeled (long-term stress, “staycation” vs. vacation, etc) v We need to prepare for the next “black swan” event and adapt the models accordingly

27 Conclusions v The VIPER Rover software estimate was challenging for multiple reasons, particularly in terms of modeling COVID effects v The impact of COVID needs continued studying and may be useful for estimating cost of future “black swan” events v Assembled a knowledgeable virtual team representing the major project functions v Invested time for software sizing and paid attention to reused code and incorporated large open source modules that required adaptation, integration, and testing v Identified a specific set of cost parameters in the parametric estimating models with settings that were sensitive to a COVID environment v Established a benchmark at PDR that can be updated throughout the life of the project

28 Definitions

• ARC – Ames Research Center • EDL – Entry, Descent, Landing • ASCoT – Analogy Software Cost Tool • ESLOC – Equivalent (new) Source • BoE – Basis of Estimate Lines of Code • CADRe – Cost Analysis Data Requirement • FY – Fiscal Year • CCSDS – Consultative Committee for • GDS – Ground Data System Space Data Systems • GRAIL - Gravity Recovery and • CDF – Cumulative Distribution Function Interior Laboratory (“S” curve) • GRO – Gamma Ray Observatory • cFS – Core Flight Software • ICE – Independent Cost Estimate • C&DH – Command and Data Handling • JPL – Jet Propulsion Laboratory • COCOMO – COnstructive COst Model • KB – Knowledge Base • COVID– COronaVIrus Disease • KNN – Nearest Neighbors • CSCI – Computer Software Configuration • KSLOC –1,000 Source Lines of Code Item • LADEE – Lunar Atmosphere and Dust • DoD – Department of Defense Environment Explorer • DS1 – • LCC – Life Cycle Cost

29 Definitions

• LCROSS – Lunar Crater Observation and • RFSW – Rover Flight Software Sensing • RGSW – Rover Ground Software • LRO – Lunar Reconnaissance Orbiter • ROS2 – Robot Operating System2 • M2020 – • RP – Resource Prospector • MER – • RSIM – Rover SIMulations Software • MOS – Mission Operations Support • RSW – Rover Software • MPF – • SCAT – Software Cost Analysis Tool • MSL – • SEER-SEM – Software Evaluation and • NASA – National and Space Estimation of Resources-Software Administration Estimating Model • NPR – NASA Procedural Requirement • SLOC – Source Lines of Code • NuSTAR – Nuclear Spectroscopic • SMART – Software Measures and Telescope Array Analysis Reporting Tool • ONCE – One NASA Cost Engineering • SQA – Software Quality Assurance • OSRF – Open Source Robotics Foundation • STS – Space Transportation System • PDR – Preliminary Design Review • VIPER – Volatiles Investigating Polar Exploration Rover • PySLiC – Python Software Line Counter • WM – Work Month

30 References

Artist’s Impression of VIPER on the Lunar Surface, VIPER home page, https://www.nasa.gov/viper. Jorgensen, M. and Grimstad, S., Over-optimism in software development projects: “The Winner’s Curse”, 15th International Conference on Electronics, Communications and Computers, ISBN: 0-7695-2283-1, IEEE, 2005. NASA Procedural Requirement (NPR) 7120.8A, NASA Research and Technology Program and Project Management Requirements”, https://nodis3.gsfc.nasa.gov/, September 14, 2018. NASA Procedural Requirement (NPR) 7120.5E, NASA Space Flight Program and Project Management Requirements”, https://nodis3.gsfc.nasa.gov/, Change Number 18, March 12, 2020. Taber, William, Python Source Line Counter (PySLiC) User Guide, JPL, Release Version 1.0, May 22, 2019. NASA Ames Research Center, Resource Prospector Mission Rover Analysis of Alternatives Flight Software Cost Estimate, Software Basis of Estimate Review, December 19, 2013. Johnson, James, et al., One NASA Cost Engineering Database (ONCE) Overview and Update, 2019 NASA Cost and Schedule Symposium, Houston, TX, August 2019. Hihn, Jairus, et al., ASCoT, the NASA Analogy Software Cost Tool Suite: Expanding Our Estimation Horizons, 2018 IEEE Aerospace Conference, Big Sky, MT, March 2018. One NASA Cost Engineering (ONCE) Database, available at: https://oncedata.hq.nasa.gov/Main.aspx Software Measures and Analysis Reporting Tool (SMART) repository, JPL, 2020. Stukes, Sherry, et al., JPL Mission Software: State of Software Report 2018, JPL, September 2019. System Evaluation and Estimation of Resources - Software Estimating Model (SEER-SEM), Galorath Incorporated, Version 8.3, August 27, 2020. Lum, Karen, Software Cost Analysis Tool Users Document, JPL D-26304, Version 1.2, October 13, 2005.

31