Software Cost Estimation for COVID 2021 NASA Cost and Schedule Symposium Virtual Event April 7, 2021 Sherry Stukes Georgia Bajjalieh Jet Propulsion Laboratory, California Logical-R Joint Venture, LLC Institute of Technology NASA Ames Research Center 4800 Oak Grove Drive Moffett Field CA 94035 Pasadena, CA 91109 [email protected] [email protected] DISCLAIMER The cost information contained in this document is of a budgetary and planning nature and is intended for informational purposes only. It does not constitute a commitment on the part of NASA, JPL, and/or Caltech. © 2021 California Institute of Technology. Government sponsorship acknowledged. Contents vBackground and Overview vHistorical Data vModels and Methodologies vFindings vSummary 2 Background and Overview 3 Background - VIPER vVIPER rover mission to the moon’s south pole v100 earth-days vMission to find water ice vWhere is water ice located? vWhere are the highest concentrations? vHow deep is the ice? vFirst U.S. rover launched to the lunar surface since the Apollo missions in the early 1970’s *VIPER in Formulation Phase when the Rover Software Cost Estimate was conducted in 2020 Reference: VIPER home page, https://www.nasa.gov/viper 4 Key Cost Estimating Challenges (1) v Parametric models are not designed to capture unique aspects of VIPER Rover Software v VIPER is being developed as a research and technology project, rather than as a space flight program v VIPER is a risk tolerant mission similar to NASA “Class D” payloads v Key portions of the rover’s software are being designed and implemented as ground software v Reflecting affects of tailored NASA Procedural Requirement (NPR) requirements v No direct Lunar Rover analogies v Draw on bits and pieces from prior projects v Look for systems as similar as possible 5 Key Cost Estimating Challenges (2) v Integration of large open source software products v RGSW – Gazebo (3D dynamic simulator) v RSIM – ROS2 (Robotic Operating System stereo image processing and tools) v Impact of COVID v Models do not directly capture pandemic cost impact v Agile development may be less efficient as Agile principles promote face-to-face meetings and a highly collaborative environment v Personnel productivity may be different than expected 6 Rover Flight Software Architecture v3 CSCIs vRover Ground Software (RGSW) vRover Flight Software (RFSW) vRover SIMulations Software (RSIM) vDecomposed to module level for software lines of code sizing vPlanned for high software reuse vIntegration large open source modules 7 Historical Data 8 Historical Data – Resources v Reviewed Resource Prospector BoE document dated 12/19/2013 v Limited information and details for five BoEs v Researched updated information v Data Sources v JPL SMART (Software Measures and Analysis Reporting Tool) – JPL proprietary v CADRe (Cost Analysis Data Requirement) – available on ONCE site https://oncedata.hq.nasa.gov/Main.aspx v ASCoT (Analogy Software Cost Tool) – also available on ONCE site v NASA projects v Used “JPL Mission Software: State of Software Report 2018” as a reference for software size 9 Historical Data Collection Project Data Project Center Source Why Relevant Reviewed CADRe and ASCoT data with LADEE ARC Lunar orbiter, Class D mission Software Project Manger for LADEE LCROSS ARC Only sizing data available Robotic lunar impactor, low-cost LRO GSFC ASCoT data only Robotic lunar orbiter, 3D mapping Data from CADRe, ASCoT, and SMART GRAIL JPL Lunar orbiter and simulation software (used for RFSW and RSIM) Rover, tech demo, all new software MPF JPL Data from CADRe code MER JPL Data from CADRe, ASCoT, and SMART Rover MSL JPL Data from CADRe, ASCoT, and SMART Rover Preliminary software size data from M2020 JPL Rover, subset analogous to RGSW Project GRO GSFC SMART data only Simulation software STS GRC SMART data only Simulation software Centaur STS 51E JSC SMART data only Simulation software and 61I 10 Historical Data Collection What We Found v Software database content v Variety of sources v Across NASA Centers v Organized by element v Established NASA sources v Easily accessible VIPER rover software elements compare favorably with historical data. 11 Historical Flight Software Data Analogies VIPER RFSW size is in family with JPL Reference: JPL State of Software Report, 2018 historical projects. 12 Models and Methodologies 13 Models and Methodologies SEER-SEM and v Parametric Models COCOMO are in v Primary – SEER-SEM wide use across vDeveloped by Galorath Incorporated NASA. vV8.3 vWidely used within NASA v Cross-check – COCOMO (SCAT) vJPL version of COCOMO II vModel implemented in Excel and enhanced to capture uncertainty with Monte Carlo add-in vAvailable to NASA Centers from JPL v Analogy Software Cost Tool – (ASCoT) v Rules of Thumb (SLOC/WM) v Compare and contrast cost estimates 14 SEER-SEM Details Input Data Summary vSLOC (Logical) Rover Software Size Summary vNew SW Reuse Reuse % % Reuse New Total ESLOC Element “as is” Modified Redesign Recode Source SLOC SLOC SLOC vReused “as is” Units SLOC SLOC % % Project SLOC SLOC SLOC vReused RGSW 26371 2860 25 50 RP 10638 40490 17256 modified RFSW 55858 85885 7 26 LADEE 53530 196096 89218 v % re-design RSIM 303700 21600 15 15 RP, OSRF 26320 351930 30852 v % re-code Notes Terms Projects Logical SLOC SLOC - Source Lines of Code LADEE - Lunar Atmosphere Dust and Environment Explorer v % re-test ESLOC - Equivalent (new) Source Lines of Code RP -Resource Prospector RFSW - Rover Flight Software OSRF - Open Source Robotics Foundation vTotal delivered RGSW - Rover Ground Software vEquivalent new RSIM - Rover Simulation Software vKnowledge Bases vEstablishes Rover Software Knowledge Base Selections default Software Acquistion Development Development Platform Application parameter Element Method Method Standard settings RFSW Unmanned Space Flight Systems New & Reused Agile - Novice Commercial vBased on RGSW Unmanned Space Flight Systems New & Reused Agile - Novice Commercial industry RSIM Unmanned Space Simulation New & Reused Agile - Novice Commercial averages vModified for VIPER 15 SEER-SEM Details Software Sizing vSample of software sizing vSized by module vConducted software reuse study vCounted with PySLiC Logical Logical Rework Rework Retest SLOC Soure SLOC repo/folder Description Category Design Code Code Language Inherited Project Expected Notes Modified rockCraterGen rock and crater generation Legacy 15% 30% 50% Matlab 1000 RP 1200 1000 craterDisplay crater marking GUI Legacy 30% 30% 20% C++ 1200 RP 2200 1200 moonShine synthetic DEM generator Legacy 10% 20% 40% C++ 6400 RP 7000 6400 craterProfile improve crater profile New C++ 500 tile to Gazebo model tile_to_gazebo processing Legacy 50% 40% 20% Python 650 RP 800 650 NSS protocol New C++ 300 RP NSS/NIRVSS Sim behavior Legacy 15% 50% 10% Python 1000 RP 2000 1000 interface to external components New C++ 1000 high uncertainty fault ui New C++ 1500 WheelSlipPlugin Legacy 10% 10% 100% C++ 400 OSRF 400 A lot of testing and tweaking will be done on 400this plugin SDF parameters manual process a LOT of manual effort goes into this CAD to URDF manual process New XM L 2000 a LOT of manual effort goes into this Rover Appearance shaders for rover model New GLSL 400 Effort under represented vipers im_t ools util scripts and tools Legacy 10% 30% 30% C++ 1000 RP 2000 1000 Gazebo compositor rewrite Legacy 0.10% 0.250% 1% C++ 284500 OSRF 284500 deleted from total, reuse as is for sdformat Legacy 0.00% 0.00% 0.00% C++ 32300 OSRF 32300 Gazebo Inherited Expected New Code 325300 351620 26320 21600 303700 Modified As Is *Note: These are a subset of modules so do not sum to the totals. 16 SEER-SEM Details Knowledge Base (KB) Adjustments v7 categories, 37 parameters vSet at Least, Likely, and Most to create a distribution vCOCOMO parameters set independently 17 SEER-SEM Details SEER-SEM COVID Parameter Adjustments white RGSW RFSW RSIM Cost Parameter Definition Notes cells =KB Least Likely Most Least Likely Most Least Likely Most Impact* Overall capability of the software analysts Slight penalty to reflect the difficulty of team Analyst assigned to this development, in terms of member coordination of requirements Low Low+ Nom+ Low Nom- Nom+ Low Low+ Nom+ ~8% Capability their ability to function effectively as a team. analysis and design work resulting in Rate the team, not individuals. miscommunication of information. Average capability of programmers assigned Programmer to the development; includes factors related Slight penalty to reflect reduced programmer Low+ Nom- Hi Low+ Nom- Hi Low+ Nom- Hi 3% Capability to capability, motivation and working interaction in the working environment. environment. Represents the Ames software development Rates the development organization's use of Development organization with slight penalty for reduced software engineering processes and methods Nom Nom+ Hi- Nom Nom+ Hi- Nom Nom+ Hi- ~2% Practices Use capabilities due to COVID. Coordinated with that are considered to be "best practices". the VIPER Mobility software leads. Organizational and site diversity of the project team. This parameter should consider Increased Likely to account for being circumstances that prevent project team Multi Site Nom Hi Hi Nom Hi Hi Nom Hi Hi+ ~5% geographically distributed. RSIM Most could members from being able to collaborate on be worse than RGSW and RFSW. the spur of the moment. Consider physical location, time zone differences, etc. Resource Availability of the host and target machines to Lowered slightly to account for reduced Low Nom- Nom Low-