NASA Instrument Cost Model: Mission Class’s Impact, Imputation, Multiple Builds and More: New Features in the Latest Version of NICM Joseph Mrozinski, NICM Task Lead [email protected] April 15, 2021 Copyright 2021 California Institute of Technology. Government sponsorship acknowledged. Disclaimer: The cost information contained in this document is of a budgetary and planning nature and is intended for informational purposes only. It does not constitute a commitment on the part of JPL and/or Caltech. Agenda • Acknowledgements • NICM Introduction • NICM 9: Latest Tool Improvements • Mission Class-Based Subsystem CERs • Imputation-Based System CERs • Multiple Build Estimations • Search Engine Improvements • Screenshot Generator • Space Scientific Instrument Taxonomy (SSIT) jpl.nasa.gov Agenda • Acknowledgements • NICM Introduction • NICM 9: Latest Tool Improvements • Mission Class-Based Subsystem CERs • Imputation-Based System CERs • Multiple Build Estimations • Search Engine Improvements • Screenshot Generator • Space Scientific Instrument Taxonomy (SSIT) jpl.nasa.gov NICM Stakeholders • Sponsor: NASA HQ OCFO/SID • Special thank you to James Johnson • Legacy Co-Sponsor: JPL Cost Estimation & Pricing Section • Development Team • JPL Systems Modeling, Analysis & Architectures Group • JPL Engineering Cost Estimation Group • JPL Technical Division Experts • Science • Communications, Tracking, and Radar • Instruments and Science Data Systems • Mechanical Systems • Last but not least, all of the NASA Centers, Contractors, Universities and others who have built instruments and contribute data to NICM jpl.nasa.gov NICM Team NICM 9 Team • Gary Ball • Luther Beegle • Justin Boland • Kyle Brown • Robert Cesarone • Mike DiNicola • Michael Fong • Melissa Hooke • Joe Mrozinski • Al Nash • Michael Saing • Sherry Stukes • Marc Walch jpl.nasa.gov NICM Team NICM Alumni/Advisors NICM Consultants • Daniel Belter • Kamrooz Parchamazad • Ed Caro • Grace Chen • John Pearson • George Fraschetti • Jerry Clark • Matthew Ramirez • Tom Fraschetti • George Fox • Carrie Selski • Dean Johnson • Hamid Habib-Agahi • Cesar Sepulveda • Ken Klaasen • John Jack • Louise Veilleux • Chris Paine • Leora Juster • Keith Warfield • Brian Sutin • Eric Kwan • Wayne Zimmerman • David Swenson • Richard Levin • Xiaoyan Zhou jpl.nasa.gov NICM Introduction Joseph Mrozinski, NICM Task Lead [email protected] November 2, 2020 Copyright 2020 California Institute of Technology. Government sponsorship acknowledged NICM Introduction • NICM is the NASA Instrument Cost Model: • Instrument Cost and Schedule (B/C/D) Estimating Tool Suite based off of previously flown space flight instruments across all of NASA • Includes objective-input-based parametric cost and schedule models, cost and schedule analogy tools and JCL capabilities. • Models exist at both the total instrument and instrument subsystem levels.. • Civil Servant copy includes all normalized data as well as a data Search Engine • Users: • All NASA Centers, Contractors, Universities etc. • Proposal Teams as well as by Proposal Evaluators. • Over 600 individuals have attended NICM Training sessions • For Training Contact: [email protected] • Download the NICM Excel file from: • www.oncedata.com (Civil and Contractor version) • www.software.nasa.gov (Contractor Version only) jpl.nasa.gov A Brief NICM History • NICM began collecting data in 2004 • Foreshadowing: You’ll notice our CERs are all in FY04. This is why. • NICM has collected and normalized data non-stop 2004-present. • Newly collected data feeds the updates to the NICM CERs. • Trying to cram the highlights of 16 years of NICM history into 8 bullets: • NICM I: released in 2006, with each tool in an individual workbook. • NICM II-III: a flood of new data pours in after the NICM I release. • NICM IV: In-situ instruments added. All tools combined into a single workbook. • NICM V: schedule estimating and JCL added. • NICM VI: NICM-E capability and Cluster Tool added. • NICM VII: Telescope estimating capability added. • NICM VIII: Mission Class as a cost driver added. • NICM 9: Data Imputation Utilized, Multiple Build estimates introduced jpl.nasa.gov NICM Data • Interviewing, Analyzing, Normalizing and Reviewing technical and cost Data is the heart and main strength of the NICM. Good models require good data. • We are stringent when it comes to the quality, applicability and completeness of the data: before data is used for modeling, all records and normalization approaches are reviewed and blessed by both individuals who built the hardware as well as the multi-disciplinary NICM Team. • NICM 9 includes 299 Data Records jpl.nasa.gov Modeling Methodology • Cluster Analysis • Identifies Instrument Groupings from Attribute Values • Assesses Consistency of Groups with Instrument Types • Principal Components Analysis • Standard Data Mining Technique that • Finds Potential Cost Drivers from Instrument Attributes • Identifies NICM Data Outliers – Revisit data with technical experts • Finds separation in the data (i.e. clustering) • Addresses multi-collinearity in data for regression analysis • Bootstrap Cross Validation • Bootstrap: Process for generating meaningful statistics without assuming asymptotic normality. • Cross Validation: Partitioning of data set into training and testing sets. Out-of-sample validation. • Bootstrap technique also used to perform statistical tests for regression analysis. • Imputation: Allows for use of incomplete records. jpl.nasa.gov Model Types • NICM contains the following modeling types: • Cost Estimating Relationships (CERs). • Schedule Estimating Relationships (SERs). • The CERs exist at two different levels: • The instrument System or Total Level. • The instrument Subsystem and Wrap Level. • The SERs only produce a total instrument schedule (they do not provide subsystem schedules). jpl.nasa.gov Example Model: Optical Planetary CER Imputed Instruments used in this CER: instruments used: ALICE_Rosetta CFI CIRS ALICE CRISM CRISP CTX IUVS DLRE IRAC IRS MDI ISS ITS JunoCam Ralph LOLA LORRI LROC SECCHI M3 MARCI MASCS UVCS MCS MDIS MICAS MIPS MIR MLA MOC-MO MOLA-MO MRI MSI NavCam NIR NIS NLR NSP ONC PMIRR TES_MO THEMIS TLP UVIS UVS - Juno VIMS VIS_LCROSS VSP Alternative form of equation: ���� = 392 ����������!.#$ ����!.%& �����!.%' ��� ��������������� ().!& where MissionClassBin = 0 if Class A or B, & MissionClassBin = 1 if Class C or D jpl.nasa.gov NICM 9: Latest Tool Improvements Joseph Mrozinski, NICM Task Lead [email protected] November 2, 2020 Copyright 2020 California Institute of Technology. Government sponsorship acknowledged Agenda • Acknowledgements • NICM Introduction • NICM 9: Latest Tool Improvements • Mission Class-Based Subsystem CERs • Imputation-Based System CERs • Multiple Build Estimations • Search Engine Improvements • Screenshot Generator • Space Scientific Instrument Taxonomy (SSIT) jpl.nasa.gov Mission Class-Based Subsystem CERs • NICM VIII introduced the use of Mission Class (A/B vs. C/D) in the NICM System Tool. However, Mission Class was not examined at the instrument subsystem level. • NICM 9 expands on this by finding Mission Class as a statistically meaningful driver for the following instrument subsystems: • Electronics Earth Orbiting • Electronics Planetary • Mechanical/Structures • Thermal • Mission Class was not found to be a driver for the following subsystems: • Optics • Antenna • Detectors • Software jpl.nasa.gov Agenda • Acknowledgements • NICM Introduction • NICM 9: Latest Tool Improvements • Mission Class-Based Subsystem CERs • Imputation-Based System CERs • Multiple Build Estimations • Search Engine Improvements • Screenshot Generator • Space Scientific Instrument Taxonomy (SSIT) jpl.nasa.gov Imputation Data imputation is a statistical method used to handle missing data in a dataset by probabilistically filling a data observation’s missing value(s) based on 1) the partial data that is available for that data observation and 2) the completed observations (those not missing any values) in the dataset. Advantages: 1. Instruments for which the mass and peak power are known, but the cost is missing still contain knowledge about NASA instruments. 2. The addition of these data observations increases sample size, which, in general, leads to CERs with greater statistical power (e.g. a better understanding of the effect size of variables used in CERs) and increases predictive strength of the CERs. 3. Bringing these incomplete instrument records into the NICM tool increases the number of analogy instruments available to the user through the Search Engine and the other analogy-based subtools in NICM. jpl.nasa.gov Imputation-Based System CERs • Using data imputation, the NICM team was able to make improvements (green) to the following System Tool CERs: CER Complete Imputed V8.5 V9 V8.5 V9 Records Records R^2 R^2 PE PE Optical Planetary 43 6 89% 89% 48% 45% Optical Earth Orbiting 35 10 85% 85% 48% 42% Passive Microwave 12 2 86% 86% 37% 34% Fields 12 2 74% 74% 44% 41% Particles Planetary 30 7 60% 62% 52% 47% Particles Earth Orbiting 24 3 77% 77% 53% 53% Body Mounted 13 1 72% 72% 64% 62% Arm/Mast Mounted 12 1 59% 60% 52% 50% jpl.nasa.gov Imputation-Based System CER Example Passive Microwave NICM 9 NICM 8.5 Passive Microwave Instruments Passive Microwave Instruments Cost = 1,686 * TotalMass^0.39 * Cost = 1,741 * TotalMass^0.38 * TotalMaxPwr^0.38 TotalMaxPwr^0.39 2 2 R = 86%, PE = 34%, Ncom = 12, Nimp = 2 R = 86%, PE = 37%, N = 12 jpl.nasa.gov Agenda • Acknowledgements • NICM Introduction • NICM 9: Latest Tool Improvements