PCEC Updates for 2018

2018 NASA Cost & Schedule Symposium August 15, 2018

Brian Alford Mark Jacobs Booz Allen Hamilton TGS Consultants Shawn Hayes Richard Webb TGS Consultants KAR Enterprises

NASA MSFC

Victory MIPSSSolutions Team SB Diversity Overview

• PCEC Overview • Robotic Spacecraft Updates • Crewed & Space Transportation System Update • Summary

Victory Solutions MIPSS Team 2 What is PCEC?

• The Project Cost Estimating Capability (PCEC) is the primary NASA-sponsored parametric cost tool for spacecraft estimates – Developed and maintained by NASA at MSFC beginning in late 2013 – Excel Add-in that provides capabilities and cost estimating artifacts used to build a spacecraft cost estimate in Excel – Based on more than 70 missions/system elements, but with separate approaches for modeling different types of systems • Robotic Spacecraft (Robotic SC) • Crewed & Space Transportation Systems (CASTS) – Completely transparent tool: no code passwords, protected sheets, etc. – Available to the general public via ONCE and the NASA Software Catalog (https://software.nasa.gov/)

Victory Solutions MIPSS Team 3 Current State of PCEC

• PCEC v2.2.1 was released in May 2018 – Fixes minor bugs in some templates and a minor code revision – No updates to the Library other than latest New Start Inflation indices – Update provided via Excel updater file; includes routines for making corrections to user-installed Interface and existing v2.2 estimates • Current user counts as of late July 2018 – 490+ Users / Requestors – 45+ Countries represented – ~60 users added since last year’s Symposium

Victory Solutions MIPSS Team 4 PCEC User Distribution

Estimated based on user-provided data

Victory Solutions MIPSS Team 5 PCEC ROBOTIC SPACECRAFT UPDATES

6 PCEC Robotic Spacecraft Topics

1. New Missions 2. Entry System CERs 3. Milestone-Specific Data Analyses

Victory Solutions MIPSS Team 7 Current Robotic SC Missions & New Candidates

• Missions 1-42 are used for PCEC v2.2.1 CERs • Additional Candidates 1-6 have been tested with PCEC v2.2.1; Overall results are generally good • Data normalization is complete for Additional Candidates 1-4 and nearly complete for 5-6 • Additional Candidates 7-11 should have launch CADRe data available soon (before the end of FY19) • PCEC CERs could be updated in FY19 incorporating at least the first 6 additional candidates

Victory Solutions MIPSS Team 8 PCEC v2.2.1 Results for New Candidates Actual $)/Actual $ $)/Actual Actual – (Estimate

• Results compare the PCEC estimate to unadjusted Launch CADRe cost data • PCEC Phase BCD estimates are within 20% for 4 missions (average error is -19%) • Several of these missions experienced ‘External Impacts’ that will be normalized out of the data used for the PCEC CERs • External Impacts are the primary driver for the bias towards low estimates for 3 of these missions (GPM, MMS, and GOES-R) Victory Solutions MIPSS Team 9 PCEC Entry System CERs NASA WBS Elements Included

GOAL: Derive CERs to capture the cost of various atmospheric entry hardware systems for robotic spacecraft missions (Phases B-D)

• The PCEC Entry System elements are included in the spacecraft (SC) portion of the WBS, but separate from the more typical subsystems. – Thermal Protections System (TPS) is WBS 6.5.1.8 – Parachute System is WBS 6.5.1.9 – Airbag System is WBS 6.5.1.10

Victory Solutions MIPSS Team 10 Entry System CERs Approach & Mission Set

• Costs for Thermal Protection Systems (TPS), Parachutes, and Air Bag Entry System elements are not included in PCEC v2.2.1 • Costs for these items have been derived using data from 9 missions • PCA was used to help identify technical characteristics driving costs • CERs have been developed for Non-recurring, Recurring, and Phase BCD Total costs

Victory Solutions MIPSS Team 11 Entry System CERs Thermal Protection Systems (TPS), 1 of 2

• Definition: Covers design, fabrication, and I&T for all TPS material; Includes TPS on Heat Shield, Backshell, and Parachute; Does not include structure, adhesive, or other supporting elements TPS Non-Recurring Cost (NRC) TPS Recurring Cost (RC)

• Input Parameters: • Input Parameters: – PD - Peak Deceleration (g) – PD - Peak Deceleration (g) – MPS - Mean Pressure at the Surface (kPa) – HD – Heatshield Diameter (m) • Cost Estimating Relationship (CER): – HPU – Heatshield Production Units 1236 × (!")0.3761 × (#!$)−0.1847 • Cost Estimating Relationship (CER): • Adjusted R^2: 0.52 1437 × (%!&) × (!")0.2625 × (%")0.6958 • Range of Error: -51 to 73% • Adjusted R^2: 0.70

Victory Solutions MIPSS Team • Range of Error: -32 to 77% 12 Entry System CERs Thermal Protection Systems (TPS), 2 of 2

• Definition: Covers design, fabrication, and I&T for TPS material; Does not include structure, adhesive, or other supporting elements

TPS – Alternate CER (NRC+RC)

• Input Parameters: – PD - Peak Deceleration (g) – SEM - System Entry Mass (kg) • Cost Estimating Relationship (CER): 230 × (!")0.4459 × (#$%)0.4045 • Adjusted R^2: 0.93 • Range of Error: -26 to 26%

Victory Solutions MIPSS Team 13 Entry System CERs Parachutes, 1 of 2

• Definition: Covers design, fabrication, and I&T for the Parachute System; Includes all flight parachutes, lines, and mortar; Does not include support structure Parachutes Non-Recurring Cost (NRC) Parachutes Recurring Cost (RC)

• Input Parameters: • Input Parameters: – MPD: Main Parachute Diameter (m) – PSM - Parachute System Mass (Chutes, • Cost Estimating Relationship (CER): Mortar) (kg) 8.5 × (!"#)1.9541 – MPS - Mean Pressure at the Surface (kPa) – PPU – Parachute Production Units • Adjusted R^2: 0.50 • Cost Estimating Relationship (CER): • Range of Error: -51 to 119% 564 × (""$) × ("%!)0.4983 × (!"%)−0.0880 • Adjusted R^2: 0.83 Victory Solutions MIPSS Team 14 • Range of Error: -36 to 32% Entry System CERs Parachutes, 2 of 2

• Definition: Covers design, fabrication, and I&T for the Parachute System; Includes all flight parachutes, lines, and mortar; Does not include support structure

Parachutes – Alternate CER (NRC+RC)

• Input Parameters: – MPD: Main Parachute Diameter (m) – PPU – Parachute Production Units • Cost Estimating Relationship (CER): 37 × (!"#)1.7903 × (""$)0.91 • Adjusted R^2: 0.87 • Range of Error: -42 to 44%

Victory Solutions MIPSS Team 15 Entry System CERs Air Bag Deceleration Systems

• Definition: Covers design, fabrication, and I&T for an Air Bag Deceleration System; Includes all air bags, and inflation system; Does not include support structure

Air Bag Deceleration – CER (NRC+RC)

Development Cost vs Air Bag System Mass • Input Parameters: – ASM: Airbag System Mass (kg) – APU – Airbag Production Units • Cost Estimating Relationship (CER): 6338 + (72.459 × !"# × !$%) • Adjusted R^2: 1 • Range of Error: -15 to 3%

Victory Solutions MIPSS Team 16 Milestone-Specific Analyses Background

• Question: Can we correlate cost model inputs known at the start of Phase B to Actual costs (at launch)?

• Cost models are typically developed by correlating technical/programmatic inputs at launch to actual costs • If CERs were developed using Phase B inputs and Actual costs, would they perform better than typical cost models at the start of Phase B?

GOAL: Improve understanding of how inputs known at the start of Phase B correlate to actual costs at launch; Approach utilizes the same cost data normalization process applied to PCEC Robotic Missions and explores correlations (new CERs) and relationships between changes in cost model inputs and costs (Project & PCEC estimates)

Victory Solutions MIPSS Team 17 Milestone-Specific Analyses Phase B Cost Data Normalization

• Cost data representing the start of Phase B has been normalized for 24 projects (same process as used for PCEC v2.2) • Most of these projects are AO missions (and CSR data was used) • Difficult to clearly identify/collect Phase B start data for many projects

Victory Solutions MIPSS Team 18 Milestone-Specific Analyses CER Results Overview

Case 1 CER Results: • Use the exact same CER inputs by subsystem as the v2.2.1 CERs • Error range at the Flight System Level – CBE: -72% to 66% CBE + Contingency: -73% to 67% • CERs are oversubscribed since the number of data points used to develop the new CERs are significantly less than the PCEC v2.2 CERs (~20 vs ~40 data points)

Case 2 CER Results: • Use the same CER discovery process employed during the development of the PCEC v2.2.1 CERs; different inputs allowed • Error range at the Flight System Level – CBE: -59% to 36% CBE + Contingency: -59% to 39% • Some of the inputs are counter-intuitive; These CERs should be refined before considering release

Victory Solutions MIPSS Team 19 Milestone-Specific Analyses Results Summary

Based on Development (BCD) $ w/o Reserves, LV, Instruments, GDS, or Science Team

• Results using the ‘New Phase B CERs’ appear a little better than using PCEC v2.2 for the 24 missions used in this analysis • ‘Robustness’ of the New Phase B CERs may be questionable based on results using missions outside this 24-mission data set Victory Solutions MIPSS Team 20 Milestone-Specific Analyses Interesting CER Variable Observations

• Mass remains important across all CER cases • Heritage/Parts Rating variables become more important and schedule variables become less important when comparing Case 2 CERs to the original PCEC v2.2 CERs

Victory Solutions MIPSS Team 21 Milestone-Specific Analyses Cost Growth Correlations to Input Changes

Schedule Growth Mass Growth Change in Heritage

• Correlations between cost growth from Phase B and changes in technical and programmatic inputs were explored; however, good correlations between changes in cost model inputs and cost growth were not found • Of all the cost model inputs explored, schedule appears to have a much stronger correlation to cost growth than other inputs • Mass growth did not appear to be a driver for flight system cost growth • Although most projects had a much lower heritage rating at launch than estimated at Phase B, loss of heritage did not appear to be a driver for cost growth

Victory Solutions MIPSS Team 22 Milestone-Specific Analyses CER Performance Summary Actual $)/Actual $ $)/Actual Actual – (Estimate Underestimating

Victory Solutions MIPSS Team 23 Milestone-Specific Analyses Case Comparisons by Mission )/Actual Underestimating Predicted - Actual % Difference = (

Based on Development (BCD) $s w/o Reserves, LV, Instruments, GDS, or Science Team

Victory Solutions MIPSS Team 24 Milestone-Specific Analyses Overall Error Comparisons by Case )/Actual Underestimating Predicted - Actual % Difference = (

Based on Development (BCD) $s w/o Reserves, LV, Instruments, GDS, or Science Team

Victory Solutions MIPSS Team 25 Milestone-Specific Analyses Error Histogram Comparison by Case )/Actual Predicted Underestimating -

Actual Based on Development (BCD) $ w/o Reserves, LV, Instruments, GDS, or Science Team % Difference = (

• Case 1 performs similarly to PCEC v2.2 but the CERs are over constrained • Case 2 has a tighter error range than PCEC v2.2 but does not perform as well on a mission by mission basis • PCEC v2.2 CERs have the lowest error in the majority of cases Victory Solutions MIPSS Team 26 Milestone-Specific Analyses Findings & Conclusions

• New CERs created with Phase B Inputs and Launched Costs show potential to improve modeling accuracy at Phase B • Observed trend of higher importance of heritage & parts inputs for the Phase B CERs, where schedule was more critical for the Launch CERs • Some inputs for the New Phase B CERs are not working as expected – Similar experience creating the PCEC CERs – refinements to eliminate counter- intuitive inputs did not significantly change parametric performance • Improvements are needed to refine the inputs – Attempted to create ‘Figures-of-Merit’ combining various inputs w/o success, but further exploration of input candidates may reveal some Phase B-unique items • Adding data points would be helpful – The 24 mission set limited the number of usable CERs inputs • Correlating Phase B Inputs to Launch Actual Costs embeds reserves – Can complicate comparisons to lower-level project costs

Victory Solutions MIPSS Team 27 PCEC CASTS MODEL UPDATES

28 Propulsion Cost Model (PCM) Overview

• What is PCM? – One of two add-ons in-work to CASTS • Standalone model to PCEC/CASTS • Linkable to PCEC estimate similar to NICM, MOCET – Second add-on: Architecture Model • Life Cycle Cost, time-phased, f(activity level), multiple elements – Activity level: fixed and variable (marginal) cost as function of flight rate over time, lot buy size, learning effects, etc. • Why PCM? – Plato: “Necessity is the mother of invention”(?) – Or not . . .

I don’t think necessity is the mother of invention. Invention, in my opinion, arises directly from idleness, possibly also from laziness—to save oneself trouble. — Agatha Christie

Victory Solutions MIPSS Team 29 PCM Capability

PCM Capability – Ultimately: Liquid Rocket Engines, Solid Rocket Motors, Nuclear Thermal Propulsion, . . . – Current focus/near term release = Liquid Rocket Engines Propulsion Cost Model Current Focus

Liquid Solid Nuclear Engines Motors Themal Other? Engine Cycle Monolithic Thermionic Hypersonic Thrust Segmented Thermoelec RBCC Propellants Small Cycle Ion Test Approach Total Impulse Thermal Ctl Solar Sail Availability – Similar approach to PCEC/CASTS Current release – Publicly released model (spreadsheet) target is Fall 2018 – Unrestricted and Restricted documentation (Liquid Engine) • Manual plus (restricted) source database/calibrations

Victory Solutions MIPSS Team 30 PCM Liquid Engines Summary

• Based on Liquid Rocket Engine Cost Model (LRECM) – Developed by Rocketdyne (circa 1992-2003) – “Bought” by NASA mid-90’s; updated mid ‘00’s – “Engineering” model – limited number data points

• Modifications for PCM version – Adding additional data points • Propellant combinations, pressure (versus pump) – fed – Modifying/changing CER’s – Adding Functional Breakdowns • Labor (Engineering(s), QA, Test, etc.) + Materials & Subcontracts • System elements – pump, powerhead, controller, etc. • “Wraps” (SE&I, Program Management, Government Support, etc.)

Victory Solutions MIPSS Team 31 PCM Liquid Engines Cost Elements

Design, Development, Test, & Engineering Production Design / Development Engineering Labor Total Production Quantity Development Test Hardware Average Production Rate per Year System Test Hardware Integration,Assembly,Checkout Theoretical First Unit Hardware System Test Operations Integration,Assembly,Checkout System Test Labor System Engineering & Integration Development/Qualification Test Propellants Program Management Tooling and Ground Support Equipment Total Theoretical First Unit Tooling Mechanical/Electrical GSE Total Production Cost System Engineering & Integration Average Unit Production Cost Program Management DDT&E Total Operations and Support

Total Production Quantity

Ops and Support Cost per Engine per Flight Total Ops and Support Cost Average O&S Cost per Engine

Victory Solutions MIPSS Team 32 PCM Source Database

Added to Chamber LRECM Thrust - Engine Reus/ Pressure Data Set Engine Vac (Klbf) Cycle Propellant Expend (kpsia) F1 1,748 Gas Gen RP/LOX Expend 982 J2 230 Gas Gen LH2/LOX Expend 763 X J2X 294 Gas Gen LH2/LOX Expend 1,340 X LR91 103 Gas Gen Hypergolic Expend 860 X LR87 269 Gas Gen Hypergolic Expend 857 O MA5 463 Gas Gen RP/LOX Expend 711 O RS27 232 Gas Gen RP/LOX Expend 700 RS68 745 Gas Gen LH2/LOX Expend 1,488 X Viking VI 168 Gas Gen Hypergolic Expend 800 X LM Ascent 4 Press-Fed Hypergolic Expend 100 X LM Descent 10 Press-Fed Hypergolic Expend 125 X OMS 6 Press-Fed Hypergolic Reus 125 X RL10A3 17 Split Expand* LH2/LOX Expend 480 X RL10C1 23 Split Expand* LH2/LOX Expend 633 X RD180 933 Stg Combust (1) RP/LOX Expend 3,870 SSME 470 Stg Combust (2) LH2/LOX Reus 3,300 *Split Expander = use Gas Generator

Victory Solutions MIPSS Team 33 PCM Liquid Engines Key Variables

• Key Variables – Technical Characteristics • Vacuum Thrust (Klbf), Chamber Pressure – “K1” = Engine Cycle + Propellants • Engine Cycle: Gas Generator, Staged Combustion (1, 2), Pressure Fed • Propellants: Fuel (RP, LH2, A-50) + Oxidizer (LO2, N2O4) – Subjective Variables • Manufacturing Maturity, Design Maturity, Certification Approach, etc. • CER Example: Flight Unit Unadjusted TFU = K1 Factor x 0.2455 (92$) x Thrust0.54 x Pc Factor x Mfg Factor Where… • K1 Factor = from lookup table; f(Cycle, Propellant) • Pc Factor = multi-order polynomial: f(Pc, K1) • Manufacturing Factor = lookup table; Mfg Maturity Level, Mfg Automation Level

Adjusted TFU = Unadjusted TFU x Learning Factor x Rate Factor • Alternative: Unadjusted TFU feeds Architecture Model, learning/rate effects are f(architecture production/flight rate), lot buy factors, and phasing, etc. Victory Solutions MIPSS Team 34 Next Steps & Summary

• PCEC Development – Continue PCEC tool development to incorporate new CERs, create linkages to PCM and NICM VIII, plus other minor updates – Next release including these updates will likely be by the end of the CY – Enhance training & communication efforts • Robotic SC – Continue new Mission Normalization & Analysis – Update PCEC CERs adding recently normalized missions – Continued exploring ways to use milestone dataset and CERs to provide better insights on PCEC prediction capability • CASTS – Finalize PCM (liquid engines) version and further development – Functional Breakdown for normal CASTS CERs – Architecture modeling

Victory Solutions MIPSS Team 35 Closing

• Questions? • Demo: Thursday, 16 Aug from 11-12 in Building 3 Auditorium

PCEC Email Contact: [email protected] Application Website(s): ONCE (NASA Civil Servants) https://software.nasa.gov/ , search for PCEC

Victory Solutions MIPSS Team 36 BACKUP

37 PCEC Entry System Model Approach

GOAL: Derive CERs to capture the cost of various atmospheric entry hardware systems for robotic spacecraft missions (Phases B-D)

New Input Candidates Different Input Combinations; PCEC Data Collection of Additional Info; Inputs Tech Combination Inputs Inputs

Body Inputs Candidate CERs

Preliminary Inputs Thermal Protection Systems Parachute Systems Airbag Systems Multiple Regression

Other No Data CER Analysis Acceptable CER Methods Performance PCA Performance? Key Driver Analysis Input Sets Attempt to Identify Yes Supports Common Attributes Constructive to Explain Processes Regression Based CERs Error/Residuals PCEC CERs

Victory Solutions MIPSS Team 38 PCEC Models Principle Component Analysis 1) A correlation matrix was generated to get a sense of the of the dependency between variables. • Several of the variables appeared to be correlated, making PCA an attractive method to apply to the data set. 2) The principal components were determined using an algorithm developed in Python. • The first 6 principal components which account for 85% of variance in the data set were selected and used to determine which of the 20 variables were most likely related to cost. 3) For each of the 21 data sets examined, 4 subsets of the 20 variables were run through a multiple regression routine to determine the new cost estimating relationships. Victory Solutions MIPSS Team 39 PCEC CER Development Process

PCA Results

Minimal Model Run Regression

Victory Solutions MIPSS Team 40