PCEC Updates for 2018 2018 NASA Cost & Schedule Symposium August 15, 2018 Brian Alford Mark Jacobs Booz Allen Hamilton TGS Consultants Shawn Hayes Richard Webb TGS Consultants KAR Enterprises NASA MSFC Victory MIPSSSolutions Team SB Diversity Overview • PCEC Overview • Robotic Spacecraft Updates • Crewed & Space Transportation System Update • Summary Victory Solutions MIPSS Team 2 What is PCEC? • The Project Cost Estimating Capability (PCEC) is the primary NASA-sponsored parametric cost tool for spacecraft estimates – Developed and maintained by NASA at MSFC beginning in late 2013 – Excel Add-in that provides capabilities and cost estimating artifacts used to build a spacecraft cost estimate in Excel – Based on more than 70 missions/system elements, but with separate approaches for modeling different types of systems • Robotic Spacecraft (Robotic SC) • Crewed & Space Transportation Systems (CASTS) – Completely transparent tool: no code passwords, protected sheets, etc. – Available to the general public via ONCE and the NASA Software Catalog (https://software.nasa.gov/) Victory Solutions MIPSS Team 3 Current State of PCEC • PCEC v2.2.1 was released in May 2018 – Fixes minor bugs in some templates and a minor code revision – No updates to the Library other than latest New Start Inflation indices – Update provided via Excel updater file; includes routines for making corrections to user-installed Interface and existing v2.2 estimates • Current user counts as of late July 2018 – 490+ Users / Requestors – 45+ Countries represented – ~60 users added since last year’s Symposium Victory Solutions MIPSS Team 4 PCEC User Distribution Estimated based on user-provided data Victory Solutions MIPSS Team 5 PCEC ROBOTIC SPACECRAFT UPDATES 6 PCEC Robotic Spacecraft Topics 1. New Missions 2. Entry System CERs 3. Milestone-Specific Data Analyses Victory Solutions MIPSS Team 7 Current Robotic SC Missions & New Candidates • Missions 1-42 are used for PCEC v2.2.1 CERs • Additional Candidates 1-6 have been tested with PCEC v2.2.1; Overall results are generally good • Data normalization is complete for Additional Candidates 1-4 and nearly complete for 5-6 • Additional Candidates 7-11 should have launch CADRe data available soon (before the end of FY19) • PCEC CERs could be updated in FY19 incorporating at least the first 6 additional candidates Victory Solutions MIPSS Team 8 PCEC v2.2.1 Results for New Candidates Actual $)/Actual $ $)/Actual Actual – (Estimate • Results compare the PCEC estimate to unadjusted Launch CADRe cost data • PCEC Phase BCD estimates are within 20% for 4 missions (average error is -19%) • Several of these missions experienced ‘External Impacts’ that will be normalized out of the data used for the PCEC CERs • External Impacts are the primary driver for the bias towards low estimates for 3 of these missions (GPM, MMS, and GOES-R) Victory Solutions MIPSS Team 9 PCEC Entry System CERs NASA WBS Elements Included GOAL: Derive CERs to capture the cost of various atmospheric entry hardware systems for robotic spacecraft missions (Phases B-D) • The PCEC Entry System elements are included in the spacecraft (SC) portion of the WBS, but separate from the more typical subsystems. – Thermal Protections System (TPS) is WBS 6.5.1.8 – Parachute System is WBS 6.5.1.9 – AirbaG System is WBS 6.5.1.10 Victory Solutions MIPSS Team 10 Entry System CERs Approach & Mission Set • Costs for Thermal Protection Systems (TPS), Parachutes, and Air Bag Entry System elements are not included in PCEC v2.2.1 • Costs for these items have Been derived using data from 9 missions • PCA was used to help identify technical characteristics driving costs • CERs have Been developed for Non-recurring, Recurring, and Phase BCD Total costs Victory Solutions MIPSS Team 11 Entry System CERs Thermal Protection Systems (TPS), 1 of 2 • Definition: Covers design, fabrication, and I&T for all TPS material; Includes TPS on Heat Shield, Backshell, and Parachute; Does not include structure, adhesive, or other supporting elements TPS Non-Recurring Cost (NRC) TPS Recurring Cost (RC) • Input Parameters: • Input Parameters: – PD - Peak Deceleration (g) – PD - Peak Deceleration (g) – MPS - Mean Pressure at the Surface (kPa) – HD – Heatshield Diameter (m) • Cost Estimating Relationship (CER): – HPU – Heatshield Production Units 1236 × (!")0.3761 × (#!$)−0.1847 • Cost Estimating Relationship (CER): • Adjusted R^2: 0.52 1437 × (%!&) × (!")0.2625 × (%")0.6958 • Range of Error: -51 to 73% • Adjusted R^2: 0.70 Victory Solutions MIPSS Team • Range of Error: -32 to 77% 12 Entry System CERs Thermal Protection Systems (TPS), 2 of 2 • Definition: Covers design, fabrication, and I&T for TPS material; Does not include structure, adhesive, or other supporting elements TPS – Alternate CER (NRC+RC) • Input Parameters: – PD - Peak Deceleration (g) – SEM - System Entry Mass (kg) • Cost Estimating Relationship (CER): 230 × (!")0.4459 × (#$%)0.4045 • Adjusted R^2: 0.93 • Range of Error: -26 to 26% Victory Solutions MIPSS Team 13 Entry System CERs Parachutes, 1 of 2 • Definition: Covers design, Fabrication, and I&T For the Parachute System; Includes all Flight parachutes, lines, and mortar; Does not include support structure Parachutes Non-Recurring Cost (NRC) Parachutes Recurring Cost (RC) • Input Parameters: • Input Parameters: – MPD: Main Parachute Diameter (m) – PSM - Parachute System Mass (Chutes, • Cost Estimating Relationship (CER): Mortar) (kg) 8.5 × (!"#)1.9541 – MPS - Mean Pressure at the SurFace (kPa) – PPU – Parachute Production Units • Adjusted R^2: 0.50 • Cost Estimating Relationship (CER): • Range of Error: -51 to 119% 564 × (""$) × ("%!)0.4983 × (!"%)−0.0880 • Adjusted R^2: 0.83 Victory Solutions MIPSS Team 14 • Range of Error: -36 to 32% Entry System CERs Parachutes, 2 of 2 • Definition: Covers design, fabrication, and I&T for the Parachute System; Includes all flight parachutes, lines, and mortar; Does not include support structure Parachutes – Alternate CER (NRC+RC) • Input Parameters: – MPD: Main Parachute Diameter (m) – PPU – Parachute Production Units • Cost Estimating Relationship (CER): 37 × (!"#)1.7903 × (""$)0.91 • Adjusted R^2: 0.87 • Range of Error: -42 to 44% Victory Solutions MIPSS Team 15 Entry System CERs Air Bag Deceleration Systems • Definition: Covers design, fabrication, and I&T for an Air Bag Deceleration System; Includes all air bags, and inflation system; Does not include support structure Air Bag Deceleration – CER (NRC+RC) Development Cost vs Air Bag System Mass • Input Parameters: – ASM: Airbag System Mass (kg) – APU – Airbag Production Units • Cost Estimating Relationship (CER): 6338 + (72.459 × !"# × !$%) • AdjusteD R^2: 1 • Range of Error: -15 to 3% Victory Solutions MIPSS Team 16 Milestone-Specific Analyses Background • Question: Can we correlate cost model inputs known at the start of Phase B to Actual costs (at launch)? • Cost models are typically developed by correlating technical/programmatic inputs at launch to actual costs • If CERs were developed using Phase B inputs and Actual costs, would they perform better than typical cost models at the start of Phase B? GOAL: Improve understanding of how inputs known at the start of Phase B correlate to actual costs at launch; Approach utilizes the same cost data normalization process applied to PCEC Robotic Missions and explores correlations (new CERs) and relationships between changes in cost model inputs and costs (Project & PCEC estimates) Victory Solutions MIPSS Team 17 Milestone-Specific Analyses Phase B Cost Data Normalization • Cost data representing the start of Phase B has been normalized for 24 projects (same process as used for PCEC v2.2) • Most of these projects are AO missions (and CSR data was used) • Difficult to clearly identify/collect Phase B start data for many projects Victory Solutions MIPSS Team 18 Milestone-Specific Analyses CER Results Overview Case 1 CER Results: • Use the exact same CER inputs by subsystem as the v2.2.1 CERs • Error range at the Flight System Level – CBE: -72% to 66% CBE + Contingency: -73% to 67% • CERs are oversubscribed since the number of data points used to develop the new CERs are significantly less than the PCEC v2.2 CERs (~20 vs ~40 data points) Case 2 CER Results: • Use the same CER discovery process employed during the development of the PCEC v2.2.1 CERs; different inputs allowed • Error range at the Flight System Level – CBE: -59% to 36% CBE + Contingency: -59% to 39% • Some of the inputs are counter-intuitive; These CERs should be refined before considering release Victory Solutions MIPSS Team 19 Milestone-Specific Analyses Results Summary Based on Development (BCD) $ w/o Reserves, LV, Instruments, GDS, or Science Team • Results using the ‘New Phase B CERs’ appear a little better than using PCEC v2.2 for the 24 missions used in this analysis • ‘Robustness’ of the New Phase B CERs may be questionable based on results using missions outside this 24-mission data set Victory Solutions MIPSS Team 20 Milestone-Specific Analyses Interesting CER Variable Observations • Mass remains important across all CER cases • Heritage/Parts Rating variables become more important and schedule variables become less important when comparing Case 2 CERs to the original PCEC v2.2 CERs Victory Solutions MIPSS Team 21 Milestone-Specific Analyses Cost Growth Correlations to Input Changes Schedule Growth Mass Growth Change in Heritage • Correlations between cost growth from Phase B and changes in technical and programmatic inputs were explored; however, good correlations between changes