PCEC v2.3 (For Real This Time) 2021 NASA Cost & Schedule Symposium 14 Apr 2021 Brian Alford Mark Jacobs Booz Allen Hamilton TGS Consultants Shawn Hayes Richard Webb TGS Consultants KAR Enterprises NASA MSFC Victory MIPSSSolutions Team SB Diversity Outline • PCEC Overview • Robotic SC Updates • CASTS Updates • PCEC v2.3 Interface Updates • Closing Victory Solutions MIPSS Team 2 What is PCEC? • The Project Cost Estimating Capability (PCEC) is the primary NASA in-house developed parametric tool for estimating the cost of robotic missions, launch vehicles, crewed vehicles, etc. – Overarching tool for creating an estimate that spans the full NASA WBS – CERs included out-of-the-box for estimating the costs of a flight system (e.g., thermal) and support functions (e.g., project management) – Connects to other NASA-sponsored specialized tools to cover the complete NASA WBS (e.g., NICM, MOCET) – Excel-based (presented as add-in in the Ribbon) with completely visible calculations and code – Consists of the PCEC Interface (the Ribbon and supporting code) and the PCEC Library (the artifacts used to estimate cost) – Available to the General Public Victory Solutions MIPSS Team 3 What is PCEC? Cont’d • PCEC comprises two primary ‘models’, offered seamlessly to the user under a single, integrated tool – Robotic Spacecraft (Robotic SC) – Crewed and Space Transportation Systems (CASTS) • These models have separate data normalizations, collections of CERs, core WBSs, modeling approaches, estimating template worksheets, and scope – Estimating artifacts constitute the PCEC Library embedded within the Interface – Normalizations and CER workbooks are stored in the REDSTAR Library and ONCE database for NASA users • Interface provides visibility into the CERs, CER statistics, variable definitions and values, and other tools useful in building estimates Victory Solutions MIPSS Team 4 What’s the latest status of PCEC? • PCEC v2.3 was released last week (For real, this time!) – E-mail announcement went out to latest user list – Available on both the ONCE Database (for both CS and SC ONCE users) and the NASA Software Catalog (for all others) – Software Catalog Users: Go to https://software.nasa.gov/app/ • Currently ~850+ ‘Downloaders’ spread across 49 Countries Victory Solutions MIPSS Team 5 PCEC v2.3 ROBOTIC SC UPDATES 6 What’s New for the Robotic Spacecraft Model? • New Missions • Expanded set of CERs • New Mission Elements • Updated CER development and validation process • Model Performance Summary • Future Plans Victory Solutions MIPSS Team 7 PCEC Current Mission Set Missions for PCEC v2.2.1 • Seven additional missions have been added to the PCEC v2.2.1 data set, bringing the normalized data set to a total of 49 missions Additional Missions for PCEC v2.3 • Seven new candidate missions are currently in development: Victory Solutions MIPSS Team 8 New Entry Hardware CERs MSL • PCEC v2.3 includes CERs for Thermal Protection Systems (TPS) and Parachutes – Parachute (6) • Includes cost of parachutes, lines and mortar • Cost drivers include: parachute mass and diameter, number of units – TPS (5) • Includes ablative material • Cost drivers include: peak deceleration, mean surface pressure, TPS mass and entry mass Victory Solutions MIPSS Team 9 New Mission Elements • In order to better estimate the cost of complex, multi-element spacecraft such Estimate A: as MSL, the PCEC team experimented Cruise Stage (1) MSL with several CER development approaches without much success – The small data set worked against Estimate B: development of viable CERs EDL System (2),(3),(5),(6) • Ultimately, these multi-element spacecraft were split into their respective parts in the PCEC dataset that was used to develop the v2.3 CERs Estimate C: Rover (4) • PCEC Best Practice: Complex multi- element systems such as MSL, Deep Impact, Insight, etc. should be estimated as separate elements Victory Solutions MIPSS Team 10 CER Development Process Overview • A complete overhaul of all of the PCEC robotic spacecraft CERs has been completed and is included in the v2.3 release of the model. • These new CERs reflect the significant amount of additional mission data that has been made available through CADRe/ONCE – Meets original charter goal of continual model improvement in order to capture changes in the NASA portfolio (e.g. focus on smaller missions) • An updated 6 step process for CER development has been implemented in the creation of the v2.3 robotic spacecraft model DATA NORMALIZE OUTLIER PRINCIPLE REGRESSION EXPERT MINING DATA REMOVAL COMPONENT REVIEW & ANALYSIS ITERATION Victory Solutions MIPSS Team 11 CER Development (1 of 3) • Ongoing mission data collection effort since FY 2015 • Main sources of data used to support CER development DATA MINING include: – LRD CADRes and their supporting documentation – REDSTAR library resources at MSFC • Incomplete contractor data at the spacecraft subsystem level has recently become problematic for newly launched missions • To date, 49 missions (59 flight elements) have been normalized using an 8-step process which aligns each NORMALIZE mission’s cost data using the NASA standard WBS as a DATA unifying framework – Adjustments for inflation, fees/burdens, contributions, etc. • Each mission’s normalization workbook can be found in the REDSTAR library along with supporting documentation Victory Solutions MIPSS Team 12 CER Development (2 of 3) • Mission outliers were identified using box plot analysis for each CER OUTLIER • Impact of outlier removal on model performance was tested REMOVAL and shown to be necessary for the greater good • Of the 13 outliers identified, three missions proved to be outliers in nearly every CER category – Cassini, GOES-R and the MSL Rover • No single unifying element could be found to link outlier missions together given the variation in mission destination and purpose • PCA is mathematical process that transforms a data set into a smaller one that still contains most of the information PRINCIPLE contained in the larger set. COMPONENT • PCA effectively reduces the number of variables while ANALYSIS preserving as much information as possible Victory Solutions MIPSS Team 13 CER Development (3 of 3) • The PCEC regression process was implemented using a Python routine that includes log transformation of the data REGRESSION and uses a standard backward stepwise approach • Variables were limited to no more than 10% of the number of observations – Most CERs have less than four independent variables • Model selection criteria shifted from adjusted R2 to error minimization using Root Mean of the Square (RMSE) metric • Although mathematical methods can produce an array of CERs with low error that appear reasonable, care must be EXPERT taken to sanity check the results REVIEW & ITERATION • Model input parameters must make intuitive sense • PCEC preliminary CERs were reviewed by subject matter experts and iterated until mathematically viable and intuitive results were obtained for each CER Victory Solutions MIPSS Team 14 Leave One Out Cross Validation (LOOCV) Process • K-Fold cross validation testing allows for an estimate of how accurately a predictive model (i.e. CER) will perform in practice. • Leave One Out Cross Validation (LOOCV) is a subset of K-Fold cross validation with K = n (number of observations in the data set) • LOOCV is the best cross validation approach when working with small data sets. • To perform the validation, the total data set is divided into n subsets (1 observation each) • In each of the k iterations, a single observation is retained as test data while the remaining data is used to train the model. This process is repeated K = n times. • Root Mean Square of the Error (RMSE) was the primary metric used for assessing model performance (Adjusted R^2, coefficient metrics, Mean Absolute Deviation (M.A.D.) and Mean Square of the Error (M.S.E.) were also computed). Victory Solutions MIPSS Team 15 Leave One Out Cross Validation (LOOCV) Results • Cross validation results for each of the PCEC CERs is shown • Comparison of the average RMSE to the PCEC v2.3 RMSE gives confidence that the model is robust and is not overly influenced by any one data point. Victory Solutions MIPSS Team 16 Overall PCEC v2.3 Performance Robotic SC • Significant reduction in CER estimating error from PCEC v2.2.1 to v2.3 • Overall model performance calculation includes: mission support functions (PM/MA/SE/I&T), spacecraft, pre-launch MOS/GDS and Phase E (MO&DA) – Essentially all of Phases B-E (not including payload) • Error distribution was roughly centered around zero indicating an unbiased model • 70% of missions, including outliers, were estimated within +/-30% of actuals • 96% of missions, including outliers, were estimated within +/- 50% of actuals Victory Solutions MIPSS Team 17 Other Notable Changes from v2.2.1 to v2.3 • Spacecraft Communications Subsystem CER – v2.3 is now a singular CER and no longer requires selection based on amplifier type (SSPA vs. TWTA) or destination (Earth vs. planetary) • Mission Support Function (PM/MA/SE/I&T) CERs – v2.3 includes consolidated CERs for support functions at the total level (WBS 1/2/3/10) which can then be allocated to WBS 5/6 within the model • Pre-Launch MOS/GDS CER – v2.3 includes consolidated CER • Phase E, Mission Operations & Data Analysis (MO&DA) CER – v2.3 is now a singular CER and no longer requires selection based on destination or phase of operations (cruise, encounter, etc.) Victory Solutions MIPSS Team 18 Future Plans Robotic SC • Additional mission candidates • Decision