PCEC v2.3 (For Real This Time)
2021 NASA Cost & Schedule Symposium 14 Apr 2021
Brian Alford Mark Jacobs Booz Allen Hamilton TGS Consultants Shawn Hayes Richard Webb TGS Consultants KAR Enterprises
NASA MSFC
Victory MIPSSSolutions Team SB Diversity Outline
• PCEC Overview • Robotic SC Updates • CASTS Updates • PCEC v2.3 Interface Updates • Closing
Victory Solutions MIPSS Team 2 What is PCEC?
• The Project Cost Estimating Capability (PCEC) is the primary NASA in-house developed parametric tool for estimating the cost of robotic missions, launch vehicles, crewed vehicles, etc. – Overarching tool for creating an estimate that spans the full NASA WBS – CERs included out-of-the-box for estimating the costs of a flight system (e.g., thermal) and support functions (e.g., project management) – Connects to other NASA-sponsored specialized tools to cover the complete NASA WBS (e.g., NICM, MOCET) – Excel-based (presented as add-in in the Ribbon) with completely visible calculations and code – Consists of the PCEC Interface (the Ribbon and supporting code) and the PCEC Library (the artifacts used to estimate cost) – Available to the General Public Victory Solutions MIPSS Team 3 What is PCEC? Cont’d
• PCEC comprises two primary ‘models’, offered seamlessly to the user under a single, integrated tool – Robotic Spacecraft (Robotic SC) – Crewed and Space Transportation Systems (CASTS) • These models have separate data normalizations, collections of CERs, core WBSs, modeling approaches, estimating template worksheets, and scope – Estimating artifacts constitute the PCEC Library embedded within the Interface – Normalizations and CER workbooks are stored in the REDSTAR Library and ONCE database for NASA users • Interface provides visibility into the CERs, CER statistics, variable definitions and values, and other tools useful in building estimates
Victory Solutions MIPSS Team 4 What’s the latest status of PCEC?
• PCEC v2.3 was released last week (For real, this time!) – E-mail announcement went out to latest user list – Available on both the ONCE Database (for both CS and SC ONCE users) and the NASA Software Catalog (for all others) – Software Catalog Users: Go to https://software.nasa.gov/app/ • Currently ~850+ ‘Downloaders’ spread across 49 Countries
Victory Solutions MIPSS Team 5 PCEC v2.3 ROBOTIC SC UPDATES
6 What’s New for the Robotic Spacecraft Model?
• New Missions • Expanded set of CERs • New Mission Elements • Updated CER development and validation process • Model Performance Summary • Future Plans
Victory Solutions MIPSS Team 7 PCEC Current Mission Set
Missions for PCEC v2.2.1 • Seven additional missions have been added to the PCEC v2.2.1 data set, bringing the normalized data set to a total of 49 missions
Additional Missions for PCEC v2.3
• Seven new candidate missions are currently in development:
Victory Solutions MIPSS Team 8 New Entry Hardware CERs
MSL • PCEC v2.3 includes CERs for Thermal Protection Systems (TPS) and Parachutes – Parachute (6) • Includes cost of parachutes, lines and mortar • Cost drivers include: parachute mass and diameter, number of units
– TPS (5) • Includes ablative material • Cost drivers include: peak deceleration, mean surface pressure, TPS mass and entry mass
Victory Solutions MIPSS Team 9 New Mission Elements
• In order to better estimate the cost of complex, multi-element spacecraft such Estimate A: as MSL, the PCEC team experimented Cruise Stage (1) MSL with several CER development approaches without much success
– The small data set worked against Estimate B: development of viable CERs EDL System (2),(3),(5),(6) • Ultimately, these multi-element spacecraft were split into their respective parts in the PCEC dataset that was used to develop the v2.3 CERs Estimate C: Rover (4) • PCEC Best Practice: Complex multi- element systems such as MSL, Deep Impact, Insight, etc. should be estimated as separate elements
Victory Solutions MIPSS Team 10 CER Development Process Overview
• A complete overhaul of all of the PCEC robotic spacecraft CERs has been completed and is included in the v2.3 release of the model. • These new CERs reflect the significant amount of additional mission data that has been made available through CADRe/ONCE – Meets original charter goal of continual model improvement in order to capture changes in the NASA portfolio (e.g. focus on smaller missions) • An updated 6 step process for CER development has been implemented in the creation of the v2.3 robotic spacecraft model
DATA NORMALIZE OUTLIER PRINCIPLE REGRESSION EXPERT MINING DATA REMOVAL COMPONENT REVIEW & ANALYSIS ITERATION
Victory Solutions MIPSS Team 11 CER Development (1 of 3)
• Ongoing mission data collection effort since FY 2015 • Main sources of data used to support CER development DATA MINING include: – LRD CADRes and their supporting documentation – REDSTAR library resources at MSFC • Incomplete contractor data at the spacecraft subsystem level has recently become problematic for newly launched missions
• To date, 49 missions (59 flight elements) have been normalized using an 8-step process which aligns each NORMALIZE mission’s cost data using the NASA standard WBS as a DATA unifying framework – Adjustments for inflation, fees/burdens, contributions, etc. • Each mission’s normalization workbook can be found in the REDSTAR library along with supporting documentation
Victory Solutions MIPSS Team 12 CER Development (2 of 3)
• Mission outliers were identified using box plot analysis for each CER OUTLIER • Impact of outlier removal on model performance was tested REMOVAL and shown to be necessary for the greater good • Of the 13 outliers identified, three missions proved to be outliers in nearly every CER category – Cassini, GOES-R and the MSL Rover • No single unifying element could be found to link outlier missions together given the variation in mission destination and purpose
• PCA is mathematical process that transforms a data set into a smaller one that still contains most of the information PRINCIPLE contained in the larger set. COMPONENT • PCA effectively reduces the number of variables while ANALYSIS preserving as much information as possible Victory Solutions MIPSS Team 13 CER Development (3 of 3)
• The PCEC regression process was implemented using a Python routine that includes log transformation of the data REGRESSION and uses a standard backward stepwise approach • Variables were limited to no more than 10% of the number of observations – Most CERs have less than four independent variables • Model selection criteria shifted from adjusted R2 to error minimization using Root Mean of the Square (RMSE) metric • Although mathematical methods can produce an array of CERs with low error that appear reasonable, care must be EXPERT taken to sanity check the results REVIEW & ITERATION • Model input parameters must make intuitive sense • PCEC preliminary CERs were reviewed by subject matter experts and iterated until mathematically viable and intuitive results were obtained for each CER
Victory Solutions MIPSS Team 14 Leave One Out Cross Validation (LOOCV) Process • K-Fold cross validation testing allows for an estimate of how accurately a predictive model (i.e. CER) will perform in practice. • Leave One Out Cross Validation (LOOCV) is a subset of K-Fold cross validation with K = n (number of observations in the data set) • LOOCV is the best cross validation approach when working with small data sets. • To perform the validation, the total data set is divided into n subsets (1 observation each) • In each of the k iterations, a single observation is retained as test data while the remaining data is used to train the model. This process is repeated K = n times. • Root Mean Square of the Error (RMSE) was the primary metric used for assessing model performance (Adjusted R^2, coefficient metrics, Mean Absolute Deviation (M.A.D.) and Mean Square of the Error (M.S.E.) were also computed).
Victory Solutions MIPSS Team 15 Leave One Out Cross Validation (LOOCV) Results
• Cross validation results for each of the PCEC CERs is shown • Comparison of the average RMSE to the PCEC v2.3 RMSE gives confidence that the model is robust and is not overly influenced by any one data point.
Victory Solutions MIPSS Team 16 Overall PCEC v2.3 Performance Robotic SC
• Significant reduction in CER estimating error from PCEC v2.2.1 to v2.3 • Overall model performance calculation includes: mission support functions (PM/MA/SE/I&T), spacecraft, pre-launch MOS/GDS and Phase E (MO&DA) – Essentially all of Phases B-E (not including payload) • Error distribution was roughly centered around zero indicating an unbiased model • 70% of missions, including outliers, were estimated within +/-30% of actuals • 96% of missions, including outliers, were estimated within +/- 50% of actuals
Victory Solutions MIPSS Team 17 Other Notable Changes from v2.2.1 to v2.3
• Spacecraft Communications Subsystem CER – v2.3 is now a singular CER and no longer requires selection based on amplifier type (SSPA vs. TWTA) or destination (Earth vs. planetary) • Mission Support Function (PM/MA/SE/I&T) CERs – v2.3 includes consolidated CERs for support functions at the total level (WBS 1/2/3/10) which can then be allocated to WBS 5/6 within the model • Pre-Launch MOS/GDS CER – v2.3 includes consolidated CER • Phase E, Mission Operations & Data Analysis (MO&DA) CER – v2.3 is now a singular CER and no longer requires selection based on destination or phase of operations (cruise, encounter, etc.)
Victory Solutions MIPSS Team 18 Future Plans Robotic SC
• Additional mission candidates • Decision Tree Analysis – Categorizes missions based on subsystem characteristics – Enables better understanding of cost drivers for different mission types – Incorporates non-linear behavior of independent variables • v2.3 Tuning Options – Tuning options include adjustments that can be made to the PCEC CERs to improve performance for specific mission types – PCEC v2.2 Tuning Inputs included items that were not used in the CERs – PCEC v2.3 CER development more fully integrated all input candidates, making the tuning process more difficult – Early results from the CART Analysis suggests tuning may be beneficial for larger missions
Victory Solutions MIPSS Team 19 PCEC v2.3 CASTS UPDATES
20 CASTS Summary
• CASTS Changes in v2.3 – Inflation Indices – Avionics – Liquid Rocket Engines – Solid Motors
• Propulsion Cost Model – Liquid Rocket Engines – Nuclear Thermal Propulsion – Solid Rocket Motors
• Next Steps – Additions to CASTS historical database – Changes to Solid Motors CER – Modifications to other CERs Victory Solutions MIPSS Team 21 CASTS Changes in v2.3 Inflation
• Inflation Indices – Incorporated updated NASA New Start inflation index released by NASA HQ • Index updates include changes to historical factors related to past years – Applied new index to historical cost data used to generate CASTS CERs – Results in minor changes to essentially all CASTS CERs due to changes in escalated values of historical cost data used to generate CERs (<5%) • Even if the historical dataset did not change
• Net CER changes – If no changes in historical data set – CER slopes remain same or change slightly – Constant values typically 3% to 5% lower – Overall Results approximately 3% to 5% lower than v2.2 Victory Solutions MIPSS Team 22 CASTS Changes in v2.3 Avionics
• Modified independent variables to increase CERs predictive power – Was: Avionics Rating - constant that differentiates between three of the four Avionics WBS Elements (Command Control and Data Handling, Guidance Navigation and Control, and Telemetry and Tracking ). Does not apply to Range Safety – Is: Avionics Type - indicator variable that differentiates between the three Avionics WBS Elements (Command Control and Data Handling, Guidance Navigation and Control, and Telemetry and Tracking). The variable input value for CCDH and GNC is the same.
– Was: Launch Vehicle Type - reflects the overall program approach within which the system is to be operated. The LV Type is determined based on a 2 x 2 matrix: Manned or Unmanned and Expendable or Reusable. – Is: Reusability - indicator variable to identify avionics incorporated as part of a reusable launch system.
Victory Solutions MIPSS Team 23 CASTS Changes in v2.3 Liquid Rocket Engines
• The historical cost database expanded with new engine cost data relative to version 2.2, resulting in substantial modifications to the CERs. – Fuel types: added hypergolics – Cycle types: added pressure fed; differentiated between single and dual-shaft staged combustion Added to CASTS Reus Launch Thrust Data Set Engine Cycle Propellants /Expend System(s) (klbf vac) Pc F1 Gas Generator RP/LOX Expend Saturn V 1,748 982 X MA5 Gas Generator RP/LOX Expend Atlas II 463 711 RS27 Gas Generator RP/LOX Expend Delta II 232 700 J2 Gas Generator LH2/LOX Expend Saturn II 230 763 J2X Gas Generator LH2/LOX Expend not apply 294 1,340 RL10A3 Split Expander* LH2/LOX Expend Multiple 17 480 RS68 Gas Generator LH2/LOX Expend Delta IV 745 1,488 X LR87 Gas Generator Hypergolic Expend Titan IV 269 857 X LR91 Gas Generator Hypergolic Expend Titan IV 103 860 X Viking VI Gas Generator Hypergolic Expend Ariane 4, 5 168 800 SSME Stg Combustion (2 shaft) LH2/LOX Reus Shuttle 470 3,300 RD180 Stg Combustion (1 shaft) RP/LOX Expend Atlas V 933 3,870 X LM Ascent Pressure-Fed Hypergolic Expend Lunar Module 4 100 X LM Descent Pressure-Fed Hypergolic Expend Lunar Module 10 100 X OMS Pressure-Fed Hypergolic Reus Shuttle 6 125 X RL10C1 Split Expander* LH2/LOX Expend Multiple 23 480 *Split Expander = use Gas Generator Victory Solutions MIPSS Team 24 CASTS Changes in v2.3 Solid Rocket Motors • Modified Flight Unit historical cost database used to generate CERs to include Ares I First Stage Booster and Titan IV SRM (steel case- CSD) – No modifications to DDTE cost database – Removed Trident ballistic missile and Pegasus multi-stage system from Flight Unit CER data set • Added strategic ballistic missile data to database, but did not include in CER development – CERs cost data sets include launch vehicle boosters only Trident D5 Peacekeeper PK Stg 1 (MTI SR-118) Peacekeeper PK Stg s (AGC SR-119) Peacekeeper PK Stg 3 (HER SR-120) Minuteman MMI, II, III Stg 1: M55E1 Minuteman MMI Stg 2: M56A1 Minuteman MMI, II Stg 3: M57A1 Minuteman MMII, III Stg 2: SR19AJ1 Minuteman MMII, III Stg 3: SR73AJ1
Victory Solutions MIPSS Team 25 Propulsion Cost Model (PCM) Overview
• What is PCM? – Add-in for CASTS • Standalone model to PCEC/CASTS • Linkable to PCEC estimate similar to NICM, MOCET • PCM Capability – Liquid Rocket Engines, Solid Rocket Motors, Nuclear Thermal Propulsion, . . . – Non Recurring DDTE + First Unit + Production $’s • PCM Availability – Similar approach to PCEC/CASTS – Publicly released model (spreadsheet) – Unrestricted and Restricted documentation • Manual plus (restricted) source database/calibrations
Victory Solutions MIPSS Team 26 PCM Status
• Liquid Engines – Completed – final model testing within MSFC ECO – Calibrated to historical database – Documentation undergoing final edits • Nuclear Thermal – CERs completed – initial prototype model undergoing testing – Documentation in-work CERs Complete; Documentation • Solid Motors (User Guides) in work – In-work: Add kick stages, upper stages, sounding rockets, and ballistic missile stages to existing launch vehicle boosters CASTS database
Propulsion Cost Model
Liquid Solid Nuclear Engines Motors Themal Other? Engine Cycle Monolithic Thermionic Hypersonic Thrust Segmented Thermoelec RBCC Propellants Small Cycle Ion Test Approach Total Impulse Thermal Ctl Solar Sail Victory Solutions MIPSS Team 27 CASTS Next Steps
• Additions to CASTS historical database – International Space Station (ISS), Inertial Upper Stage (IUS), X33, others – (Possibly) selected SLS and Orion subsystems
• Modify CERs to reflect additional data points – Emphasis on Avionics, Crew Structures, Crew Systems
• Changes to Solid Motors CERs – Expand historical database to include/differentiate between ballistic missiles, kick motors, stages, sounding rockets, boosters • Based on PCM Solid Motors CER development – Change independent variable from Total Impulse to Gross Weight
• Others??
Victory Solutions MIPSS Team 28 PCEC v2.3 INTERFACE UPDATES
29 PCEC v2.3 Interface What are the major changes?
• Layout Changes: CERs and Worksheet Templates • ‘Launch an Estimate’: New Estimate Paths • ‘Launch an Estimate’: Userform Editing Improvements • Global Inputs & Subsystem Inputs Updates • New Model Linkages • Other Changes
Victory Solutions MIPSS Team 30 Layout Changes CERs & Estimating Templates
Outputs and Inputs Transposed
Estimating Templates Hyperlinks for • Worksheets to estimateNavigation line items in the WBS • Contain inputs, CER(s), and supporting calculations to translate inputs into an estimate for the item (development, production, etc.) • Similar look across the tool but are customized by generalMajor CER CollectionSections Go L-R Across Sheet
CER Inputs Go Down Sheet
CERs arranged ‘Vertically’
Victory Solutions MIPSS Team 31 Launch an Estimate Paths
‘Launch an Estimate’ • Primary method for building a complete estimate in PCEC • Two paths: Launch New Estimate or Resume Existing Estimate • New: Pick one of the built-in WBSs and customize • Resume: Use an existing estimate built via Launch and make changes
Launch New Now Has Two Sub-Paths
– Start from Pre-Built Workbook: Opens pre-constructed workbook stored w/the Interface and makes changes to it – Start from Blank Workbook: Starts with blank workbook; all individual items’ worksheet templates added after ‘Go’
Victory Solutions MIPSS Team 32 Launch WBS Editing
Improved Filtering ‘Launch’ Userform • Dialog that gives the user the ability to customize the starting WBSs WBS• EditingThe user can search for CERs/subsystems available, Improvementsinsert/delete/reorganize/rename line items in the WBS, and link to estimates from other models
New Options for Easy Access to CERs Building Estimate Reorganization
Victory Solutions MIPSS Team 33 Global Inputs Updates
Global Inputs Overall • Single location for entry of global-level variablesReorganization used across many subsystem estimates • Controls the ‘State’ of the model • Calculation Mode: Baseline estimate or uncertainty-adjusted estimate (CER and/or Input Uncertainty) • Active Case: Point estimate or three alternate sensitivity cases
Input Entry
Different Set of Global Variables
Victory Solutions MIPSS Team 34 Subsystem Inputs Updates
Subsystem Inputs • Single location for inputting dataReorganization on all subsystems of in the estimate file • Mass Inputs Sections Updated Mass • CER Independent Variables Calculations • Production Inputs • Inputs are linked to the associated estimating templates for each
CER Uncertainty Distribution Inputs Hyperlinks for Navigation
Line Item Naming Consistencies
Victory Solutions MIPSS Team 35 New Model Linkages
• NICM (9, 8.5, and 8): Updates to allow linking to the latest version as well as backwards compatibility • PCM (v1.0 LRE): New ability to link to output from PCM v1.0, which will be released on ONCE soon (broader release will require export control review)
Victory Solutions MIPSS Team 36 Other Notable Changes
• S-Curve Template • Backend changes allow backwards compatibility – You can edit (Resume) existing estimates created in v2.2 using the v2.3 Interface • ‘Zero-out’ Line Item available in Launch • Hyperlinks on WBS worksheet • No ‘Library swapping’ but multiple CER options – Both v2.3 and v2.2 CERs are present for use – First Pound Cost remain; V1 Legacy retired • All userforms updated to improve look/feel/usability • Calculation Errors Fixed – SOCM Score worksheet – Mass calculations when QNHA > 1 – Production calculations whenever start unit > 1
Victory Solutions MIPSS Team 37 PCEC v2.3 CLOSING
38 Summary & Next Steps
• PCEC v2.3 has finally been released, with two download options for existing and new users – Multiple improvements across the board in CERs, look and feel of the estimate workbooks, navigation improvements, etc. • The team recommends starting new estimates using v2.3 to take advantage of the new CERs and improved navigation capabilities • Other Near Term Steps – Conduct a more detailed Roll-out / demo / training session – Finalize updated historical documentation and CER workbooks; provide access to approved users on ONCE & REDSTAR – Release PCM-LRE and work other modules (NTP & Solids) – Develop the PCEC Training Course as part of PP&C curriculum
PCEC Email Contact: [email protected] Application Website(s): ONCE Database NASA Software Catalog (https://software.nasa.gov/)
Victory Solutions MIPSS Team 39 REDSTAR Updates
• The REDSTAR Library is… – A physical and digital collection of historical programmatic documents for spaceflight and aeronautics programs and projects – Accessible to NASA Civil Servants and non-Prime NASA Support Contractors • Apply for access by searching for REDSTAR in NAMS • We have a new Librarian (as of Jan 2020) – Melissa Roberts ([email protected]) – Contact her for all your REDSTAR research requests • REDSTAR’s Moving! – Decommission of Building 4200 on MSFC has necessitated that we find a new home for REDSTAR – New location on MSFC being finalized; likely to be co-located with new MSFC Technical Library – No changes to the digital library at this time (in Sharepoint)
Victory Solutions MIPSS Team 40 PCEC v2.3 BACKUP
41 Contact Information
Brian Alford Shawn Hayes Booz Allen Hamilton TGS Consultants, LLC 256-544-3737 630-797-5018 [email protected] [email protected]
Mark Jacobs Richard Webb TGS Consultants, LLC KAR Enterprises 813-839-5476 661-547-7632 [email protected] [email protected]
Victory Solutions MIPSS Team 42 PCEC Downloads – World As of April 2021
Victory Solutions MIPSS Team 43 PCEC Downloads - US As of April 2021
Victory Solutions MIPSS Team 44