Mission Operations Cost Estimation Tool (MOCET) 2019 Updates Marc Hayhurst, Elliott Tibor, Brian Wood The Aerospace Corporation Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo Rodriguez NASA Langley Research Center Science Office for Mission Assessments (SOMA) August 14, 2019 Approved for public release OTR 2019-01048. © 2019 The Aerospace Corporation Aerospace acknowledges NASA’s sponsorship under Contract NNL11AA01B Task Order 80LARC18F0021 Topics • MOCET Overview & Status • Updates in MOCET v1.4 • Planned Reserves Process • User Community • Summary & Future Work 2 MOCET Overview • The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA Science Office for Mission Assessments (SOMA) – Based on actual costs of historical missions with emphasis on competed missions – Constructed by breaking the mission operations cost into the various phases – Has few subjective inputs – Estimates total Phase E mission cost – Implemented entirely in Excel and requires no additional software or tools – Also includes a user manual which provides additional instruction and background Download from ONCE Model Portal https://oncedata.msfc.nasa.gov Available external to NASA via Mars 2034 Rover https://software.nasa.gov Mission Schedule Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37 Cruise For more information: Approach/EDL Email: [email protected] First Landed Month Landed Prime Operations 3 MOCET Mission Types Database Overview Mission/CER Type Program Missions Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn Mars Scout Phoenix Planetary Robotic Lunar Exploration LRO, LADEE New Frontiers New Horizons, Juno (Orbital Ops), OSIRIS-REx (Cruise, Approach/Orbit Insertion) Mars Exploration MRO, Odyssey, MER, MSL, MAVEN, Insight Earth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS, OCO-3, GEDI Earth Science Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, TSIS-1, ICESat-2, GRACE-FO Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI, NICER, GOLD Explorers Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI Medium Explorers (MIDEX) THEMIS, Swift, WISE, TESS Discovery Genesis, Kepler Solar Terrestrial Probes (STP) STEREO, TIMED, MMS Near Earth Discovery Living With a Star (LWS) RBSP, SDO, PSP Helio-Astro Cosmic Origins Spitzer Physics of the Cosmos Fermi, Chandra Bold = New Data in v1.4 Potential New Data for possible version 1.X update Data Sources Utilized and Consulted – NASA SAP Business – Monthly Flight Project Reviews (FPR) – NASA Cost Analysis Data Warehouse and Monthly Status Reviews (MSR) Requirements (CADRe) • Monthly Expenditures • Monthly Expenditures, Mission Events • Project development costs used & Schedule to assign mission class 4 MOCET CER Updates Planetary CER Name New Data Updated Comments Inflation Nominal/Quiescent Cruise CER X X OSIRIS-REx data incorporated Checkout Cruise CER X Landed Missions Broken Out Mars Landed Cruise CER X X New CER with Phoenix, Insight, and MSL Orbital Operations CER X X Juno & OSIRIS-REx data incorporated. Flyby/Encounter CER X Approach/Orbit Insertion/Reduction CER X X OSIRIS-REx data incorporated Approach EDL CER X X Insight data incorporated Landed Checkout CER X X Renamed; Insight data incorporated Landed Prime Operations X X Insight data incorporated Data/Sample Analysis & Archiving X Extended Orbital Ops CER X Extended Landed Ops CER X Earth/Near-Earth/Other CER Name New Data Updated Comments Inflation Earth/Near-Earth Orbiting Checkout CER X X Earth Science Prime Operations CER X X CYGNSS, SMAP Explorer Prime Operations CER X X NICER, GOLD, TESS data incorporated Near Earth Discovery Helio Astro Prime Operations CER X Instrument Only Operations CER X X New CER, Satellite and ISS Hosted 5 Explorers Prime Operations Phase CER • CER Function: = 0.5338 + 0.3740 0.3583 + 0.1225 Variable Name Units Definition Y Cost FY18 $M ∗Average monthly− checkout∗ phase cost ∗ IMe IsMedium? 0 or 1 0 No, 1 Yes, Is this a medium mission class mission? IMi IsMicro? 0 or 1 0 No, 1 Yes, Is this a micro mission class mission? IA IsAstro? 0 or 1 0 No, 1 Yes, Is this an astrophysics mission? • Goodness of Fit $1.2 Measure Value $1.0 Regression Method OLS $0.8 Standard Error (RSS) 0.07 $0.6 Average Percentage Bias 0.0% Coefficient of Determination (R^2) 0.96 Predicted $0.4 Adjusted R^2 0.95 $0.2 Pearson's Correlation Sqd (r^2) 0.96 $- Number of Observations 15 $- $0.2 $0.4 $0.6 $0.8 $1.0 $1.2 Number of Input Variables 3 Actual • Database 1. IBEX (IMe 0, IMi 0, IA 0) 6. Swift (IMe 1, IMi 0, IA 1) 11. Suzaku (IMe 0, IMi 1, IA 1) 2. AIM (IMe 0, IMi 0, IA 0) 7. WISE (IMe 1, IMi 0, IA 1) 12. IRIS (IMe 0, IMi 0, IA 0)* 3. NuSTAR (IMe 0, IMi 0, IA 1) 8. RHESSI (IMe 0, IMi 0, IA 0) 13. NICER (IMe 0, IMi 1, IA 1) 4. GALEX (IMe 0, IMi 0, IA 1) 9. CINDI (IMe 0, IMi 1, IA 0) 14. TESS (IMe 1, IMi 0, IA 1) 5. THEMIS (IMe 1, IMi 0, IA 0) 10. TWINS (IMe 0, IMi 1, IA 0) 15 GOLD (IMe 0, IMi 1, IA 0) Additional Notes • *IRIS was a SMEX mission but the development cost grew beyond small mission class boundary. In operations the IRIS average monthly cost is also higher than any other SMEX mission. • **Copies of identical instruments (THEMIS, TWINS) or suites of similarly functioned sensors (CINDI) are considered to be a single instrument since the same data processing tools can be used for them. • ***Earth & near Earth orbiting multiple spacecraft missions such as THEMIS may be estimated with MOCET without further adjusting the output estimate. 6 Mars EDL Cruise Prime Phase CER • CER Function: = 0.0056 . Variable Name Units 1 1388 Definition Y Cost ∗FY19 $M Planetary mission average monthly prime checkout cruise phase cost Mass of rover or lander plus payload instruments for which NASA will RLM Rover/Lander Mass kg > 0 fund operations (exclude contributed instruments). • Goodness of Fit $16 $14 Measure Value $12 Regression Method GERM ZMPE $10 Standard Error of the Estimate 14.2% $8 Average Percentage Bias 0.0% Predicted $6 Pearson's Correlation Sqd (R^2) 0.98 $4 Number of Observations 3 $2 Number of Input Variables 1 $- $- $2 $4 $6 $8 $10 $12 $14 $16 Actual • Database 1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6) 4. MER (A/B)* Additional Notes • *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions 7 Landed Operations Prime Phase CER • CER Function: = 0.0591 . Variable Name Units 0 7034 Definition Y Cost ∗FY19 $M Planetary mission average monthly prime landed operations phase cost Mass of rover or lander plus payload instruments for which NASA will RLM Rover/Lander Mass kg > 0 fund operations (exclude contributed instruments). • Goodness of Fit $8 $7 Measure Value $6 Regression Method GERM ZMPE $5 Standard Error of the Estimate 7.2% $4 Average Percentage Bias 0.0% Predicted $3 Pearson's Correlation Sqd (R^2) 0.98 $2 Number of Observations 3 $1 Number of Input Variables 1 $- $- $1 $2 $3 $4 $5 $6 $7 $8 Actual • Database 1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6) 4. MER (A/B)* Additional Notes • *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions 8 MOCET Reserves Approach (1 of 2) • When calculating the reserve level for operations phase estimates, a percent reserve of the base operations budget is often considered – Rule of thumb used by some considers a minimum 10% up to 25% for Phase E – Aerospace publication* showed that on average from launch estimate to final actual growth was around 10% • Not a simple matter of just looking at the offeror’s proposal and seeing if they are carrying between 10% and 25% reserve on their estimate – Offerors’ estimates and MOCET estimates vary significantly – Requires a new method of computing reserves for each MOCET estimate • To determine best method of computing reserves for MOCET – Obtained data from available Step 1 and Step 2 proposals as well as other milestones for comparison – Generated MOCET estimates using input parameters as given at Step 1, Step 2, etc. – Also, collected the offerors estimates for comparison *Phase E Cost Analysis for NASA Science Missions, Robert Bitten, Marc Hayhurst, Debra Emmons, Claude Freaner, and Voleak Roeum, AIAA SPACE 2012 Conference & Exposition. Pasadena, California. 9 MOCET Reserves Approach (2 of 2) • With the MOCET estimates generated at each milestone the percent difference from the final actual was computed for each mission – The following equation was used to compute the percent difference @ 100% = @ − ∗ • Then the overall average percent difference and standard deviation were computed • Various methods to compute reserve level were attempted and these metrics were compared at each milestone to determine which methods were most effective – Methodologies driven by historical data were considered to maintain objectivity • Concluded that the best performing method is to employ a triangle distribution with the Formal Risk (FRISK) process driven by historical percent differences of similar missions at a particular milestone – Similar missions are determined by mission category 10 Percent Difference for all Explorer Missions • As compared to Explorer Project