Mission Operations Cost Estimation Tool (MOCET) 2019 Updates

Marc Hayhurst, Elliott Tibor, Brian Wood The Aerospace Corporation

Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo Rodriguez NASA Langley Research Center Science Office for Mission Assessments (SOMA)

August 14, 2019

Approved for public release OTR 2019-01048.

• MOCET Overview & Status

• Updates in MOCET v1.4

• Planned Reserves Process

• User Community

• Summary & Future Work

2 MOCET Overview

• The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA Science Office for Mission Assessments (SOMA) – Based on actual costs of historical missions with emphasis on competed missions – Constructed by breaking the mission operations cost into the various phases – Has few subjective inputs – Estimates total Phase E mission cost – Implemented entirely in Excel and requires no additional software or tools – Also includes a user manual which provides additional instruction and background

Download from ONCE Model Portal https://oncedata.msfc.nasa.gov

Available external to NASA via

Mars 2034 Rover https://software.nasa.gov Mission Schedule Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37 Cruise For more information: Approach/EDL Email: [email protected] First Landed Month

Landed Prime Operations

3 MOCET Mission Types Database Overview

Mission/CER Type Program Missions

Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn Mars Scout Phoenix Planetary Robotic Lunar Exploration LRO, LADEE New Frontiers New Horizons, Juno (Orbital Ops), OSIRIS-REx (Cruise, Approach/Orbit Insertion) Mars Exploration MRO, Odyssey, MER, MSL, MAVEN, Insight Earth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS, OCO-3, GEDI Earth Science Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, TSIS-1, ICESat-2, GRACE-FO

Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI, NICER, GOLD

Explorers Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI

Medium Explorers (MIDEX) THEMIS, Swift, WISE, TESS Discovery Genesis, Kepler Solar Terrestrial Probes (STP) STEREO, TIMED, MMS Near Earth Discovery Living With a Star (LWS) RBSP, SDO, PSP Helio-Astro Cosmic Origins Spitzer Physics of the Cosmos Fermi, Chandra

Bold = New Data in v1.4 Potential New Data for possible version 1.X update Data Sources Utilized and Consulted – NASA SAP Business – Monthly Flight Project Reviews (FPR) – NASA Cost Analysis Data Warehouse and Monthly Status Reviews (MSR) Requirements (CADRe) • Monthly Expenditures • Monthly Expenditures, Mission Events • Project development costs used & Schedule to assign mission class 4 MOCET CER Updates Planetary CER Name New Data Updated Comments Inflation

Nominal/Quiescent Cruise CER X X OSIRIS-REx data incorporated

Checkout Cruise CER X Landed Missions Broken Out

Mars Landed Cruise CER X X New CER with Phoenix, Insight, and MSL

Orbital Operations CER X X Juno & OSIRIS-REx data incorporated.

Flyby/Encounter CER X

Approach/Orbit Insertion/Reduction CER X X OSIRIS-REx data incorporated

Approach EDL CER X X Insight data incorporated

Landed Checkout CER X X Renamed; Insight data incorporated

Landed Prime Operations X X Insight data incorporated

Data/Sample Analysis & Archiving X

Extended Orbital Ops CER X

Extended Landed Ops CER X

Earth/Near-Earth/Other

CER Name New Data Updated Comments Inflation

Earth/Near-Earth Orbiting Checkout CER X X

Earth Science Prime Operations CER X X CYGNSS, SMAP

Explorer Prime Operations CER X X NICER, GOLD, TESS data incorporated

Near Earth Discovery Helio Astro Prime Operations CER X

Instrument Only Operations CER X X New CER, Satellite and ISS Hosted 5 Explorers Prime Operations Phase CER • CER Function: = 0.5338 + 0.3740 0.3583 + 0.1225 Variable Name Units Definition Y 𝑌𝑌Cost FY18 $M ∗Average𝐼𝐼𝐼𝐼𝐼𝐼 monthly− checkout∗ phase𝐼𝐼𝐼𝐼𝐼𝐼 cost ∗ 𝐼𝐼𝐼𝐼 IMe IsMedium? 0 or 1 0 No, 1 Yes, Is this a medium mission class mission? IMi IsMicro? 0 or 1 0 No, 1 Yes, Is this a micro mission class mission? IA IsAstro? 0 or 1 0 No, 1 Yes, Is this an astrophysics mission?

• Goodness of Fit $1.2

Measure Value $1.0

Regression Method OLS $0.8 Standard Error (RSS) 0.07 $0.6 Average Percentage Bias 0.0% Coefficient of Determination (R^2) 0.96 Predicted $0.4 Adjusted R^2 0.95 $0.2

Pearson's Correlation Sqd (r^2) 0.96 $- Number of Observations 15 $- $0.2 $0.4 $0.6 $0.8 $1.0 $1.2 Number of Input Variables 3 Actual • Database

1. IBEX (IMe 0, IMi 0, IA 0) 6. Swift (IMe 1, IMi 0, IA 1) 11. Suzaku (IMe 0, IMi 1, IA 1) 2. AIM (IMe 0, IMi 0, IA 0) 7. WISE (IMe 1, IMi 0, IA 1) 12. IRIS (IMe 0, IMi 0, IA 0)* 3. NuSTAR (IMe 0, IMi 0, IA 1) 8. RHESSI (IMe 0, IMi 0, IA 0) 13. NICER (IMe 0, IMi 1, IA 1) 4. GALEX (IMe 0, IMi 0, IA 1) 9. CINDI (IMe 0, IMi 1, IA 0) 14. TESS (IMe 1, IMi 0, IA 1) 5. THEMIS (IMe 1, IMi 0, IA 0) 10. TWINS (IMe 0, IMi 1, IA 0) 15 GOLD (IMe 0, IMi 1, IA 0)

Additional Notes • *IRIS was a SMEX mission but the development cost grew beyond small mission class boundary. In operations the IRIS average monthly cost is also higher than any other SMEX mission. • **Copies of identical instruments (THEMIS, TWINS) or suites of similarly functioned sensors (CINDI) are considered to be a single instrument since the same data processing tools can be used for them. • ***Earth & near Earth orbiting multiple spacecraft missions such as THEMIS may be estimated with MOCET without further adjusting the output estimate.

6 Mars EDL Cruise Prime Phase CER • CER Function: = 0.0056 . Variable Name Units 1 1388 Definition Y 𝑌𝑌Cost ∗FY19𝑅𝑅𝑅𝑅𝑅𝑅 $M Planetary mission average monthly prime checkout cruise phase cost Mass of rover or lander plus payload instruments for which NASA will RLM Rover/Lander Mass kg > 0 fund operations (exclude contributed instruments).

• Goodness of Fit $16 $14 Measure Value $12 Regression Method GERM ZMPE $10 Standard Error of the Estimate 14.2% $8 Average Percentage Bias 0.0%

Predicted $6 Pearson's Correlation Sqd (R^2) 0.98 $4 Number of Observations 3 $2 Number of Input Variables 1 $- $- $2 $4 $6 $8 $10 $12 $14 $16 Actual • Database

1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6) 4. MER (A/B)*

Additional Notes • *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions

7 Landed Operations Prime Phase CER • CER Function: = 0.0591 . Variable Name Units 0 7034 Definition Y 𝑌𝑌Cost ∗FY19𝑅𝑅𝑅𝑅𝑅𝑅 $M Planetary mission average monthly prime landed operations phase cost Mass of rover or lander plus payload instruments for which NASA will RLM Rover/Lander Mass kg > 0 fund operations (exclude contributed instruments).

• Goodness of Fit $8 $7 Measure Value $6 Regression Method GERM ZMPE $5 Standard Error of the Estimate 7.2% $4 Average Percentage Bias 0.0%

Predicted $3 Pearson's Correlation Sqd (R^2) 0.98 $2 Number of Observations 3 $1 Number of Input Variables 1 $- $- $1 $2 $3 $4 $5 $6 $7 $8 Actual • Database

1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6) 4. MER (A/B)*

8 MOCET Reserves Approach (1 of 2)

• When calculating the reserve level for operations phase estimates, a percent reserve of the base operations budget is often considered – Rule of thumb used by some considers a minimum 10% up to 25% for Phase E – Aerospace publication* showed that on average from launch estimate to final actual growth was around 10%

• Not a simple matter of just looking at the offeror’s proposal and seeing if they are carrying between 10% and 25% reserve on their estimate – Offerors’ estimates and MOCET estimates vary significantly – Requires a new method of computing reserves for each MOCET estimate

• To determine best method of computing reserves for MOCET – Obtained data from available Step 1 and Step 2 proposals as well as other milestones for comparison – Generated MOCET estimates using input parameters as given at Step 1, Step 2, etc. – Also, collected the offerors estimates for comparison

*Phase E Cost Analysis for NASA Science Missions, Robert Bitten, Marc Hayhurst, Debra Emmons, Claude Freaner, and Voleak Roeum, AIAA SPACE 2012 Conference & Exposition. Pasadena, California. 9 MOCET Reserves Approach (2 of 2)

• With the MOCET estimates generated at each milestone the percent difference from the final actual was computed for each mission – The following equation was used to compute the percent difference @ 100% = @ 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 − 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 ∗ 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 • Then the overall average percent difference and standard deviation were computed

• Various methods to compute reserve level were attempted and these metrics were compared at each milestone to determine which methods were most effective – Methodologies driven by historical data were considered to maintain objectivity

• Concluded that the best performing method is to employ a triangle distribution with the Formal Risk (FRISK) process driven by historical percent differences of similar missions at a particular milestone – Similar missions are determined by mission category

10 Percent Difference for all Explorer Missions

• As compared to Explorer Project estimates, average percent difference for MOCET is significantly less – Interesting to note that on average Step 1 estimates were better than those at Step 2 Project Estimate % Difference (Including Reserves) MOCET Estimate % Difference (Without Reserve) 250% 250% Mission 1 Mission 1 Mission 2 Mission 2 Mission 3 Mission 3 200% 200% Mission 4 Mission 4 Mission 5 Mission 5 Mission 6 Mission 6 150% Mission 7 150% Mission 7 Mission 8 Mission 8 Average Average

100% 100% 72% 71%

Percent Difference 48% Percent Difference 50% 34% 50% 24% 9% 9% 10% 8% Underestimate Underestimate 6% 0% 2% 0% 0%

-50% -50% Overestimate Proposal CSR PDR CDR Launch EOM Overestimate Proposal CSR PDR CDR Launch EOM

11 Percent Difference for all Planetary Missions

• As compared to Planetary Project estimates, average percent difference for MOCET is significantly less – Interesting to note that on average Step 1 estimates were better than those at Step 2 Project Estimate % Difference (Including Reserves) MOCET Estimate % Difference (Without Reserve)

Mission 1 Mission 1

80% Mission 2 80% Mission 2 Mission 3 Mission 3 Mission 4 Mission 4

60% Mission 5 60% Mission 5 50% Average Average 44% 40% 40% 40% 32% Differenece 20% 20% 14% 10% 9% 10% 6% Percent Difference

Percent 3%

Underestimate 0%

Underestimate -2% 0% 0%

-20% -20% Overestimate Overestimate

-40% -40% Proposal CSR PDR CDR Launch EOM Proposal CSR PDR CDR Launch EOM

12 MOCET Reserves Process Overview

Cost Estimates Triangular Distribution of Cost Estimates H M L

Note: L M H Most-

Likely (M) Probability is average Cost of all Lower Most-Likely Upper estimates Limit Limit

Example Cost Distribution • The Formal Risk (FRISK) process driven by historical percent differences of similar 70th Percentile missions at a particular milestone • Similar missions are determined by MOCET Estimate mission category • Adjusted mission estimates are fed into a triangular distribution UFE Estimate • Triangular distribution is fed into FRISK to produce an s-curve and determine the appropriate reserve level

13 ONCE Downloads and Users

• To date MOCET has been downloaded from ONCE 183* times since the initial release • *Downloads include those from inactive users and duplicate downloads – ONCE output generally only shows active users and unique downloads • v1.0 30 downloads; v1.1 60 downloads; v1.2 34 downloads; v1.3 59 downloads as of June 2019 • As of June 2019, 86 unique users have downloaded MOCET from ONCE • Since the release of v1.0 the number of users has increased steadily

v 1.0 v 1.1 v 1.2 v 1.3 v 1.0 v 1.1 NCS 2016 v 1.2 NCS 2017 v 1.3 NCSS 2018 NCS 2016 NCS 2017 NCSS 2018 Cumulative Downloads Downloads by Month

200 18 17

180 16 160 14 140 12 12 11 11 120 10 10 100 9 9

80 8 7 7 7 7 6 60 6 5 5 5 5 5 5 40 4 4 4 3 3 3 3 3 20 2 2 2 2 2 2 2 1 1 1 1 1 1

0 10/6/2015 1/4/2016 4/3/2016 7/2/2016 9/30/2016 12/29/2016 3/29/2017 6/27/2017 9/25/2017 12/24/2017 3/24/2018 6/22/2018 9/20/2018 12/19/2018 3/19/2019 6/17/2019 0 0 0 0 0 0 0 0

0 Oct-15 Nov-15 Dec-15 Jan-16 Feb-16 Mar-16 Apr-16 May-16 Jun-16 Jul-16 Aug-16 Sep-16 Oct-16 Nov-16 Dec-16 Jan-17 Feb-17 Mar-17 Apr-17 May-17 Jun-17 Jul-17 Aug-17 Sep-17 Oct-17 Nov-17 Dec-17 Jan-18 Feb-18 Mar-18 Apr-18 May-18 Jun-18 Jul-18 Aug-18 Sep-18 Oct-18 Nov-18 Dec-18 Jan-19 Feb-19 Mar-19 Apr-19 May-19 Jun-19

14 Public Release on software.nasa.gov

• MOCET has been approved for public release via software.nasa.gov – The direct link can be found here: https://software.nasa.gov/software/LAR-18894-1

• Available to almost anyone* worldwide

• User must create an account on software.nasa.gov

• Fill out the request form

• *A signature authority must be provided for those representing businesses – Cannot be an engineer or those with similar titles – An authorized signature authority is considered • president of the company • general counsel • or contract officer

• If representing self as a private individual, an additional signature is not required

• Rules may be different for other user types, Academic etc.

• Contact MOCET help staff at [email protected] with any issues you may encounter 15 software.nasa.gov Downloads and Users

• There are MOCET 31 users on software.nasa.gov since its release there in April of 2017 • Most users are from Industry (14) and from NASA (7) • There are also 5 Representing Self, 3 Other Civil, and 2 Academic • Of the 31 users, 11 are also international coming from Africa, Asia, Australia, Europe, and South America

16 Summary & Next Steps

• New data is being added to the model which will be released as MOCET v1.4 • MOCET will continue to be periodically updated with new mission data • Model is currently being used by both evaluators and proposers • We will continue to engage and grow the user community

Download MOCET ONCE Model Portal https://oncedata.hq.nasa.gov Available external to NASA via https://software.nasa.gov

For More Information Demo session (see agenda for time and room) Email: [email protected]

Publications • Mission Operations Cost Estimation Tool (MOCET) Version 1.3 and Beyond, 2019 IEEE Aerospace Conference, Big Sky, MT • The Mission Operations Cost Estimation Tool (MOCET): Development History and 2018 Updates, 2018 AIAA SPACE and Astronautics Forum and Exposition. Orlando FL • Mission Operations Cost Estimation Tool (MOCET) v1.3, 2018 NASA Cost and Schedule Symposium, August 2018, Greenbelt MD • Mission Operations Cost Estimation Tool (MOCET), 2017 IEEE Aerospace Conference, Big Sky, MT • Mission Operations Cost Estimation Tool (MOCET) FY17 Update, 2017 NASA Cost and Schedule Symposium, August 2017, Washington DC • Mission Operations Cost Estimation Tool (MOCET) Update, 2016 NASA Cost Symposium, August 2016, Cleveland OH • Mission Operations Cost Estimation Tool (MOCET), 2015 NASA Cost Symposium, August 2015, Mountain View CA