Mission Operations Cost Estimation Tool (MOCET) 2020/2021 Updates

Marc Hayhurst, Brian Wood The Aerospace Corporation Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo Rodriguez NASA Langley Research Center Science Office for Mission Assessments (SOMA)

April 14, 2021

• MOCET Overview

• Mission Data Updates

• Reserves Estimation

• Level 2 WBS Investigation

• ISS and Extended Missions

• User Community

• Conclusion

2 MOCET Overview

• The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA Science Office for Mission Assessments (SOMA) – Based on actual costs of historical missions with emphasis on competed missions – Constructed by breaking the mission operations cost into the various phases – Has few subjective inputs – Estimates total Phase E mission cost – Implemented entirely in Excel and requires no additional software or tools – Also includes a user manual which provides additional instruction and background

Download from ONCE Model Portal https://oncedata.hq.nasa.gov

Available external to NASA via

Mars 2034 Rover https://software.nasa.gov Mission Schedule Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37 Cruise For more information: Approach/EDL Email: [email protected]

First Landed Month

Landed Prime Operations

3 Mission Data Updates

Mission/CER Type Program Missions Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn Mars Scout Phoenix Planetary Robotic Lunar Exploration LRO, LADEE New Frontiers New Horizons, Juno, OSIRIS-REx Mars Exploration MRO, Odyssey, MER, MSL, MAVEN, Insight, Mars 2020 Earth System Science Pathfinder GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS, OCO-3, (ESSP) GEDI Earth Science Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, TSIS-1, Earth Systematic Missions (ESM) ICESat-2, GRACE-FO Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI, NICER, GOLD Explorers Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI Medium Explorers (MIDEX) THEMIS, Swift, WISE, TESS, ICON Discovery Genesis, Kepler Near Earth Solar Terrestrial Probes (STP) STEREO, TIMED, MMS Discovery Helio- Living With a Star (LWS) RBSP, SDO, PSP Astro Cosmic Origins Spitzer Physics of the Cosmos Fermi, Chandra Bold = New Data in 2020 Potential New Data for possible 2021 update

• Numerous missions updates in 2020 in each category • Fewer new missions in 2021 & COVID-19 challenges in data

4 Reserves Estimation Cost Estimates H M • Reserves process relies on calculated historical percent difference L • Percent differences applied to base MOCET estimate

�� − �� @ �� ∗ 100% �� @ �� = ��

Triangular Distribution Planetary Missions of Cost Estimates • Low, Most-Likely, and High derived from MOCET Estimate % Difference point estimates 60% • Formal Risk (FRISK) process used to 50% derive 70th percentile reserves value L M H 40% Probability 30% Cost Lower Most- Upper 20% 14% 10% Limit Likely Limit Example Cost Distribution 9% 6% 10% 3% -2% 0% 70th Percentile Percent Difference -10% MOCET -20% Estimate -30%

CSR PDR CDR EOM UFE Launch Proposal Estimate

Mission 1 Mission 2 Mission 3 Mission 4 Mission 5 Average

5 Level 2 WBS Investigation

• Level 2 WBS Modeling investigation performed this year • Not much precedent for this research • Study included 27 missions with actual Phase E operations cost available • NASA Systems, Applications and Products (SAP) monthly financial database is the most comprehensive data source • Next best source is NASA End of Mission/End of Prime Mission (EOM/EOPM) Cost Analysis Data Requirement (CADRe) reports • Both sources presented challenges as the bookkeeping of Level 2 costs in Phase E appears to have some inconsistencies • SAP has Level 2 detail available but accuracy depends on management center • CADRe EOM/EOPM Level 2 WBS cost data is only available by FY, unlike SAP which is monthly

6 Level 2 WBS Investigation

• Within the data with full Level 2 WBS detail • Additional bookkeeping issues found • E.g., no cost bookkept under science, large amount of cost under Payload • Initially normalization of these costs was attempted • Apprehension remained if this was the correct approach • The final determined approach was to downselect to the best data • Percent rank plots were used to look for trends and attempt to identify potentially incorrectly bookkept data Phase Data Source Missions Mission Type SAP OSIRIS-REx • Percent rank plots were also useful to Cruise SAP MAVEN establish nominal percent cost ranges for w/Checkout CADRe MESSENGER SAP OSIRIS-REx evaluating reasonable Level 2 WBS Nominal Cruise CADRe MESSENGER

allocations Planetary Flyby/Encounter N/A None

Approach/Orbit SAP OSIRIS-REx • Down selected perceived best quality data Insertion SAP MAVEN SAP OSIRIS-REx included 8 missions Orbital SAP MAVEN • 3 planetary and 5 EO Operations CADRe MESSENGER SAP GPM Checkout SAP MMS SAP Fermi Earth SAP GPM Orbiting SAP MMS Orbital SAP Fermi Operations CADRe OCO-2 CADRe NuSTAR 7 Level 2 WBS Investigation • Percent rank plots for two major Level 2 WBS categories of Science and Missions Operations shown below • In both graphs there is a noticeable split between planetary and Earth Orbiting (EO) missions • Some individual missions stand out well above the rest likely due to incorrect bookkeeping • Examples like this were removed

Science / Technology Mission Operations 100% 100% All Earth orbiting Outside normal range 90% 1 planetary 90% mission is 1 Earth orbiting 80% outside normal 80% mission is range outside 70% 70% normal range Mostly planetary 60% 60%

50% 50% Mostly planetary

Percent Rank 40% Percent Rank 40%

30% 30%

20% 20% Almost all Earth orbiting 10% 10% Outside normal range Outside normal range 0% 0% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Percent Cost by WBS Element Percent Cost by WBS Element

Planetary Science Earth Science Planetary Science Earth Science Astrophysics Heliophysics Astrophysics Heliophysics

8 Level 2 WBS Investigation Cruise w/ Checkout Nominal Cruise Flyby/Encounter* EPO, EPO, PM, I&T, 1% EPO, PM, 3% I&T, 1% SE, 3% I&T, 0% 0% PM, 5% SE, 4% 2% 6% 2% SE, 2% GDS, MA, 0% GDS, MA, 1% GDS, 3% 0% 0% MA, 0%

Science, 15% Science, MO, MO, Science, 33% Payloads, 28% 40% 49% MO, 4% 54% Spacecraft, 27% Payload Payloads, Spacecraft, Spacecraft, s, 4% 6% 7% 2%

• A proportional allocation model was Approach/Orbit Insertion Orbital Operations developed by taking the average EPO, PM, 3% I&T, 0% EPO, PM, 3% I&T, 1% percent cost from these 8 missions 3% SE, 2% 1% SE, 2% GDS, MA, 0% GDS, • Done for each individual operational 0% 0% MA, 0% phase in the model Percentages are applied to a top level MOCET estimate to derive the Level 2 cost Science, Science, values 31% MO, 34% • Planetary model is shown at right MO, 48% • *No data was available for the 59% flyby/encounter phase. Payloads, • An average was taken of the 2% Spacecraft, Payloads, approach/orbit insertion and Spacecraft, 6% orbital operations phases 0% 5% • Assumed to be a reasonable mix of science and missions operations activities • Observation: planetary missions appear to spend more on missions operations, likely because of the remote nature of operations and diversity of operational phases

9 Level 2 WBS Investigation

• Earth Orbiting (EO) model is shown below • Observation: unlike planetary missions, EO missions tend to spend the most on science, likely because of the continuous and homogenous nature of operations.

Checkout Operations I&T, 0% PM, 4% I&T, 3% EPO, 2% EPO, 1% SE, 0% GDS, 6% MA, 0% PM, 14% SE, 7% GDS, 17% MO, 20% MA, 1% MO, 1%

Spacecraft, Science, Spacecraft, 2% Science, 13% 17% 60% Payloads, Payloads, 6% 24%

• The results of this proportional allocation model for EO/Planetary model have been compared against final actual costs, as well as proposed costs at Step 2 evaluations • Initial results are encouraging, as potentially this model can be used to identify misallocation of Phase E costs early in the lifecycle • This model is still however limited including only 8 missions • Does not cover landed missions or instrument only missions at this time • As future work, it is planned to continue to collect data, and also to conduct interviews with projects and potentially help correct data where it appears cost may have been bookkept incorrectly

10 ISS Missions & Extended Missions

• ISS Instrument Capability • Initial version of an instrument only CER was developed • First version CER combined Spacecraft hosted and ISS Hosted • Produced less than desired results • More ISS instrument data has become available • Assumption ISS instrument missions operated differently that Spacecraft hosted • Greater technical detail has been gathered • Mass based CER has been derived for ISS instruments • This CER will likely be included in the next release of MOCET

• MOCET Extended Mission Cost Investigation • Users have expressed interest in being able to model extended missions • A large amount of extended mission data has been collected • Evaluating data for trends and determining the best way to model it • Planetary extended orbital and landed operations models were previously developed • Developing model of Earth Orbiting missions • Typically, extended mission cost continues to decrease • There are some exceptions to this general rule

11 ONCE Downloads and Users

• To date MOCET has been downloaded from ONCE 238* times since the initial release • *Downloads include those from inactive users and duplicate downloads • ONCE output generally only shows active users and unique downloads • Version 1.0 - 30, Version 1.1 - 60, Version 1.2 – 37, and Version 1.3 – 111

• As of October 2020, 98 unique users have downloaded MOCET from ONCE • Since the release of v1.0 the number of users has increased steadily

Cumulative Downloads Cumulative Users 250 120 NCS 2016 NCSS 2018 NCSS 2019 NCS 2017 NCS 2016 NCS 2017 NCSS 2018 NCSS 2019 100 200

v 1.3 v 1.3 80 150 v 1.2 v 1.2 60 100 v 1.1 40 v 1.1

50 v 1.0 20v 1.0

0 0 Oct-15 Jan-16 Apr-16 Jul-16 Oct-16 Jan-17 Apr-17 Jul-17 Oct-17 Jan-18 Apr-18 Jul-18 Oct-18 Jan-19 Apr-19 Jul-19 Oct-19 Jan-20 Apr-20 Jul-20 Oct-20 Jan-21 Jul-16 Jul-17 Jul-18 Jul-19 Jul-20 Oct-15 Oct-16 Oct-17 Oct-18 Oct-19 Oct-20 Jan-16 Apr-16 Jan-17 Apr-17 Jan-18 Apr-18 Jan-19 Apr-19 Jan-20 Apr-20 Jan-21

12 software.nasa.gov Downloads and Users

• There are MOCET 88 users on software.nasa.gov since its release there in April of 2017 • Most users are from Industry (39), Representing Self (21), Academic (11), and from NASA (10) • There are also 7 Other Civil/Government • Of the 88 users, 31 are also international coming from Africa, Asia, Australia, Europe, and South America

Non

US, 4 - US, 3

US, 10 Other Civil

NASA US, 29

Non-US, 5 Industry Academic

US, 6

Non

Rep. Self US, 8 -US, 10

US, 13

Non

13 Conclusion

• Status of current 2020 and planned 2021 updates presented

• MOCET will continue to be periodically updated with new mission data

• Model is currently being used by both evaluators and proposers

• We will continue to engage and grow the user community

Download MOCET ONCE Model Portal https://oncedata.hq.nasa.gov Available external to NASA via https://software.nasa.gov

For More Information Email: [email protected]

Publications • Mission Operations Cost Estimation Tool (MOCET) 2020, 2020 IEEE Aerospace Conference, Big Sky, MT • Mission Operations Cost Estimation Tool (MOCET) Version 1.3 and Beyond, 2019 IEEE Aerospace Conference, Big Sky, MT • The Mission Operations Cost Estimation Tool (MOCET): Development History and 2018 Updates, 2018 AIAA SPACE and Astronautics Forum and Exposition. Orlando FL • Mission Operations Cost Estimation Tool (MOCET) v1.3, 2018 NASA Cost and Schedule Symposium, August 2018, Greenbelt MD • Mission Operations Cost Estimation Tool (MOCET), 2017 IEEE Aerospace Conference, Big Sky, MT • Mission Operations Cost Estimation Tool (MOCET) FY17 Update, 2017 NASA Cost and Schedule Symposium, August 2017, Washington DC • Mission Operations Cost Estimation Tool (MOCET) Update, 2016 NASA Cost Symposium, August 2016, Cleveland OH • Mission Operations Cost Estimation Tool (MOCET), 2015 NASA Cost Symposium, August 2015, Mountain View CA