Mission Operations Cost Estimation Tool (MOCET) Version 1.3

Marc Hayhurst, Brian Wood The Aerospace Corporation

Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo Rodriguez NASA Langley Research Center Science Office for Mission Assessments (SOMA)

August 15, 2018

• MOCET Overview & Status

• Updates in MOCET v1.3

• User Community

• Summary & Future Work

2 MOCET Overview

• The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA Science Office for Mission Assessments (SOMA) – Based on actual costs of historical missions with emphasis on competed missions – Constructed by breaking the mission operations cost into the various phases – Has few subjective inputs – Estimates total Phase E mission cost – Implemented entirely in Excel and requires no additional software or tools – Also includes a user manual which provides additional instruction and background

Download from ONCE Model Portal https://oncedata.msfc.nasa.gov

Available external to NASA via

Mars 2034 Rover https://software.nasa.gov Mission Schedule Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37 Cruise For more information: Approach/EDL Email: [email protected]

First Landed Month

Landed Prime Operations

3 MOCET Timeline Presentation at NASA Cost MOCET v1.0 Symposium release 2015 2015

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NASA Cost MOCET v1.1 MOCET User’s Symposium MOCET User’s 2016 release Workshop 1 2016 v1.1 Demo Workshop 2

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

MOCET v1.2 Initial approval NASA Cost and MOCET Cost and release; for MOCET Schedule Schedule Team Award IEEE Conf.* public release Symposium 2017 2017 Win!!!

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NASA Cost and MOCET v1.3 Schedule AIAA SPACE 2018 release Symposium 2018 Orlando 2018

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

*M. R. Hayhurst et al., "Mission Operations Cost Estimation Tool (MOCET)," 2017 IEEE Aerospace Conference, Big Sky, MT, 2017, pp. 1-13.

4 MOCET Mission Types Database Overview

Mission/CER Type Program Missions

Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn Mars Scout Phoenix, Insight (Cruise, EDL, Landed Ops) Planetary Robotic Lunar Exploration LRO, LADEE New Frontiers New Horizons, Juno (Orbital Ops), OSIRIS-REx (Cruise, Flyby) Mars Exploration MRO, Odyssey, MER, MSL, MAVEN (Orbital Ops) Earth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2 (Additional Ops), CYGNSS Earth Science Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM (Additional Ops), SMAP, Jason-3

Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI, NICER (Instrument Ops)

Explorers Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI

Medium Explorers (MIDEX) THEMIS, Swift, WISE Discovery Genesis, Kepler Solar Terrestrial Probes (STP) STEREO, TIMED, MMS (Orbital Ops) Near Earth Discovery Living With a Star (LWS) RBSP, SDO Helio-Astro Cosmic Origins Spitzer Physics of the Cosmos Fermi, Chandra

Bold = New Data in v1.3 Potential New Data for possible version 1.X update Data Sources Utilized and Consulted – NASA SAP Business – Monthly Flight Project Reviews (FPR) – NASA Cost Analysis Data Warehouse and Monthly Status Reviews (MSR) Requirements (CADRe) • Monthly Expenditures • Monthly Expenditures, Mission Events • Project development costs used & Schedule to assign mission class 5 MOCET v1.3 CER Updates Planetary

CER Name New New Eq. Updated Comments Data Form Inflation

Nominal/Quiescent Cruise CER X X OSIRIS-REx data incorporated

Checkout Cruise CER X X OSIRIS-REx data incorporated

Orbital Operations CER X X Juno & MAVEN data incorporated.

Flyby/Encounter CER X

Approach/Orbit Insertion/Reduction CER X

Approach EDL CER X

First Landed Month CER X

Landed Prime Operations X

Data/Sample Analysis & Archiving X

Extended Orbital Ops CER X

Extended Landed Ops CER X

Earth/Near-Earth Orbiting

CER Name New Data New Eq. Updated Comments Form Inflation

Earth/Near-Earth Orbiting Checkout CER X X MMS data incorporated

Earth Science Prime Operations CER X

Explorer Prime Operations CER X

Near Earth Discovery Helio Astro Prime Operations CER X X MMS data incorporated

6 MOCET v1.3 Update Overview

• MOCET CERs and tool have been updated to include new mission data and released as v1.3 – New missions not previously included in MOCET are OSIRIS-REx and MMS – Additional updated data for ongoing missions already found in MOCET include Juno and MAVEN – The same statistical processes used for previous versions have been applied to this update – The new v1.3 CERs have been benchmarked against v1.2 – Overall the performance of the v1.3 updates is about the same as v1.2

• All new/updated data is consistent with existing CERs – All updates resulted in coefficient changes, without major changes to variables or mission types – Experimentation was performed to find best fit with different equation types and other prominent variable as needed – In nearly all cases the existing variables and equation forms were found to be the best performing CERs

• This update affected five CERs – Nominal/Quiescent Cruise CER – OSRIS-REx – Checkout Cruise CER – OSIRIS-REx – Orbital Operations – Juno, MAVEN – Near Earth Discovery Helio-Astro Prime Operations CER - MMS – Checkout for Earth/Near Earth Orbiters - MMS

• Mission class guide was also reevaluated and updated as needed to be consistent with inflation – Nearly all bins still adequately fit mission data set; Flagship/Large threshold raised to $1B FY18 – Clarification also added to mission class guide: • Missions included in MOCET are assigned a Mission Class based on actual Phase BCD cost and not based on Program or cost cap alone. Selection of mission class for proposed missions should be based estimated lifecycle cost. The programs and cost caps are meant to serve only as a generic guide

• Inflation has also been updated to FY18 for all CERs and latest inflation table has been incorporated 7 Mission Class Planetary

Mission NASA Planetary Programs Latest Example Missions Class Funded Cost Cap** ($M) Development Cost* Discovery $450 (FY15 w/o Ops) GRAIL Medium <$400 Mars Scout $325 (FY03 w/LV/Ops) Phoenix Discovery $450 (FY15 w/o Ops) Large $400-$1,000 New Frontiers $850 (w/o Ops) Juno Directed Survey MRO Mars Exploration MSL Flagship >$1,000 Outer Planets Flagship Cassini Bold = Updated for v1.3 Note: Missions included in MOCET are assigned a Mission Class based on actual Phase BCD cost and not based on Program or cost cap alone. Selection of mission class for proposed missions should be based estimated lifecycle cost. The programs and cost caps shown above are meant to serve only as a generic guide. *FY18$M Excludes contributions, launch vehicle and operations costs **Sources: 1) Discovery 2014 Announcement of Opportunity, as amended http://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=438340/solicitationId=%7BFE7B4C63-873D-63C1-4D15- 1D46E2FEA949%7D/viewSolicitationDocument=1/discovery-2014-amend1.pdf 2) https://discovery.larc.nasa.gov/ 3) New Frontiers 4 Announcement of Opportunity, December 9 2016 https://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=548004/solicitationId=%7BF65A5657-0E72-362E-2D4C- DE87A16A82B7%7D/viewSolicitationDocument=1/NF4%20AO.pdf 4) https://newfrontiers.larc.nasa.gov/ 5) http://www.nasa.gov/centers/langley/news/releases/2002/02-090.html (Mars Scout) 8 Mission Class Earth Science, Explorers, and Near Earth Discovery Helio Astro

Mission NASA Funded Programs Latest Example Missions Class Development Cost Cap** ($M) Cost* Missions of Opportunity (MO) $55 Suzaku Micro <$60 Earth Venture-Instrument (EVI) $90 for all selected Earth Systematic Missions (ESM) ACRIMSAT Earth Venture-Instrument (EVI) $90 for all selected TEMPO Small Explorers (SMEX) $164 (FY w/ Ops LV) GALEX Small $60-$120 Earth Systems Science Pathfinder (ESSP) GRACE Earth Systematic Missions (ESM) SORCE Earth Venture-M $150 CYGNSS Medium Explorers (MIDEX) $250 (FY17 w/ ops) Swift Explorers (EX) $200 (FY11) TESS Medium $120-$400 Earth Systems Science Pathfinder (ESSP) CloudSat Earth Systematic Missions (ESM) ICESat Discovery $450 (FY15 w/o Ops) Genesis Solar Terrestrial Probes (STP) TIMED Earth Systematic Missions (ESM) LDCM Discovery $450 (FY15 w/o Ops) Kepler Large $400-$1,000 Solar Terrestrial Probes (STP) STEREO Living With a Star (LWS) SDO Physics of the Cosmos (PCOS) Fermi Earth Systematic Missions (ESM) Aqua, Aura Flagship >$1,000 Cosmic Origins (COR) Spitzer Physics of the Cosmos (PCOS) Chandra Bold = Updated for v1.3 Note: Missions included in MOCET are assigned a Mission Class based on actual Phase BCD cost and not based on Program or cost cap alone. Selection of mission class for proposed missions should be based estimated lifecycle cost. The programs and cost caps shown above are meant to serve only as a generic guide. *FY18$M Excludes contributions, launch vehicle and operations costs **Sources: 1) Explorers: http://explorers.gsfc.nasa.gov/missions.html; Explorers (EX) class specified FY11 dollars and excludes the launch vehicle. Missions of Opportunity (MO) on a no-exchange-of-funds basis. Year dollars unspecified for all others and includes launch and operations. 2) 2016 Astrophysics Medium Explorer Announcement of Opportunity, November 14, 2016, https://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=535096/solicitationId=%7B8FB8BF8F-8248-8FB1-DDC5- 54E7AE878E5A%7D/viewSolicitationDocument=1/2016%20Astro%20MIDEX%20AO_Amend2.pdf 3) 2016 Heliophysics Small Explorer Announcement of Opportunity, November 14, 2016, https://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=524225/solicitationId=%7BA0C496AC-9B9D-8F7D-A506- B1695BF9BDE8%7D/viewSolicitationDocument=1/2016%20Helio%20SMEX%20AO_amend1_clarify.pdf 4) Earth Science: http://nspires.nasaprs.com/external/ 5) Announcement of Opportunity NASA’s Second Stand Alone Missions of Opportunity Notice (SALMON-2). FY14 dollars, excludes launch vehicle and integration to platform costs. Includes operations. 6) Announcement of Opportunity Earth Venture – 2 Released June 17, 2011; Amended September 1, 2011. FY14 dollars, Includes launch vehicle and operations. 7) Discovery 2014 Announcement of Opportunity, as amended http://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=438340/solicitationId=%7BFE7B4C63-873D-63C1-4D15- 9 1D46E2FEA949%7D/viewSolicitationDocument=1/discovery-2014-amend1.pdf Near Earth Discovery Helio-Astro Prime Phase CER • CER Function:푌 = 1.1902 + 0.7486 ∗ 퐼퐿 + 5.7491 ∗ 퐼퐹 Variable Name Units Definition Near Earth Discovery Helio-Astro mission average monthly prime Y Cost FY18 $M operations phase cost IL IsLarge? 0 or 1 0 No, 1 Yes IF IsFlagship? 0 or 1 0 No, 1 Yes

• Goodness of Fit $8 Measure Value $7 Regression Method OLS $6 $5 Standard Error (RSS) 0.20 $4 Average Percentage Bias 0.0% $3 Coefficient of Determination (R^2) 0.99 $2 Adjusted R^2 0.99

$1 (FY18$M) Pearson's Correlation Sqd (r^2) 0.99 $- Number of Observations 10 $- $1 $2 $3 $4 $5 $6 $7 $8 Actual Avg. Monthly Cost (FY18$M)

Number of Input Variables 2 Cost Monthly Avg. Predicted • Database

1. Fermi (IsLarge? 1, IsFlagship? 0) 5. TIMED (IsLarge? 0, IsFlagship? 0)* 9. Chandra (IsLarge? 0 IsFlagship? 1) 2. Kepler (IsLarge? 1, IsFlagship? 0) 6. GP-B (IsLarge? 0, IsFlagship? 1) 10. STEREO (IsLarge? 1, IsFlagship? 0)** 3. SDO (IsLarge? 1, IsFlagship? 0) 7. Genesis (IsLarge? 0, IsFlagship? 0) 11. MMS (IsLarge? 1, IsFlagship? 0) 4. Spitzer (IsLarge? 0, IsFlagship? 1) 8. VAP (IsLarge? 1, IsFlagship? 0)**

Additional Notes • *TIMED only has four months of data for the prime mission available • **Earth & near Earth orbiting multiple spacecraft missions such as VAP and STEREO may be estimated with MOCET without further adjusting the output estimate. • Although GP-B was assumed to be a flagship mission, the actual Phase BCD development cost was not available.GP-B also operated quite differently than the other multi- instrumented flagship astrophysics missions, as it used sensitive gyroscopes and a guide telescope. For these reasons it has been removed.

10 Nominal/Quiescent Cruise Prime Phase CER • CER Function: 푌 = 1.4109 + 1.0455 ∗ 퐼퐿 − 0.9301 ∗ 퐼푄 Variable Name Units Definition Planetary mission average monthly prime nominal or quiescent cruise Y Cost FY18 $M phase cost IL IsLarge? 0 or 1 Is this mission a large mission class per MOCET definitions? IQ IsQuiescent? 0 or 1 Is this cruise period a quiescent mode?

• Goodness of Fit $3.0

Measure Value $2.5 Regression Method OLS $2.0 Standard Error (RSS) 0.26 $1.5 Average Percentage Bias 0.0% Coefficient of Determination (R^2) 0.91 $1.0 Adjusted R^2 0.87 $0.5

Pearson's Correlation Sqd (r^2) 0.91 (FY18$M) $- Number of Observations 7 $- $0.5 $1.0 $1.5 $2.0 $2.5 $3.0 Actual Avg. Monthly Cost (FY18$M)

Number of Input Variables 2 Cost Monthly Avg. Predicted • Database

1. MESSENGER – Nominal (IL 0, IQ 0) 4. New Horizons – Nominal (IL 1, IQ 0) 7. OSIRIS-REx – Nominal (IL 1, IQ 0) 2. Dawn - Nominal (IL 0, IQ 0) 5. Stardust - Quiescent (IL 0, IQ 1)** 8. GRAIL (A/B) – (IL 0, IQ 0)* 3. Juno - Nominal (IL 1, IQ 0) 6. New Horizons - Quiescent (IL 1, IQ 1) 9. Cassini – (IL 0, IQ 0)***

Additional Notes • Some missions may have nominal, quiescent, and checkout periods of cruise. Each of these phases should be broken out and estimated separately (e.g. 4 months of cruise with checkout, 12 months of nominal cruise, 36 months of quiescent ops.) • *GRAIL was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions • **Stardust cruise considered as quiescent during the original prime mission. Limited data was available about the original mission so this assumption is based on description of the NEXT Mission. • ***Cassini was not included in the CER since it the only example of a flagship class mission currently available

11 Checkout Cruise Prime Phase CER • CER Function: 푌 = 2.3223 + 0.4800 ∗ 퐼퐿 + 2.2850 ∗ 퐸푁 Variable Name Units Definition Y Cost FY18 $M Planetary mission average monthly prime checkout cruise phase cost IL IsLarge? 0 or 1 Is this mission a large mission class per MOCET definitions? Does this cruise require enhanced navigation: • Cruise precedes: entry, descent, landing, aerobraking, or precise targeting operations OR • There are >2 thrust control maneuvers during this cruise with checkout period OR EN EnhancedNavigation? 0 or 1 • Total cruise may be less than 6 months (excluding approach phases),

• Goodness of Fit $6

Measure Value $5

Regression Method OLS $4 Standard Error (RSS) 0.34 $3 Average Percentage Bias 0.0% Coefficient of Determination (R^2) 0.94 $2

Adjusted R^2 0.92 $1 (FY18$M) Pearson's Correlation Sqd (r^2) 0.94 $- Number of Observations 8 $- $1 $2 $3 $4 $5 $6 Actual Avg. Monthly Cost (FY18$M)

Number of Input Variables 2 Cost Monthly Avg. Predicted • Database

1. MESSENGER (IL 0, EN 0) 5. Phoenix (IL 0, EN 1) 9. MER (A/B) – Checkout* 2. Juno (IL 1, EN 0) 6. MRO (IL 1, EN 1) 10. MSL – Checkout** 3. New Horizons (IL 1, EN 1) 7. Deep Impact (IL 0, EN 1) 4. MAVEN (IL 0, EN 0) 8.OSIRIS-REx (IL 1, EN 1)

Additional Notes • Some missions may have nominal, quiescent, and checkout periods of cruise. Each of these phases should be broken out and estimated separately (e.g. 4 months of cruise with checkout, 12 months of nominal cruise, 36 months of quiescent ops.) • Mars missions and Deep Impact assumed to use “checkout during cruise” rate until approach phase begins to account for compressed activities in short duration cruises • *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions • **MSL was not included in the CER since it the only example of a flagship class mission currently available

12 Orbital Operations Prime Phase CER • CER Function:푌 = 1.2589 + 0.0145 ∗ 푃푀 Variable Name Units Definition Y Cost FY18 $M Planetary mission average monthly prime orbital operations phase cost PM Payload Mass* kg > 0 Mass of payload portion for which NASA will fund operations

$4.0 • Goodness of Fit $3.5 $3.0 Measure Value $2.5 Regression Method GERM ZMPE $2.0 Standard Error of the Estimate 12.0% $1.5 Average Percentage Bias 0.0% $1.0

Pearson's Correlation Sqd (r^2) 0.79 $0.5 (FY18$M) Number of Observations 8 $- Number of Input Variables 1 $- $1.0 $2.0 $3.0 $4.0

Actual Avg. Monthly Cost (FY18$M) Predicted Avg. Monthly Cost Monthly Avg. Predicted

• Database

1. MESSENGER (PM 47.2) 5. Dawn (PM 21.0) 9.GRAIL (PM 17.7) 2. Mars Odyssey (PM 44.3) 6. MRO (PM 134.0) 10. Cassini (PM 337.7)** 3. LADEE (PM 49.6) 7. LRO (PM 97.1) 4. MAVEN (PM 67.3) 8. Juno (PM 170.8)

Additional Notes • Radio Science experiments generally are part of the S/C telecom system and do not need to be considered when totaling up the payload mass. Only include the mass of instruments for which NASA is performing operations. • **GRAIL was not included in the model due to using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions • ***Cassini was not included in the CER since it the only example of a flagship class mission currently available

13 Overall Model Performance

• The model was compared against actual mission costs to gauge performance • Standard Deviation for all missions was found to be 20% (v1.2 21%, v1.1: 21%, v1.0: 25%) • Planetary: 25% (v1.2 27%, v1.1: 27%, v1.0: 32%) • Earth/Near Earth Orbiting: 17% (v1.2 17%, v1.1 17%, v1.0: 22%) Variance from Actual Cost (Estimate-Actual/Actual %) v1.3 v1.3 Delta Act. Var. Est. Var. 80% Avg. $ 0.92 5% 1% STDEV $ 9.98 20% 19% 60% Median $ 0.77 2% 2% 40% Maximum $ 27.80 67% 40% 25% 20% Minimum $ (32.71) -39% -63% 20% 17% 0%

-20% -17% -25% -20% -40%

-60%

-80%

% Variance In Model %Variance Not In Model EO Std. Dev. PL Std. Dev. Overall Std. Dev. v1.2 Delta Act. Var. Est. Var. Avg. $ 1.55 6% 2% v1.2 Variance from Actual Cost (Estimate-Actual/Actual %) STDEV $ 8.82 21% 19% 80% Median $ 0.85 3% 3% Maximum $ 22.85 67% 40% 60% Minimum $ (26.60) -38% -61% 40% 27% 21% 20% 17% 0%

-20% -17% -27% -21% -40%

-60%

-80% % Variance In Model %Variance Not In Model EO Std. Dev. PL Std. Dev. Overall Std. Dev. 14 ONCE Downloads and Users

• To date MOCET has been downloaded from ONCE 142* times since the initial release • *Downloads include those from inactive users and duplicate downloads – ONCE output generally only shows active users and unique downloads • v1.0 30 downloads; v1.1 60 downloads; v1.2 34 downloads; v1.3 18 downloads as of June 2018 • As of June 2018, 76 unique users have downloaded MOCET from ONCE • Since the release of v1.0 the number of users has increased steadily

v 1.0 v 1.1 NCS 2016 v 1.2 NCS 2017 v 1.3 v 1.0 v 1.1 NCS 2016 v 1.2 NCS 2017 v 1.3 Cumulative Downloads Downloads by Month

160 18 17

140 16

120 14 12 100 12 11 11 10 10 80 9 8 60 7 7 7 6 6 5 5 40 4 4 4 4 3 3 3 20 2 2 2 2 2 1 1 1 1 1 1 0 0 0 0 0 0 0

15 Public Release on software.nasa.gov

• MOCET has been approved for public release via software.nasa.gov – The direct link can be found here: https://software.nasa.gov/software/LAR-18894-1

• Available to anyone* worldwide

• User must create an account on software.nasa.gov

• Fill out the request form

• *A signature authority must be provided for those representing businesses – Cannot be an engineer or those with similar titles – An authorized signature authority is considered • president of the company • general counsel • or contract officer

• If representing self as a private individual, an additional signature is not required

• Rules may be different for other user types, Academic etc.

• Contact MOCET help staff at [email protected] with any issues you may encounter 16 software.nasa.gov Downloads and Users

• There are MOCET 22 users on software.nasa.gov since its release there in April of 2017 • Most users are from Industry (8) and from NASA (7) • There are also 4 Representing Self, 2 Other Civil, and 1 Academic • Of the 22 users, 8 are also international coming from Africa, Australia, Europe, and South America • As of the end of June, 8 users had downloaded v1.3 • These statistics were only collected shortly after the release of v1.3

17 Summary & Next Steps

• New data has been added and the updated model has been released as MOCET v1.3

• MOCET v1.3 performs about the same as v1.2

• MOCET will continue to be periodically updated with new mission data

• Model is currently being used by both evaluators and proposers

• We will continue to engage and grow the user community

Download MOCET ONCE Model Portal https://oncedata.msfc.nasa.gov Available external to NASA via https://software.nasa.gov

For More Information Demo session (see agenda for time and room) Email: [email protected] 2017 IEEE Publication, Past NCSS Briefings, Future AIAA 2018 and IEEE 2019 Publications • M. R. Hayhurst et al., "Mission Operations Cost Estimation Tool (MOCET)," 2017 IEEE Aerospace Conference, Big Sky, MT, 2017, pp. 1-13 • Mission Operations Cost Estimation Tool (MOCET) FY17 Update, 2017 NASA Cost and Schedule Symposium, August 2017, Washington DC • Mission Operations Cost Estimation Tool (MOCET) Update, 2016 NASA Cost Symposium, August 2016, Cleveland OH • Mission Operations Cost Estimation Tool (MOCET), 2015 NASA Cost Symposium, August 2015, Mountain View CA