Mission Operations Cost Estimation Tool (MOCET) FY17 Update

Marc Hayhurst, Brian Wood, Shirin Eftekharzadeh The Aerospace Corporation

Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo Rodriguez NASA Langley Research Center Science Office for Mission Assessments (SOMA)

August 29th, 2017

• MOCET Overview & Status

• Updates in MOCET v1.2

• Public Release of MOCET

• User Community

• Summary & Future Work

2 MOCET Overview

• The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA Science Office for Mission Assessments (SOMA) – Based on actual costs of historical missions with emphasis on competed missions – Constructed by breaking the mission operations cost into the various phases – Has few subjective inputs – Estimates total Phase E mission cost – Implemented entirely in Excel and requires no additional software or tools – Also includes a user manual which provides additional instruction and background

Download from ONCE Model Portal https://oncedata.msfc.nasa.gov

Now available external to NASA via

Mars 2034 Rover https://software.nasa.gov Mission Schedule Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37 Cruise For more information: Approach/EDL Email: [email protected]

First Landed Month

Landed Prime Operations

M. R. Hayhurst et al., "Mission Operations Cost Estimation Tool (MOCET)," 2017 IEEE Aerospace Conference, Big Sky, MT, 2017, pp. 1-13.

3 MOCET Timeline

Presentation at NASA Cost MOCET v1.0 2015 Symposium release 2015

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NASA Cost MOCET v1.1 MOCET User’s Symposium MOCET User’s 2016 release Workshop 1 2016 v1.1 Demo Workshop 2

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

MOCET v1.2 Initial approval NASA Cost and release; for MOCET Schedule IEEE Conf. public release Symposium 2017 2017

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

4 MOCET Mission Types Database Overview

Mission/CER Type Program Missions

Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn (updated Orbital Ops) Mars Scout Phoenix Planetary Robotic Lunar Exploration LRO, LADEE New Frontiers New Horizons, Juno (Cruise, Orbit Insertion), OSIRIS-REx Mars Exploration MRO, Odyssey, MER, MSL, MAVEN (updated Orbital Ops) Outer Planets Flagship Cassini (Orbit Insertion) Earth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS Earth Science Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, Jason-3

Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI

Explorers Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI

Medium Explorers (MIDEX) THEMIS, Swift, WISE Discovery Genesis, Kepler Solar Terrestrial Probes (STP) STEREO, TIMED, MMS Near Earth Discovery Living With a Star (LWS) RBSP, SDO Helio-Astro Cosmic Origins Spitzer Physics of the Cosmos Fermi, Chandra Bold = New Data in version 1.2 Potential New Data for version 1.X Data Sources Utilized and Consulted – NASA SAP Business – Monthly Flight Project Reviews (FPR) – NASA Cost Analysis Data Warehouse and Monthly Status Reviews (MSR) Requirements (CADRe) • Monthly Expenditures • Monthly Expenditures, Mission Events • Project development costs used & Schedule to assign mission class 5 MOCET v1.2 CERs Updates Planetary CER Name New New Eq. No Comments Data Form Change Cruise CER X Minor update with complete Juno Cruise Phase Data Juno incorporated along with Cassini and “Outer Planets” Approach/Orbit Insertion/Reduction CER X input option Updated Dawn & MAVEN data incorporated. “IsMars?” Orbital Operations CER X variable determined to be no longer significant.

Flyby/Encounter CER X

Approach EDL CER X

First Landed Month CER X

Landed Prime Operations X

Data/Sample Analysis & Archiving X

Extended Orbital Ops CER X

Extended Landed Ops CER X

Earth/Near-Earth Orbiting

CER Name New New Eq. No Comments Data Form Change Earth/Near-Earth Orbiting Checkout CER X Earth Science Prime Operations CER X

Explorer Prime Operations CER X

Near Earth Discovery Helio Astro Prime Operations CER X

6 MOCET v1.2 Update

• MOCET CERs and tool were updated to include new Juno data and released as v1.2 – Additional data from prime missions accumulated since v1.1 were also included: Dawn, MAVEN – The same statistical processes used for v1.1 were applied to this update – The new v1.2 CERs have been benchmarked against v1.1 – Overall the performance of the v1.2 updates is about the same as v1.1

• This update affected three CERs – Nominal/Quiescent Cruise CER • Updated to include the complete Juno Cruise Phase • Resulted in only a minor change to the coefficients – Approach/Orbit Insertion/Reduction • The Juno Jupiter Orbit Insertion phase is now included in this CER • It was also possible to include the Cassini mission which resulted in better performance of the CER • Resulted in change in coefficients and the addition of a new variable – Orbital Operations • Only four months of Juno operations at Jupiter were available to include into this CER – At this time recommend waiting until more data is accumulated to include Juno in this CER • Rates for Dawn and MAVEN were updated with monthly data since the last update. • An update to the MAVEN payload mass input variable value was also performed per user feedback • Resulted in updates of coefficients and elimination of one variable due to new data and significance testing

7 Nominal/Quiescent Cruise Prime Phase CER • CER Function: Y 1.2537  0.8624* IL  0.7836* IQ Variable Name Units Definition Planetary mission average monthly prime nominal or quiescent cruise Y Cost FY13 $M phase cost IL IsLarge? 0 or 1 Is this mission a large mission class per MOCET definitions? IQ IsQuiescent? 0 or 1 Is this cruise period a quiescent mode?

• Goodness of Fit 2.5

Measure Value 2 Regression Method OLS Standard Error (RSS) 0.20 1.5

Average Percentage Bias 0.0% 1 Coefficient of Determination (R^2) 0.92 Predicted Adjusted R^2 0.87 0.5

Pearson's Correlation Sqd (r^2) 0.92 0 Number of Observations 6 $- $0.50 $1.00 $1.50 $2.00 $2.50 Number of Input Variables 2 Actual • Database

1. MESSENGER – Nominal (IL 0, IQ 0) 4. New Horizons – Nominal (IL 1, IQ 0) 7. GRAIL (A/B) – (IL 0, IQ 0)* 2. Dawn - Nominal (IL 0, IQ 0) 5. Stardust - Quiescent (IL 0, IQ 1)** 8. Cassini – (IL 0, IQ 0)*** 3. Juno - Nominal (IL 1, IQ 0) 6. New Horizons - Quiescent (IL 1, IQ 1)

Additional Notes • Some missions may have nominal, quiescent, and checkout periods of cruise. Each of these phases should be broken out and estimated separately (e.g. 4 months of cruise with checkout, 12 months of nominal cruise, 36 months of quiescent ops.) • *GRAIL was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions • **Stardust cruise considered as quiescent during the original prime mission. Limited data was available about the original mission so this assumption is based on description of the NEXT Mission. • ***Cassini was not included in the CER since it the only example of a flagship class mission currently available

8 Approach, Orbit Insertion, Reduction Prime Phase CER • CER Function: Y  0.5024  0.9478* PC 1.2668*OP Variable Name Units Definition Y Cost FY13 $M Planetary mission average monthly prime orbital operations phase cost PC Preceding Cruise FY13 $M Average monthly cost of the preceding cruise phase OP OuterPlanets? 0 or 1 Is the orbit insertion being done at one of the outer planets (beyond Mars)?

• Goodness of Fit 9 Measure Value 8 7 Regression Method OLS 6 Standard Error (RSS) 0.16 5 Average Percentage Bias 0.0% 4 Coefficient of Determination (R^2) 0.99 Predicted 3 2 Adjusted R^2 0.99 1 Pearson's Correlation Sqd (r^2) 0.99 0 Number of Observations 8 $- $2.00 $4.00 $6.00 $8.00 $10.00 Number of Input Variables 2 Actual • Database

1. MESSENGER – MOI (PC $1.53, OP 0) 4. MAVEN – MOI (PC $1.98, OP 0) 7. Juno – JOI (PC $2.03, OP 1) 2. Dawn – Vesta (PC $0.98, OP 0) 5. GRAIL – LOI (PC $1.60, OP 0) 6. Cassini – SOI (PC $6.85, OP 1) 3. Dawn – Ceres (PC $0.97, OP 0) 6. MRO – MOI (PC $5.03, OP 0)

Additional Notes • None

9 Orbital Operations Prime Phase CER • CER Function: Y 1.016  0.0157* PM Variable Name Units Definition Y Cost FY13 $M Planetary mission average monthly prime orbital operations phase cost PM Payload Mass* kg > 0 Mass of payload portion for which NASA will fund operations

4 • Goodness of Fit 3.5 3 Measure Value 2.5 Regression Method GERM ZMPE 2 Standard Error of the Estimate 11.0%

Predicted 1.5 Average Percentage Bias 0.0% 1 Pearson's Correlation Sqd (r^2) 0.86 0.5 Number of Observations 7 0 Number of Input Variables 1 $- $0.50 $1.00 $1.50 $2.00 $2.50 $3.00 $3.50 $4.00 Actual

• Database

1. MESSENGER (PM 47.2) 4. MAVEN (PM 67.3) 7. LRO (PM 97.1) 2. Mars Odyssey (PM 44.3) 5. Dawn (PM 21.0) 9.GRAIL (PM 17.7) 3. LADEE (PM 49.6) 6. MRO (PM 134.0) 10. Cassini (PM 337.7)**

Additional Notes • Radio Science experiments generally are part of the S/C telecom system and do not need to be considered when totaling up the payload mass. Only include the mass of instruments for which NASA is performing operations. • **GRAIL was not included in the model due to using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support estimation of planetary multiple spacecraft missions • ***Cassini was not included in the CER since it the only example of a flagship class mission currently available

10 Overall Model Performance • The model was compared against actual mission costs to gauge performance • Standard Deviation for all missions was found to be 20% (v1.1: 20%, v1.0: 25%) • Planetary: 26% (v1.1: 27%, v1.0: 32%) • Earth/Near Earth Orbiting: 16% (v1.0: 22%, No changes in v1.2)

v1.2

v1.1

Performance of v1.2 is about the same as v1.1

11 Public Release of MOCET

• MOCET has been approved for public release via software.nasa.gov – The direct link can be found here: https://software.nasa.gov/software/LAR-18894-1

• Now available to U.S. contractors and universities etc.

• User must create an account on software.nasa.gov

• Fill out the request form

• Signature authority for a business cannot be an engineer or those with similar titles – An authorized signature authority is considered • president of the company • general counsel • or contract officer

• Rules may be different depending on type of institution – Academic/University – Individual Contractor etc.

• Contact MOCET help staff at [email protected] with issues

12 Downloads vs. Time

• To date MOCET has been downloaded from ONCE 106* times since the initial release • *Includes all downloads from all users including users that may be currently inactive • Version 1.0 was downloaded 30 times between October 2015 and March 2016 • Version 1.1 has been downloaded 60 times from March 2016 to April 2017 • Version 1.2 has been downloaded 16 times from April 2017 to June 2017

Cumulative Downloads Downloads by Month 120 14 v 1.0 Release v 1.1 Release NCS 2016 v 1.2 Release v 1.0 Release v 1.1 Release NCS 2016 v 1.2 Release 12 12 100 11 10 10 10 80 9

8 60 7 7 7 6 6 40 5 4 4 4 20 3 3 3 2 2 1 1 0 0 0 0 0 0

13 Users vs. Time

• As of June 2017, 59 unique users have downloaded MOCET from ONCE • Since the release of v1.0 the number of users has increased steadily

Cumulative New Users New Users by Month 70 14 v 1.0 Release v 1.1 Release NCS 2016 v 1.2 Release v 1.0 Release v 1.1 Release NCS 2016 v 1.2 Release 12 60 12

50 10

40 8 30 6 6 5 5 20 4 4 4 3 3 3 3 3 10 2 2 2 2 1 1 0 0 0 0 0 0 0 0

14 Summary & Next Steps

• New data has been added and the updated model has been released as MOCET v1.2

• MOCET v1.2 performs about the same as v1.1

• MOCET has been released publicly via software.nasa.gov

• MOCET IEEE paper* has been published for further background on the model

• MOCET will continue to be periodically updated with new mission data

• Model is currently being used by both evaluators and proposers

• We will continue to engage and grow the user community

Download MOCET ONCE Model Portal https://oncedata.msfc.nasa.gov Now available external to NASA via https://software.nasa.gov

For More Information Demo session (see agenda for time and room) Email: [email protected]

*M. R. Hayhurst et al., "Mission Operations Cost Estimation Tool (MOCET)," 2017 IEEE Aerospace Conference, Big Sky, MT, 2017, pp. 1-13.