Cost Estimation of Electronic Parts in NASA Space Missions 2019 NASA Cost & Schedule Symposium Johnson Space Center, Houston, TX Meagan Hahn & Rachel Sholder Johns Hopkins Applied Physics Lab Parametric Cost Analyst Agenda
• Background/Hypothesis • Research Methodology • Statistical Analysis and Findings • Conclusions • Recommendations for Future Research
Cost Estimation of EEE Parts 14 August 2019 2 Background/Hypothesis • Working group created to improve initial cost estimates of Electrical, Electronic, and Electromechanical (EEE) parts - Estimates created by EEE parts group—particularly for certified proposals—require a Bill of Materials (BOM) - Accurate BOM (and therefore an accurate estimate) is not feasible during early design phase - Flawed estimating process results in major cost growth in EEE parts as design is refined (in some cases exceeding 100%) ° True for spacecraft and instruments • Major goal of the working group was to Develop a CER to be used in early design phase - Eliminate reliance on “guesstimated” BOM - Identify high-level parameters that can be defined and quantified Pre-Phase A through Phase B - Quantify uncertainty of initial estimates to inform program managers (PMs) of potential cost risk for adequate reserve allocation • We hypothesize that there are technical and programmatic variables that influence EEE costs
Cost Estimation of EEE Parts 14 August 2019 3 Methodology: Data Collection & Normalization
• Multi-disciplinary team identified many potential cost drivers based on experience across various hardware builds (see slide 6) • Technical data was collected from technical leads for each identified instrument and spacecraft box - Effort to ensure consistent assumptions across inputs; e.g. defining radiation environment, quantifying board complexity - Cost data was collected via our internal cost database - All costs inflated to $FY19 using NASA New Start Inflation Index • Total of 30 instruments and SC boxes (15 each) in analysis where data was available for all of the identified variables - Due to limited data availability, analysis includes data points for hardware that has not yet flown ° Assumes reasonable EAC ° Identified weakness in current analysis ° Cost growth in current projects in development (out of family with historical costs) argues they should be included in the analysis to help inform future projects
Cost Estimation of EEE Parts 14 August 2019 4 Methodology: Datasets
• Data separated into multiple groups for regression analyses - Consensus amongst technical leads that there are likely different cost drivers for instrument EEE parts and spacecraft EEE parts - May allow for more targeted and accurate CERs - Instruments: n = 15 - SC Boxes: n = 15 - Instruments & SC Boxes: n = 30 Instruments SC Boxes LORRI Van Allen Probes PDU MDIS Parker Solar Probe PDU JEDI DART PDU RBSPICE Van Allen Probes PSE LUCY LORRI Parker Solar Probe PSE DRACO DART PSE EIS Van Allen Probes Radio PIMS Parker Solar Probe Radio MISE DART Radio Psyche GRNS Europa Clipper Radio WISPR DPU EMM Radio EPI-Lo Europa Clipper PME GRAIL Europa Clipper TPE SKA Europa Clipper Rad Mon MESSENGER DPU EGNS
Cost Estimation of EEE Parts 14 August 2019 5 Methodology: Initial Key Variables
• Identified 33 total potential variables - Instruments: 32 - SC Boxes: 21 - Instruments & SC Boxes: 20 - Data collected for all initial key variables
Dependent Potential Independent Variables Variable Programmatic Technical Cost of EEE parts Launch Year Radiation Custom Magnetics Mission Class Low-Dose Testing Required New ASIC Parts Class FPGA Count Standard ASIC External Partners Memory Count Connectors with mech qual (I) In-House Mission Builds Unique Boards (SC) Harness Development Schedule (I) Board Count EM Count EVMS required (I) Board Area FM Count Sponsor (I) Complexity FS Count Planetary Protection required (I) Density Factor (I) Reuse/Heritage Optics (I) Non-standard parts (I) Hybrid Count (I) Contamination (I) PEMS (I) Mechanisms/Deployables (I) (I) = Instrument only variable (SC) = SC only variable
Cost Estimation of EEE Parts 14 August 2019 6 Variable Consolidation
• A lot of variables for a small data set - Identified variables exceed degrees of freedom, especially when separating instruments from spacecraft • Correlation and regression analysis demonstrated that several variables could be consolidated - Consolidated variables also easier to identify earlier in the design phase when this CER will be helpful
IC Count= FPGA Count+ New ASIC Count+ Standard ASIC Count EMFMFS = EM Count+ FM Count+ FS Count FMFS = FM Count+ FS Count
V Cost Estimation of EEE Parts 14 August 2019 7 Methodology: Statistical Analysis
• Simple linear regressions run as diagnostics to identify potential cost- drivers and relationships - Useful to help remove variables that are redundant/dependent • Multivariate regressions to identify significant cost drivers of EEE parts cost (discussed starting on slide 11)
Cost Estimation of EEE Parts 14 August 2019 8 Single-Variable Regression: Board Area of Instruments
Instruments: Board Area vs Cost of EEE Parts 6000000
y = 1001.3x + 618030 5000000 R² = 0.5679
4000000
3000000
2000000
1000000 Cost ofCost(FY$19) EEEParts
0 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Board Area • Board Area indicates relatively strong relationship to cost of EEE parts • Fairly robust R-squared at 57%
Cost Estimation of EEE Parts 14 August 2019 9 Single-Variable Regression: Launch Year of Instruments
Instruments: Launch Year vs Cost of EEE Parts 5000000 4500000 4000000 3500000 3000000 y = 140962x - 3E+08 2500000 R² = 0.2927 2000000 1500000 1000000 500000 Cost ofCost(FY$19) EEEParts 0 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024 Launch Year
• Predicted we would see a strong correlation with launch year (cost growth) • Moderate/low R-squared of 29% • Example of how anecdotal cost drivers are not always true cost drives; launch year is also not significant in the multi-variate regression analyses
Cost Estimation of EEE Parts 14 August 2019 10 Multivariate Regression Analysis • Ordinary Least Squares (OLS) • P-value < 0.05 to reject the null hypothesis • Analysis of multicollinearity, heteroscedasticity, and cross- Instruments SC Boxes Both Parts Class Mission Class Launch Year validation to: External Partners Parts Class Mission Class - Reduce number of overly correlated Memory Count Radiation Parts Class predictor variables Board Area Low Dose Rad Low Dose Rad - Guarantee that the error terms are Complexity Total Boards Hybird Count EMFMFS Count Harness In-House Mission Builds unbiased (no pattern) FMFS Count Hybird Count - Ensure overfitting isn’t an issue due In-House Mission Builds to the high number of predictor variables compared to the number of observations
The next few slides will demonstrate the regression analysis performed for Instruments only. Similar methods were used for the anlaysis of SC boxes and the analysis of Instruments & SC boxes.
Cost Estimation of EEE Parts 14 August 2019 11 6-Variable Regression Results
• Very high R-squared • Significant F-value for full model • 4 out of 6 variables are statistically significant
Regression Statistics Multiple R 0.95600794 R Square 0.913951182 Adjusted R Square 0.849414568 Standard Error 575582.0503 Observations 15
ANOVA df SS MS F Significance F Regression 6 2.81503E+13 4.69171E+12 14.16175 0.000713213 Residual 8 2.65036E+12 3.31295E+11 Total 14 3.08006E+13
Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0% Intercept 672993.9932 1031273.574 0.652585318 0.532324 -1705127.133 3051115.119 -1705127.133 3051115.119 Parts Class -629289.254 227125.523 -2.770667275 0.024272 -1153041.649 -105536.8587 -1153041.649 -105536.8587 External Partners -262368.8118 459955.5942 -0.570422048 0.584053 -1323028.314 798290.6904 -1323028.314 798290.6904 Memory Count -56772.5947 37317.7716 -1.52132864 0.166671 -142827.5303 29282.34093 -142827.5303 29282.34093 Board Area 929.8238255 278.4765166 3.338966736 0.010247 287.6558267 1571.991824 287.6558267 1571.991824 Complexity 293447.1376 120923.0307 2.426726619 0.041413 14598.12866 572296.1465 14598.12866 572296.1465 FMFS Count 77973.41104 26714.66036 2.918749854 0.019328 16369.29378 139577.5283 16369.29378 139577.5283
Cost Estimation of EEE Parts 14 August 2019 12 Correlation Analysis
Parts Class External Partners Memory Count Board Area Complexity FMFS Count Parts Class 1 External Partners -0.616793955 1 Memory Count -0.575617681 0.373296578 1 Board Area -0.401378124 0.328523124 0.849286543 1 Complexity -0.566770721 0.679185466 0.459010066 0.413581146 1 FMFS Count 0.119608562 0.158661656 0.003254671 0.151622414 0.03204648 1
• Want to eliminate high predictor variable correlations to decrease multicollinearity • External partners and complexity removed due to high correlations and less significance in the simple linear regressions
Parts Class Board Area Complexity FMFS Count Parts Class 1 Board Area -0.401378124 1 Complexity -0.566770721 0.413581146 1 FMFS Count 0.119608562 0.151622414 0.03204648 1
Cost Estimation of EEE Parts 14 August 2019 13 Adjusted 4-Variable Regression • High R-squared • Significant F-value for full model • All variables statistically significant
Regression Statistics Multiple R 0.94176 R Square 0.886913 Adjusted R Square 0.841678 Standard Error 590182.9 Observations 15
ANOVA df SS MS F Significance F Regression 4 2.73175E+13 6.82937E+12 19.60682 0.00010052 Residual 10 3.48316E+12 3.48316E+11 Total 14 3.08006E+13
Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0%Upper 95.0% Intercept 222741.7 1016156.874 0.21920016 0.830903 -2041396.86 2486880 -2041396.86 2486880 Parts Class -448583 191874.7382 -2.337894599 0.041488 -876106.473 -21059.4 -876106.473 -21059.4 Board Area 587.0483 162.3701687 3.615493433 0.004725 225.264997 948.8316 225.264997 948.8316 Complexity 258822.8 107266.0613 2.412905033 0.036502 19819.1405 497826.5 19819.1405 497826.5 FMFS Count 80246.67 26010.75416 3.085134229 0.011539 22291.0961 138202.2 22291.0961 138202.2
Cost Estimation of EEE Parts 14 August 2019 14 Is 4 variables too many?
Parts Class Board Area Complexity FMFS Count Parts Class 1 Board Area -0.401378124 1 Complexity -0.566770721 0.413581146 1 FMFS Count 0.119608562 0.151622414 0.03204648 1
• Correlation between parts class and complexity is moderately high - Multicollinearity ruled out with variance inflation factor (VIF) • Checked for overfitting - Ran 15 regressions (each with one observation removed from dataset) for both the 4 variable model and a 2 variable model (next slide) - How well does model predict the missing observation? • Overfitting could not be ruled out with confidence - Multiple cross-validation diagnostic tests were run - The predictive R-squared indicated the 4 variable model was better - The mean predicted error indicated the 2 variable model was better - To maintain conservatism, the 2 variable model is preferred because of the limited dataset
Cost Estimation of EEE Parts 14 August 2019 15 Final Model
• Moderately robust R-squared • Significant F-value for full model • All variables statistically significant
Regression Statistics Multiple R 0.86333737 R Square 0.74535142 Adjusted R Square 0.70290999 Standard Error 808462.324 Observations 15
ANOVA df SS MS F Significance F Regression 2 2.3E+13 1.15E+13 17.56188 0.00027268 Residual 12 7.84E+12 6.54E+11 Total 14 3.08E+13
CoefficientsStandard Error t Stat P-value Lower 95% Upper 95% Lower 95.0%Upper 95.0% Intercept -1397976.28 754885.8 -1.8519 0.088789 -3042731.11 246778.5 -3042731.11 246778.5 Board Area 747.077847 212.5955 3.514081 0.00427 283.872044 1210.284 283.872044 1210.284 Complexity 371650.527 128524.1 2.89168 0.013534 91620.57 651680.5 91620.57 651680.5
Cost Estimation of EEE Parts 14 August 2019 16 Test for Heteroscedasticity
Regression Residuals as a Function of EEE Cost 2000000
1500000
1000000
500000 Residuals 0 Linear (Residuals)
Residuals 0 1000000 2000000 3000000 4000000 5000000 6000000
-500000
-1000000
-1500000 Predicted Cost • No quantitative pattern to regression residuals (trendline on the x-axis) • Errors are uncorrelated and distributed normally (constant variance) • ‰ Thus, the model is homoscedastic
Cost Estimation of EEE Parts 14 August 2019 17 Error Range CER Prediction vs EEE Cost Actuals 6000000
5000000
4000000
3000000
2000000
CER Prediction (FY$19 PredictionCER 1000000
0 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000 5000000 EEE Actuals (FY$19)
• Average prediction error 38% • Predictor error standard deviation of 1.089 • While not trivial, this error range is lower than the cost growth currently being experienced between initial estimates and final cost. It allows the project to plan accordingly and account for potential cost growth/risk.
Cost Estimation of EEE Parts 14 August 2019 18 Regression Statistics Summary
Instruments SC Boxes Instruments & SC Boxes Adjusted A-Squared 0.703 Adjusted A-Squared 0.648 Adjusted A-Squared 0.682 F-Statistic 0.000273 F-Statistic 0.000752 F-Statistic 2.99E-07
Signficant Variables P-value Signficant Variables P-value Signficant Variables P-value Board Area 0.00427 Parts Class 0.007717 Parts Class 2.08E-05 Complexity 0.013534 Hybrid Count 0.016911 Complexity 0.001243
I Hybrid Count 0.000559
Cost of EEE parts of Instruments (FY$19) ==747(Board Area)+ 371651(Complexity) - 1397976
Cost of EEE parts of SC Boxes (FY$19) ==2892965 - 602361 (Parts Class)+ 239920 (Hybrjd Count)
Cost of EEE parts of Instruments &SC Boxes (FY$19) ==1316666 - 686852 (Parts Class)+ 264041 (Complexjty) + 25084 7 (Hybrjd Count)
V Cost Estimation of EEE Parts 14 August 2019 19 Conclusions/Observations • For both instruments and SC boxes, we have identified several key variables that may help predict EEE parts costs - There are a lot of statistically significant variables in the simple linear regressions compared to the multiple linear regressions • On average, these CERs are accounting for ~68% of total EEE parts costs with a ~37% error range - While not perfect, this is a starting point for early design phases - The error bounds allows us to quantify risk and allocate reserves accordingly • Limited data set and inconsistency in cost tracking across projects for EEE parts may be inhibiting identification of statistically significant predictor variables • While not all variables are statistically significant (in this analysis), the full set provides a list for technical leads to consider when generating a BUE
Cost Estimation of EEE Parts 14 August 2019 20 Suggestions for Future Research • Some diagnostic tests revealed a distinction between optical and particle instruments but there was no statistically significant relationship with the multivariate regression - Need more data in order to articulate a statistically significant difference in cost and cost drivers with different instrument types • Identification of other variables that may impact EEE costs - Ex: data processing/rate • CERs to predict parts associated labor - Current analysis focused on predicting hardware costs - Some variables are very labor driven (radiation) • Need more data (especially from flown instruments & SC boxes) - Will allow outliers to be tossed - Will allow for more predictive variables
Cost Estimation of EEE Parts 14 August 2019 21 . Back Up
Independent Variables Variable Quantification Definition Launch Year year what year did the mission launch? Mission Class A-D (1 to 4 scale) mission class risk Parts Class 1 to 3 with +/- (1 to 5 scale) the lower the grade, the more screening/processing (higher cost) Radiation low, med, high krad (1 to 3 scale) radiation tolerance prior to EEE parts degredation Low-Dose Testing Required Y/N was there a low dose testing requirement? External Partners Y/N ex: JPL FPGA Count Qty field programmable gate array Memory Count Qty SRAM, PROM, EEPROM Board Count Qty total board count Unique Boards Qty total unqiue designs Board Area cm^2 per side Complexity 1 to 10 scale of design and requirements Density Factor % how populated is board Non-standard parts (I) Qty parts that required purchase instructions PEMS (I) Qty plastic encapsulated microcircuits Custom Magnetics Qty transformers/inductors wound from ferrite cores per cusom specifications New ASIC Y/N design uses new APL application specific integrated circuit Standard ASIC Y/N design uses existing APL application specific integrated circuit Connectors with mech qual (I) Qty qualifying a new connector type that has never flown before Harness Y/N wiring EM Count Qty engineering model FM Count Qty flight model FS Count Qty flight spare Reuse/Heritage % rough estimate of design reuse Hybrid Count Qty collection of components packaged together with bond wires in microelectronics lab In-House Mission Builds Y/N was the box built in house? Mechanisms/Deployables Qty actuators, mag boom, etc. Development Schedule Qty B-D months EVMS required Y/N earned value management Sponsor Dummy variable NASA, JPL, GSFC Planetary Protection required Y/N different requirements depending on flyby, orbiter, rover, lander, etc. Optics Y/N are there optical elements? Contamination Y/N was there a contamination control requirement?
Cost Estimation of EEE Parts 14 August 2019 23