NASA Cost and Schedule Symposium August 13-15, 2019 • Houston, TX
System Integration Schedule Estimating Relationships (SERs)
Presented by: Marc Greenberg Strategic Investment Division (SID) National Aeronautics and Space Administration Outline
• Task Objectives • Background • Schedule Data from SMART • Methodology – Equation 1: Y = 0.3617 X -0.568 – Equations 2a, 2b, 3a and 3b – “Knee in the Curve” for Equation 1 • Example for notional project – Apply equations 2a, 2b, 3a and 3b – Sensitivity analysis • Conclusions and Future Work
Slide 2 Task Objectives a • Estimate “ideal” (Design Sched.)/(Sys. I&T Sched.) ratio
• Create schedule estimating relationships (SERs) to predict the following durations: – SRR (System Readiness Review) to CDR (Critical Design Review) – SIR (System Integration Review) to LRD (Launch Readiness Date) – SRR to LRD (i.e., total schedule less time from ATP b to SRR)
• Demonstrate how these SERs could be used in practice
• Discuss use of sensitivity analysis.
(a) Schedule data was readily available so no data collection effort was required. (b) ATP: Authority to Proceed Slide 3 Background: Milestones from SRR to LRD
Phase A-B Phase D-E
System Level SRR LRD System Level Design Requirements
Subsystems
Element
Item Level SIR Component Design Requirements PDR
CDR
Phase C All Design Requirements Complete
Major Milestones: SRR: Systems Readiness Review PDR: Preliminary Design Review CDR: Critical Design Review SIR: System Integration Review LRD: Launch Readiness Date Slide 4 Schedule Data from SMART NASA’s Schedule Management and Relationship Tool (SMART) provided schedule data from 55 science missions. Values in Months. Mission SRR-PDR PDR-CDR CDR-SIR SIR-LRD SRR-LRD Mission SRR-PDR PDR-CDR CDR-SIR SIR-LRD SRR-LRD CONTOUR 8 11 14 5 38 SWAS 31 10 10 52 103 Dawn 6 8 7 33 54 WIRE 23 0 17 18 58 Deep Impact 10 11 16 20 57 WISE 7 23 17 13 60 Deep Space 1 3 9 11 15 38 NuSTAR 11 8 12 17 48 Galileo 9 33 15 82 139 OCO 12 25 20 11 68 InSight 28 9 9 39 85 ACE 12 12 19 16 59 Juno 30 11 12 16 69 ACTS 10 24 37 27 98 MAP 8 5 35 14 62 AIM 8 10 0 30 48 MAVEN 11 12 12 17 52 Aqua 5 14 1 47 67 MER-A 3 14 3 16 36 Aura 5 14 13 34 66 MER-B 3 14 4 15 36 Calipso 8 18 11 26 63 MESSENGER 13 10 10 19 52 CHIPSAT 12 8 14 8 42 MRO 6 10 11 16 43 DART 5 4 7 25 41 MSL 6 12 9 46 73 EO-1 8 10 5 21 44 OSIRIS-REX 10 13 11 19 53 FAST 14 10 15 33 72 Phoenix 11 8 5 16 40 Glory 12 11 19 38 80 Stardust 6 9 7 13 35 IMAGE 9 5 20 12 46 Chandra X-Ray 23 15 20 22 80 IRIS 3 7 18 13 41 COBE 15 25 35 15 90 LANDSAT-7 12 13 16 27 68 FGST (GLAST) 33 15 23 22 93 LANDSAT-8 14 10 18 14 56 GALEX 3 8 26 20 57 NPP 35 9 62 36 142 IBEX 4 8 13 13 38 OSTM/Jason-2 10 44 7 12 73 Kepler 12 24 19 10 65 Aquarius 12 36 11 25 84 RHESSI 7 4 13 26 50 MMS 20 13 25 31 89 SAMPEX 3 10 14 11 38 SMAP 22 9 9 22 62 SDO 11 13 0 59 83 TRACE 2 1 16 13 32 Spitzer 7 12 41 19 79 VABP (RBSP) 8 14 11 23 56 STEREO 19 14 18 26 77 Slide 5 Methodology: Equation 1: Y = 0.3617 X -0.568
Relative Sys I&T Schedule vs. "Design-to-Sys I&T" Schedule Ratio (SIR to LRD / SRR to LRD) versus (SRR to CDR / SIR to LRD)
0.90 As (SRR to CDR) / (SIR to LRD) increases, (SIR to LRD) / (SRR to LRD) decreases.
0.80 i.e., As relative design schedule increases, relative sys I&T schedule decreases.
0.70 Y: (SIR to CDR) ~ Sys I&T vs. Total Sched. 0.60 (SRR to LRD) Potentially insufficient design schedule X: (SRR to CDR) ~ Design vs. Sys I&T 0.50 (SIR to LRD)
0.40
0.30 Y = 0.3617 X -0.568 2 0.20 R = 0.79 (MUPE)
0.10 Sys I&T Schedule as % of Project Schedule = (SIR to LRD LRD / LRD) (SRR ) to Schedule (SIR to = as %ScheduleI&T of Project Sys 0.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Design Schedule relative to Sys I&T Schedule = (SRR to CDR) / (SIR to LRD) Slide 6 Methodology: Equations 2a, 2b, 3a and 3b
(SRR to CDR) Note: Schedule data in Months; Sample size = 55 (SIR to LRD)
Eq. 2a: SRR to CDR = 0.3060 * SRR to LRD 1.041 Design vs Sys I&T 0.4306 Adj-R2 (MUPE) = 0.86; Avg. = 24.5 mo. SE = 4.3 mo.; MAD = 13.4%
Rearrange Equation 2a to create Equation 2b … Eq. 2b: SRR to LRD = 3.1196 * SRR to CDR 0.9607 Design vs Sys I&T -0.4137
Eq. 3a: SIR to LRD = 0.3060 * SRR to LRD 1.041 Design vs Sys I&T -0.5694 Adj-R2 (MUPE) = 0.91; Avg. = 23.4 mo. SE = 4.5 mo.; MAD = 13.4%
Rearrange Equation 3a to create Equation 3b … Eq. 3b: SRR to LRD = 3.1196 * SIR to LRD 0.9607 Design vs Sys I&T 0.5470 Q: How can we use each SER if we must know not one but up to three independent variables? A: Simplify “Design vs Sys I&T” input! Slide 7 Methodology: “Knee in the Curve” for Eq. 1
Relative Sys I&T Schedule vs. "Design-to-Sys I&T" Schedule Ratio (SIR to LRD / SRR to LRD) versus (SRR to CDR / SIR to LRD)
0.90 1. Connect endpoints of Y = 0.3617 X -0.568
0.80 2. Calculate Slope of straight line = -0.159
0.70 Knee in the curve: Point on curve where distance between line & curve is greatest. Must find the 0.60 point tangent to curve parallel to straight line.
0.50 3. Derivative of Y = 0.3617 X -0.568 -1.568 0.40 = Straight line = -0.2055 X
0.30
0.20
0.10 4. Given Y’ = -0.2055 X -1.568 … find X where slope = -0.159 Sys I&T Schedule as % of Project Schedule = (SIR to LRD LRD / LRD) (SRR ) to Schedule (SIR to = as %ScheduleI&T of Project Sys 0.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 X= 1.175Design Schedule relative to Sys I&T Schedule = (SRR to CDR) / (SIR to LRD) Slide 8 Methodology: “Knee in the Curve” for Eq. 1
Calculation of "Knee in the Curve" for: Relative Sys I&T Schedule vs. "Design-to-Sys I&T" Schedule Ratio 0.90 Knee in the curve: That point on the curve
0.80 where distance between the “connector” line and curve is greatest …
0.70 … that point on the curve is (1.175, 0.330) X = “Design vs Sys I&T” = 1.175
0.60 “Design vs. Sys I&T” = (SRR to CDR) (SIR to LRD)
0.50 For all SERs, we assume an “ideal” 0.40 Design vs Sys I&T Y = 0.330 input = 1.175 0.30
0.20
X = 1.175 y = 0.3617x-0.568 0.10 Sys I&T Schedule as % of Project Schedule = (SIR to LRD) LRD) LRD) (SRR / to Scheduleto (SIR = % as ScheduleI&Tof Project Sys
0.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Design Schedule relative to Sys I&T Schedule = (SRR to CDR) / (SIR to LRD) Slide 9 Example for Notional Project: Equation 2a
Eq. 2a: SRR to CDR = 0.3060 * SRR to LRD 1.041 Design vs Sys I&T 0.4306 • Input 1: SRR to LRD = 60 mos. (e.g., constraint for planetary mission “M”) • Input 2: Design vs Sys I&T = 1.175 (“knee-in-the-curve” value) SRR to CDR = 0.3060 * 60 1.041 1.175 0.4306 = 23.3 mos. 1.175 = SRR to CDR so … SIR to LRD = 23.3 = 19.8 mos. SIR to LRD 1.175 CDR to SIR = (SRR to LRD) – (SRR to CDR) – (SIR to LRD) = 16.9 mos. 60 23.3 19.8
SRR to LRD (Given) = 60 months We can now rearrange
SRR to CDR = 23.3 CDR to SIR = 16.9 SIR to LRD = 19.8 Eq. 2a to form Eq. 2b 39% 28% 33% where Y = SRR to LRD. Estimates of "SRR to CDR", "CDR to SIR" and "SIR to LRD" (in months)
0 10 20 30 40 50 60
Slide 10 Example for Notional Project: Equation 2b
Eq. 2b: SRR to LRD = 3.1196 * SRR to CDR 0.9607 Design vs Sys I&T -0.4137 • Input 1: SRR to CDR = 23.3 mos. (e.g., in-family for planetary mission “M”) • Input 2: Design vs Sys I&T = 1.175 (“knee-in-the-curve” value) SRR to LRD = 3.1196 * 23.3 0.9607 1.175 -0.4137 = 60.0 mos. 1.175 = SRR to CDR so … SIR to LRD = 23.3 = 19.8 mos. SIR to LRD 1.175 CDR to SIR = (SRR to LRD) – (SRR to CDR) – (SIR to LRD) = 16.9 mos. 60 23.3 19.8
SRR to LRD (estim) = 60
(Given) SRR to CDR = 23.3 CDR to SIR = 16.9 SIR to LRD = 19.8 39% 28% 33%
Estimates of "CDR to SIR" , "SIR to LRD" and "SRR to LRD" (in months)
0 10 20 30 40 50 60 Slide 11 Example for Notional Project: Equation 3a
Eq. 3a: SIR to LRD = 0.3060 * SRR to LRD 1.041 Design vs Sys I&T -0.5694 • Input 1: SRR to LRD = 60 mos. (e.g., constraint for planetary mission “M”) • Input 2: Design vs Sys I&T = 1.175 (“knee-in-the-curve” value) SIR to LRD = 0.3060 * 60 1.041 1.175 -0.5694 = 19.8 mos. 1.175 = SRR to CDR so … SRR to CDR = 1.175 * 19.8 = 23.3 mos. SIR to LRD CDR to SIR = (SRR to LRD) – (SRR to CDR) – (SIR to LRD) = 16.9 mos. 60 23.3 19.8
SRR to LRD (Given) = 60 months We can now rearrange
SRR to CDR = 23.3 CDR to SIR = 16.9 SIR to LRD = 19.8 Eq. 3a to form Eq. 3b 39% 28% 33% where Y = SRR to LRD. Estimates of "SRR to CDR", "CDR to SIR" and "SIR to LRD" (in months)
0 10 20 30 40 50 60
Slide 12 Example for Notional Project: Equation 3b
Eq. 3b: SRR to LRD = 3.1196 * SIR to LRD 0.9607 Design vs Sys I&T 0.5470 • Input 1: SIR to LRD = 19.8 mos. (e.g., in-family for planetary mission “M”) • Input 2: Design vs Sys I&T = 1.175 (“knee-in-the-curve” value) SRR to LRD = 3.1196 * 19.8 0.9607 1.175 0.5470 = 60.0 mos. 1.175 = SRR to CDR so … SIR to LRD = 23.3 = 19.8 mos. SIR to LRD 1.175 CDR to SIR = (SRR to LRD) – (SRR to CDR) – (SIR to LRD) = 16.9 mos. 60 23.3 19.8
SRR to LRD (estim) = 60
(Given) SRR to CDR = 23.3 CDR to SIR = 16.9 SIR to LRD = 19.8 39% 28% 33%
Estimates of "SRR to CDR" , "CDR to SIR" and "SRR to LRD" (in months)
0 10 20 30 40 50 60 Slide 13 Example for Notional Project: Sensitivity Analysis
It’s important to explore how changing SER inputs can impact schedule estimates …
Equation 2a: Estimate SRR to CDR Input 1: SRR to LRD (months) 23.3 40 45 50 55 60 65 70 75 80 1.175 1.025 14.4 16.3 18.1 20.0 21.9 23.8 25.8 27.7 29.6 60 1.100 14.8 16.8 18.7 20.7 22.6 24.6 26.6 28.5 30.5 1.175 15.3 17.2 19.2 21.3 23.3 25.3 27.3 29.4 31.4 1.250 15.7 17.7 19.8 21.8 23.9 26.0 28.1 30.1 32.2 1.325 16.1 18.2 20.3 22.4 24.5 26.6 28.8 30.9 33.1 Input 2: Design vs Sys I&T Equation 2b: Estimate SRR to LRD Input 1: SRR to CDR (months) 60.0 9 512 1 18 23.3 27 30 33 36 1.175 1.025 25.5 33.6 41.6 49.6 63.5 73.2 81.0 88.8 96.6 23.3 1.100 24.8 32.6 40.4 48.2 61.7 71.1 78.7 86.3 93.8 1.175 24.1 31.8 39.4 46.9 60.0 69.2 76.6 83.9 91.2 1.250 23.5 31.0 38.4 45.7 58.5 67.5 74.7 81.8 88.9 1.325 22.9 30.2 37.4 44.6 57.1 65.9 72.9 79.9 86.8 Input 2: Design vs Sys I&T Example: Given SRR to CDR = 23.3 mos., if Design v Sys I&T changed from 1.175 to 1.025, then SRR to LRD would increase to 63.5 mos.
Slide 14 Example for Notional Project: Sensitivity Analysis
It’s important to explore how changing SER inputs can impact schedule estimates …
Equation 3a: Estimate SIR to LRD Input 1: SRR to LRD (months) 19.8 40 45 50 55 60 65 70 75 80 1.175 1.025 14.0 15.9 17.7 19.6 21.4 23.3 25.1 27.0 28.9 60 1.100 13.5 15.2 17.0 18.8 20.6 22.3 24.1 25.9 27.7 1.175 13.0 14.7 16.4 18.1 19.8 21.5 23.2 25.0 26.7 1.250 12.5 14.2 15.8 17.5 19.1 20.8 22.4 24.1 25.8 1.325 12.1 13.7 15.3 16.9 18.5 20.1 21.7 23.3 25.0 Input 2: Design vs Sys I&T Equation 3b: Estimate SRR to LRD Input 1: SIR to LRD (months) 60.0 6 9 12 18 19.8 24 27 30 33 1.175 1.025 17.7 26.1 34.4 50.8 55.7 67.0 75.0 83.0 90.9 19.80 1.100 18.4 27.1 35.8 52.8 57.9 69.6 78.0 86.3 94.5 1.175 19.1 28.1 37.1 54.7 60.0 72.2 80.8 89.4 98.0 1.250 19.7 29.1 38.4 56.6 62.1 74.7 83.6 92.5 101.4 1.325 20.3 30.0 39.6 58.5 64.1 77.1 86.3 95.5 104.7 Input 2: Design vs Sys I&T Example: If SRR to LRD increased to 70 mos. and if Design v Sys I&T decreased to 1.025, then SIR to LRD increases from 19.8 to 25.1 mos. Slide 15 Conclusions & Future Work • Regression analysis of schedule data produced five SERs. • Assumed a regime on the curve where an “ideal” amount of design schedule leads to an “ideal” system I&T schedule. • Use of these SERs can assist estimators in bounding schedule durations from key milestone to key milestone. – Example: Is the program’s SRR to CDR estimate in-family? • Future work includes the following tasks: – Revisit data. Use outlier analysis to learn more on unusual obs. – Create approach to deviate from knee-in-the-curve (X = 1.175) – Add pred. intervals to enable showing Y-values at any probability – Where possible, add data where data is missing in SMART data. – Where data is sufficient, conduct regressions to estimate … • ATP to SRR, SRR to PDR, PDR to CDR and ATP to LRD. Slide 16