NASA Constellation Confidence Level Estimate Using ACEIT

Kelley Cyr Constellation Program Cost Estimating Team Lead

Jan. 16, 2008 2nd Annual ACEIT User Conference Topics

♦ Background • Why do a confidence assessment • Cx Confidence assessment team • Scope of Cx Confidence assessment ♦ Process • Process Overview • Model Structure • Challenges in integrating results • Handling Discrete Risks • Correlation ♦ Results ♦ Future Work

Jan. 16, 2008 2 Purpose

♦ Perform a Cost Risk Analysis on Constellation Program Cost Estimate ♦ Fulfill Congressional/OMB request for Confidence Level Estimate of Constellation Program ♦ Fulfill Administrator’s requirement for 65% confidence level ♦ Support PMR Review with ESMD at NASA HQ ♦ Establish an automated cost model with integrated risk analysis for Constellation Program ♦ Complete an initial Schedule Risk Analysis on Constellation Program ♦ Provide direction to Management indicating individual cost threats ♦ Provide direction to Management regarding phasing of program funding ♦ Ultimately ensure the mission success of Constellation by… • Provide high quality analysis to Management for use in decision making • Provide insight into cost and schedule of Constellation Projects • Measure and Manage cost threats to Projects and Program

Jan. 16, 2008 3 Historical NASA Cost Growth

100% y 90%

80%

70% Historical Data (1985-2005) 60% Historical Data (1990-2005) Historical Data (Completed Only) 50% Data Source: “ A Budgetary Analysis of NASA’s New Vision for Space Exploration,” Congressional Budget Office (CBO), Sept. 2004 40%

30%

20%

10% Space Station (86%) Apollo (64%) Mercury (92%) Gemini (143%) Confidence LevelConfidence / Cumulative Probabilit 0% -40 -20 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 Cost Growth Percentage

Jan. 16, 2008 4 NASA Policy

♦ NASA Cost Estimating Handbook 2004 • As a general rule, cost estimates at NASA should be presented at the 70% confidence level • As an entire portfolio of Projects, the budget should be presented at the 80% confidence level ♦ 7120.5D • “The life-cycle cost estimate, includes reserves, along with the level of confidence estimate provided by the reserves based on a cost-risk analysis” (Ch. 4, page 22) ♦ Administrator/SMC Policy • Projects budget at the 70% confidence level ♦ Administrator’s Position • The confidence level is selectable by managers; we’ve chosen 70% for agency projects in general, and I personally adjusted that to 65% for Cx.

Jan. 16, 2008 5 History of Constellation Cost Risk

♦ For ESAS, High Level Cost Risk Analysis Based on NASA History Used to Recommend Final Cost Reserve Levels • 65% confidence level for estimates through 2011 budget horizon totaling $31.3B ♦ For PMR 06, Cx Level II PP&C Office Conducted a Cost Risk Assessment on the Major WBS Elements for each Project • Projects provided input, Level II team incorporated data into a Risk model • Risk assessment focused on Cx program scope that was to support ISS missions (6 missions through 2015) • Confidence level of budget estimated at 50%, as well as additional funds required to achieve 65% ♦ For PMR 07, Cx Level II PP&C Office Integrated Detailed Cost Risk Analysis from Projects • Risk analysis conducted by projects and provided to Level II • Level II integrated cost risk assessments to assess overall program cost risk • Risk assessment focused on ISS IOC; also evaluated Operations, HLR and Lunar Operations

Jan. 16, 2008 6 PMR 07 Confidence Level Team

♦ Johnson Space Center • SCEA – Kelley Cyr, Vickie Gutierrez, Susan Bertsch, Steve Wilson • CEV – John Harrison, Susan Bertsch • EVA – Brian Johnson, Jen Nicholson • Program Integration – Keith Combs • Mission Ops – Brad Stewart ♦ Marshall Space Flight Center • CLV – Steve Creech, Charles Hunt, Barbara Stone-Towns ♦ Kennedy Space Center • Ground Ops – Glenn Rhodeside, Glenn Butts, Juan Gordon ♦ Aerospace Corp • Inki Min, Torrey Radcliffe, Marcus Lobbia, Dean Bucher ♦ Tecolote Research • Darren Elliott, Alf Smith, James Johnson, Mike Allen, Jeff McDowell, Troy Miller ♦ Vision Analytics (VAI) • Jim Costello, Becca Marler, Stuart Spuler

Jan. 16, 2008 7 CxP Scope – 4 Phases

♦ ISS IOC ♦ Human Lunar Return (HLR) • Through Initial Operations Capability • Through Human Lunar Return (HLR) (IOC) mission (Orion 2, Sep. 2013) mission (LSAM 2, Jun. 2019) • Includes • Includes − Development − Development − Test hardware − Test hardware − Flight hardware for IOC mission − Flight hardware for HLR mission − Fixed costs (infrastructure) thru IOC − Fixed cost (infrastructure) Thru HLR year year • Include ops costs thru IOC year • Includes all ops cost thru HLR year − If only one flight, include all ops cost − If only one flight, include all ops cost in that year in that year − If more than one flight, prorate by − If more than one flight, prorate by fraction of launch year when IOC fraction of launch year when IOC flight occurs flight occurs • Program Integration breakout • Program Integration breakout determined by project determined by project ♦ ISS Operations ♦ Lunar Operations • Everything non-Lunar not included in • Everything non-ISS not included in ISS IOC thru 2020 HLR thru 2020 • Shared costs breakout as • Shared costs breakout determined determined by project by project • Includes flight hardware production • Includes flight hardware production consistent w/ the assumptions above consistent w/ the assumptions above for IOC for HLR • Program Integration breakout • Program Integration breakout determined by project determined by project

Jan. 16, 2008 8 General Methodology: Risk Assessment

♦ Obtain Cost Requirements from ♦ Assess Project / Program Identified Projects Threats • Phased results over time by major • Project review of IRMA Threat List to WBS identify which are captured by project • Identification of costs into four program risk analysis phases (ISS IOC, ISS Operations, • Implementation of remainder into a Lunar HLR, and Lunar Operations) probabilistic risk model ♦ Convolve Uncertainties (Project Cost ♦ Obtain Cost Risk Assessments from Requirements plus Threats Projects ♦ Identify where Cx Budget Values fall • General approach on Integrated Program S-Curve • Risk results, correlation, and supporting statistics

Projects ♦ Review Risk Results with Projects • Identify coverage of risk analysis (what P TOTAL Program risks are covered) $ • Provide feedback on areas of concern Cumulative P Probability $ P ♦ Implement Correlation Cost • Project Level (elements within a $ project) Threats • Program Level (project to project) P

$ Jan. 16, 2008 9 Uncertainty Definitions

Uncertainty in the Model Not Included • CER Error • Database Coverage

Uncertainty in What Uncertainty Due to You Are Estimating Unpredictable Scenarios • Input Uncertainty • Project Re-Scope • Major system change, etc. • Mass • Acts of Congress •MV Inputs • Change in direction • Analogy Uncertainty • Funding loss, etc. •Level of Similarity • Acts of God • Additional Uncertainties • Hurricanes, etc. • Technical, RRW, etc. • Identified Threats • SSP Transition • Major Test Failures All Estimate Uncertainties

Jan. 16, 2008 10 Integration Method

♦ Developed an Integrated Cost Model Framework in ACEIT • Structured to allow direct input from Projects for phased cost requirements and risk statistics • Risk parameters and correlation applied to Project costs • Cost/Risk results bucketed into the four Cx Phases (ISS IOC, ISS Ops, HLR, Lunar Ops)

♦ Second model developed to allow inter-project correlation and handling of threats (discrete risks from IRMA) • Top-level project results by phase exported to second model • Correlation of 0.40 applied to each project • Threats not contained within risk analysis are assessed and added as a separate project

♦ Confidence Level Results Generated for: • General Total Program S-Curve • Budget Confidence over Time

Jan. 16, 2008 11 CLE Model Overview

♦ Model designed to support several activities • Detailed Integration and Allocation to Cx Phases for Project Risk Estimates • Capability to correlate Project Results at total Cx Phase level • Incorporation of Discrete Risks into overall Cx Phase Risk Results • Integration of Cost and Schedule Risk Assessments • Identification of Confidence Level of Cx Budget

♦ CLE Model consists of four separate files • Detailed Project Risk Analysis ACE File • Program Level Integration ACE File • Cost and Schedule Integrated Risk ACE File • CLE Results Excel file

Jan. 16, 2008 12 High-Level CLE Model Architecture

CLE Results (S-Curves, Phased Results, Sand Charts, Risk Over Time Charts, Cost/Sched Frontier Plots, etc)

Budget Section Budget Section Cost and Schedule Results Project Point Estimate Summary Program Risk Results Over Time (Correlated)

Project Risk Results Over Time Program Phase Results Cx Program Summary (Project Correl and Discrete) Results by Phase Risk Results by Phase (Phased TY$M and Risk Distros) Cx Top-Level Project Risk Results (IOC, ISS Ops, HLR, Lunar Ops) (IOC, ISS Ops, HLR, Lunar Ops) Schedule Risk Assessment Incorporation of Additional Risks Discrete Risks by Cx Phase Project Estimates, Phasing and Risk Data Discrete Risks Cx Schedule Data (Impacts and Likelihood) General Inputs (Manifest, etc) Integrated Cost Project Model Program Model and Schedule

Jan. 16, 2008 13 Challenges in Integrating Project Risk Data

♦ In many situations, the estimate for which a risk analysis was conducted was different than the phased estimate submitted for their PMR ’07 budget request

♦ Each project used different tools to conduct their risk analysis • NAFCOM • Crystal Ball • @Risk

♦ Projects had not structured their models to adequately bucket cost risk results into the four Cx program phases

Jan. 16, 2008 14 Three Approaches in Implementing Project Risks

♦ Method 1: Direct implementation of Risk Distributions on Project Estimates • Program Integration • Mission Operations

♦ Method 2: Application of Risk Distribution to Total Project Costs • Lander • SCIP

♦ Method 3: Project Risk Estimates Calibrated to Basis of Underlying Project Risk Analysis • EVA • Ground Operations • ARES I • ARES V • CEV

Jan. 16, 2008 15 Integration Review Placed Focus on What Risks were Encompassed by the Projects

♦ A Comprehensive Risk Analysis Usually Captures the Following: • Effort associated to conduct the activities • Labor rate changes • Weight Growth • SLOC Growth • Nominal Perturbations in Schedule

♦ A Typical Cost Risk Analysis Usually Does NOT Capture: • Major Requirement Changes • Major Design Changes (e.g., Engine change) • Manifest changes • Major Test Failures • Changes in Development/Production Plans (e.g., adding additional test cycles, additional spares, etc) • Funding impacts to planned activities that cause delays

Jan. 16, 2008 16 However, Not All Risk Analysis are Equal

♦ Inputs-Based Approach with Parametric / Analogy Cost Estimating Methods • Capture wide range as historical programs constitute the data base used for estimation • Distributions on estimate drivers capture impact of potential technical parameter changes ♦ Inputs-Based Approach with Grass Roots Estimating Methods • Usually based on a particular plan / approach with estimating method based on “effort * rate” or “material * qty” methods • Distributions placed on input parameters (effort, rate, etc) to equations • Overall risk captured is only as good as what was considered in identifying the bounds (e.g., was the very worst case considered or was just in the identified effort what could be the potential range) ♦ Output-Based Approach around Cost Results • Distribution placed on resulting cost • Overall risk captured is only as good as what was considered in identifying the bounds (e.g., was the very worst case considered or was just in the identified effort what could be the potential range)

Some Risk Analyses are More Comprehensive than

Jan. 16, 2008 Others 17 Developing a Comprehensive Cost Risk Analysis for Cx Required Inclusion of IRMA Threats

♦ Review of Project Risk Results Identified that the Project Risk Assessments were not Equal • Some projects had a broad view of risk − Used parametric models based on historical program − Thought through worst case and best case in establishing risk bounds • Others focused on assessing the risk in the current plan, and not on the impact if the plan changes − Variability in Labor Rates − Range in effort required to complete planned work − Potential risks not considered in the plan were identified as threats in IRMA ♦ In Order to Develop a Comprehensive Program Level Risk Analysis, the Project S-Curves Need to be Normalized to Same Risk Content Basis in Order to Assess Overall Cx Program Risk ♦ Need to Include IRMA Threats Not Captured by Project Risk Analysis into Overall Risk Analysis

Jan. 16, 2008 18 Summation Method for Including IRMA Threats into Cx CLE

All Projects ~ costRisk Orion ~ costRisk S-Curve S-Curve

Σ Prob(cost) Σ Prob(cost) Cost CX ~ Comprehensive S-Curve Cost Ares ~ costRisk S-Curve

Σ Prob(cost) Σ Prob(cost)

Cost

All Projects ~ Threats Orion ~ Threats S-Curve Cost S-Curve

Σ Prob(cost) Σ Prob(cost) Cost

Cost Ares ~ Threats S-Curve

Σ Prob(cost)

Cost Jan. 16, 2008 19 Cx Discrete Risk Results by Program Phase

Cx ISS IOC Cx ISS Ops Identified Threats Risk Analysis Identified Threats Statistics Statistics Calculated with 3500 iterations Calculated with 3500 iterations

100% 5% 100% 30%

90% 5% 90% 25% 80% 4% 80% D D 70% 4% a 70% a 20% 60% 3% 60%

50% 3% 50% 15%

40% 2% 40% 10% Probability (Histogr Probability (Histogr Confidence Level(C 30% 2% Confidence Level(C 30%

20% 1% 20% 5% 10% 1% 10%

0% 0% 0% 0% $0,496 $0,703 $0,910 $1,116 $1,323 $1,530 $1,736 $1,943 $2,150 $2,356 $85.89 $323.14 $560.39 $797.64 $1034.90$1272.15$1509.40$1746.65$1983.90$2221.16 TY $M TY $M

Probability Histogram Confidence Level (CDF) Probability Histogram Confidence Level (CDF)

Cx Lunar HLR Cx Lunar Ops Identified Threats Identified Threats Statistics Statistics Calculated with 3500 iterations Calculated with 3500 iterations

100% 5% 100% 6%

90% 4% 90% 5% 80% 4% 80% D D a 70% a 70% 3% 4% 60% 60% 3% 50% 50% 3% 2% 40% 40% 2% 2% Probability (Histogr Probability (Histogr Confidence Level(C 30% Confidence Level(C 30%

20% 1% 20% 1% 10% 1% 10%

0% 0% 0% 0% $0,198 $0,286 $0,375 $0,463 $0,551 $0,640 $0,728 $0,817 $0,905 $0,993 $34.26 $68.92 $103.58 $138.24 $172.90 $207.56 $242.22 $276.88 $311.54 $346.20 TY $M TY $M

Probability Histogram Confidence Level (CDF) Probability Histogram Confidence Level (CDF)

Jan. 16, 2008 20 Impact on Program S-Curve by Adding Discrete IRMA Threats

M ultiple cases Integrated Risk Program Estimate- ISS IOC Scope (InterProject Correl) Statistics 100% Calculated with 3500 iterations Using LatinHypercube 90%

80%

D 70%

60%

50%

40%

Confidence Level (C Level Confidence 30%

20%

10%

0% $15,000 $17,000 $19,000 $21,000 $23,000 $25,000 $27,000 $29,000 $31,000 $33,000 $35,000 TY $M Baseline Mean 65% Confidence Level Cx Program without Threats Mean (Cx Program without Threats) Adding in Discrete Risks Shifts the S-Curve

Jan. 16, 2008 21 Correlation

♦ Correlation within Projects Determined by Projects • Range from 0.2 to 0.7 for component level correlation • Range from 0.2 to 0.5 for subsystem level correlation ♦ Correlation between Projects for Program Level Based on Minimum Error Assumption • Conducted sensitivity analysis for inter-project correlation − Determined that maximum impact of correlation occurs in the 40-60% range − Identified that 0.40 correlation provides minimum error from 0% to 100% correlation range • 0.4 used to correlate projects to each other ♦ Comprehensive S-Curve captures correlation within projects and between projects

Jan. 16, 2008 22 Correlation Comparison Chart

Multiple case Comparison Total Program S-Curve Statistics Calculated with 3500 iterations Using LatinHypercube 100%

90%

80%

D 70%

60%

50%

40%

Confidence Level (C Level Confidence 30%

20%

10%

0% $25,000 $30,000 $35,000 $40,000 $45,000 $50,000 $55,000 TY $M Zero Parent Correl 100% Parent Correl 40% Parent Correl

Jan. 16, 2008 23 ISS IOC Confidence Level S-Curve

100%

90%

80% $26.40B 70%

60%

50%

40% Confidence Level Confidence

30%

20%

10%

0% $19.00 $21.00 $23.00 $25.00 $27.00 $29.00 $31.00 PMR 07 Submit TY $B 65% Confidence Level IOC Budget + IOC Allocated Reserves for Sept 2013 IOC Budget + IOC Reserves thru 2015 IOC Budget + All IOC Reserves thru 2013 IOC Budget + All Avail Reserves thru 2015

Jan. 16, 2008 24 Cx S-Curves by Phase

Confidence Level Estimate - IOC Confidence Level Estimate - Station Operations 100% 100% 90% 90% 80% 80% 65% 65% 70% Program 70% Budget w/ 60% 60% 50% Reseves 50% 50% 40% Program 40% Budget w/ Confidence Level Confidence

Confidence Level Confidence 30% 30% Reserves 20% 20% 10% 10% 0% 0% $14 $16 $18 $20 $22 $24 $26 $19 $21 $23TY $25 $B $27 $29 $31 TY $B

Confidence Level Estimate Confidence Level Estimate ~ Humar Lunar Return ~ Lunar Operations 100% 100%

90% 90%

80% 80% 65% 65% 70% 70%

60% 50% 60% 50% 50% Program Budget 50% w/ Reserves 40% 40% Program Confidence Level Confidence 30% Level Confidence 30% Budget w/ Reserves 20% 20%

10% 10% 0% 0% $35 $37 $39 $41 $43 $45 $47 $49 $51 $53 $55 $57 $59 $8 $9 $10 $11 $12 $13 $14 $15 $16 Jan. 16, 2008 TY $B TY $B 25 Time Phased Risk

♦ Viewing Risk Results as Cumulative Result can be Misleading • Over entire effort, program may be under-funded, but large amount of reserves thrown to program at the end. • Result is that reserves were not available when needed, effort that was planned to get done was not done, schedule pressure has increased, and program may not be able to spend available dollars ♦ Level II Developed Methodology to View Risk Results over Time • Extract Risk Results for each year for the Cumulative Costs through that Period in Time • Plot the Resulting 90% Confidence Band (5% to 95%) over Time • Plot the available budget • Identify the confidence level per year of the budget ♦ General Underlying Principles in Viewing the Results • First, assumption that the risk is evenly spread over the period, meaning that a user cannot assign specific years to have higher or lower risk than specified in the overall risk bound. • Second, underlying assumption that the estimate is phased appropriately over the time periods to capture the work that will be performed. • There is an underlying assumption that when you look at a specific year that all the work planned to that point will have been accomplished. Or meaning in an EVM standpoint, CPI and SPI both equal "1".

Jan. 16, 2008 26 Time Phased Risk

♦ Cx Program is Underfunded in Early Years, thereby Placing Additional Pressure on Schedule

Confidence Band Over Time ~ IOC 35

30

25

20 $B 15

10 5-95% Bound Available Budget 5 65% Target Level

0 2007 2008 2009 2010 2011 2012 2013 2014 Fiscal Years

Jan. 16, 2008 27 Integrating Cost and Schedule

♦ Cost And Schedule Risk Assessments give us Independent Perspectives of Confidence

♦ Key is to Link Cost And Schedule to Estimate Joint Probability • Ideally cost or schedule would be a driver for the other, but no relationship for NASA manned programs could be determined • However, research in unmanned systems shows strong relationship between cost and schedule growth

♦ Level II Implemented Method to Correlate Cost and Schedule S-Curves to Develop Joint Probability • Assumes 0.7 correlation (Strong) between cost and schedule • Further research needed to develop appropriate correlation values

Jan. 16, 2008 28 Moving Forward Plan

♦ Build an Integrated Program Level Cost Model • Allow sensitivity to input changes (schedule slips, mission model) • Allow capability to model risk based on project methodologies • Phased rollout (NT = CLV and CEV) • Incorporate Periodic Estimate Updates targeted around major program milestone activities (PMR, PPAR, Pre-NAR, IDACs, etc) ♦ Update Risk Analysis to Account for Lunar Architecture Study Results ♦ Update Risk Analysis to Account for Lunar Lander Design Study Results ♦ Support Program Approval Process (PPAR) ♦ Support Project Approval Process (Pre-NAR) ♦ Support Program Planning, Budgeting, and Execution (PPBE) Process

Jan. 16, 2008 29