Value of Systems Engineering

INCOSE SECOE Systems Engineering Center of Excellence Honourcode, Inc. Honourcode, Inc. Understanding the Value of Systems Engineering

How can we quantify worth of what we do?

Eric Honour Honourcode, Inc. Director, INCOSE Systems Engineering Center of Excellence

Value of Systems Engineering; INCOSE Symposium 6/04 2 Agenda

„ Heuristic Claim of SE „ Gathered results on Value of SE „ NASA Tracking 1980s „ “Boundary Management” study „ “Large Engineering Projects” MIT study „ “Impact of SE at NASA” (SECOE 02-02) „ “Impact of SE on Quality & Schedule” Boeing „ “SE Effectiveness” IBM study „ “Value of SE” research (SECOE 01-03)

Value of Systems Engineering; INCOSE Symposium 6/04 3 Heuristic Claim of SE

„ Better systems engineering leads to „ Better system quality/value „ Lower cost „ Shorter schedule

Traditional Design Risk

SYSTEM DETAIL PRODUCTION Time DESIGN DESIGN INTEGRATION TEST Risk Saved Time/ Cost “System Thinking” Design Time Value of Systems Engineering; INCOSE Symposium 6/04 4 NASA Tracking 1980s

Total Program Overrun 32 NASA Programs

200 GRO76 Definition $ OMV 180 Definition Percent = ------

TDRSS Target + Definition$ 160 GALL 140 IRAS Actual + Definition$ HST Program Overrun = ------TETH 120 Target + Definition$

GOES I-M MARS 100 EDO CEN LAND76 ACT ERB77 MAG 80 COBE CHA.REC. STS

60 LAND78 GRO82

SEASAT ERB88 40 VOY HEAO Program Overrun Program UARS 2 EUVE/EP DE R = 0.5206 20 ULYS SMM IUE ISEE 0 PIONVEN 0 5 10 15 20 Definition Percent of Total Estimate

Source Werner Gruhl NASA Comptroller’s Office Value of Systems Engineering; INCOSE Symposium 6/04 5 “Boundary Management” Study

„ Study of 45 high-tech new product development teams ...Ancona and Caldwell, Research Technology Management, 1990

„ Significant portion of time is spent at team boundaries Individual Time Spent „ Outside Team 14%* Successful teams spent more time in boundary * Typically limited to few individuals management „ Within Team 38% Processes used did not correlate with success „ Alone 48%

Value of Systems Engineering; INCOSE Symposium 6/04 6 “Large Engineering Projects”

„ Study of 60 LEPs (power generation, transportation, oil production, technology) The Strategic Management of Large Engineering Projects, MIT Press 2000

„ Evaluation by interviews and by objective and subjective quality measures. Percent of Projects Meeting:

Cost Targets 82%

Schedule Targets 72%

Objective Targets 45% 18% 37% Failed!

Value of Systems Engineering; INCOSE Symposium 6/04 7 “Large Engineering Projects”

Significant Findings:

„ Most important determinant was a coherent, well-developed organizational/team structure A structure of leadership creates greater success

„ Technical difficulties, social disturbance, size were not statistically linked to performance „ All projects had turbulent events „ Technical excellence could not save a socially unacceptable project Process definition is important but not sufficient.

Value of Systems Engineering; INCOSE Symposium 6/04 8 Impact of SE at NASA (SECOE 02-02)

Participating Organization Education Level

NASA 136 INCOSE 243 Bachelor 34% Total 379 Master 55% Doctor 11%

Work Experience

Job Titles 0-10 41% 11-20 37% 21+ 22% SE 56% PM 17% Other 27% Age Groups

20-39 29% 40-59 61% 60+ 10% ...significant differences by organization

Value of Systems Engineering; INCOSE Symposium 6/04 9 Key Survey Results - Cost

Percent Spent on SE • Respondents marked bracket to show percent of total cost spent on SE on last project. • Mode at 6-10% of project

0-5% 6-10% 11-15% 16% + • Few projects spent 11-15% • Unexplained bimodal response >16% (perhaps interpretation of “project”)

Cost Benefit of SE • Respondents believe strongly in cost benefit of SE • In secondary question, few respondents could quantify • INCOSE respondents much higher % SEs, had more positive beliefs Very Poor Poor Fair Good Excellent

Value of Systems Engineering; INCOSE Symposium 6/04 10 Impact of Systems Engineering on Quality and Schedule

„ Empirical evidence obtained from three parallel (same time) projects „ Each developed a complex, robotic Universal Holding Fixture (UHF) „ Each used a different level of SE „ Results are compared

Trait UHF1 UHF2 UHF3 Size 10’ x 40’ 8’ x 50’ 6’ x 14’ Accuracy ±0.005” ±0.003” ±0.003” Contact Sensors None 57 108 Vacuum Sensors 1 70 108 Real-time checks No Yes Yes Probe contours No Yes Yes NC interface No Yes Yes

Impact of Systems Engineering on Quality and Schedule – Empirical Evidence, W. Forrest Frantz, Boeing Corp. 1995 Value of Systems Engineering; INCOSE Symposium 6/04 11 Impacts

Requirements to RFP (weeks) Design to Production (weeks)

UHF3 UHF3 UHF2 UHF2 UHF1 UHF1

0102030 0204060

Overall Development Time (weeks)

„ Use of better SE reduced UHF3 „ Overall cycle time UHF2 UHF1 „ Time to create req’s „ Time to design/produce 050100 „ Time to test ...even in the face of more complex, higher quality systems!

Value of Systems Engineering; INCOSE Symposium 6/04 12 Systems Engineering Effectiveness

„ Study of 8 software product development projects during upgrade of SE processes Determining Systems Engineering Effectiveness, Bruce Barker, IBM Commercial Products, Conference on Systems Integration, Stevens Institute 2003 „ Evaluation by cost and schedule against a standard estimating method. Historical Database, New Identify Convert Cost per “Point” Product affected to Concept components “points” Evaluate Estimate Product Line Impact, Cost, Architecture Complexity Schedule Costing method applies only to project management, business management, systems engineering, system integration, and delivery into production. Application development costs are not included.

© Copyright IBM Corp 2003 Value of Systems Engineering; INCOSE Symposium 6/04 13 Used With Permission Systems Engineering Effectiveness

Significant Findings: „ Impact and complexity provide an effective method to perform parametric costing. Early parametric costing works.

„ Preliminary data indicates that the use of Systems Engineering will improve project productivity when effectively combined with the Project Management and Test Processes. Systems engineering improves productivity. $/Point Averages 2000 $1,454/pt Without SE $1,350/pt 2001 $1,142/pt With SE $944/pt 2002 $818/pt

© Copyright IBM Corp 2003 Value of Systems Engineering; INCOSE Symposium 6/04 14 Used With Permission “Value of SE” (SECOE 01-03)

„ Multi-year effort to obtain statistical data „ Correlate amount/quality of SE with project quality/success

Development Parameterized by • Technical “Size” Quality • Technical Complexity • Risk Level

(function of Technical Value, Cost, Schedule, Risk) 6-10% ?

SE Effort Value of Systems Engineering; INCOSE Symposium 6/04 15 Respondent Data

10 „ 43 respondents 8 „ 1 project not completed 6 „ Values $1.1M - $5.6B

4 „ SE Cost 0.3% - 26%

2 Number of Projects

0 0 5 10 15 20 25 SE Cost %

10 „ Cost, schedule, quality 8 correlate better with “Systems 6 Engineering Effort”:

4 „ SEE = SE Qual * (SE Cost %)

2 Number of Projects

0 0 5 10 15 20 25 SE Effort = SE Quality * (SE Cost %)

Value of Systems Engineering; INCOSE Symposium 6/04 16 Cost Overrun vs. SE Effort

3.0 90% Assurance (1.6σ) 2.6

2.2 Average Cost Overrun

1.8

1.4 Actual/Planned Cost Actual/Planned 1.0 0% 4% 8% 12% 16% 20% 24% 28%

0.6

SE Effort = SE Quality * SE Cost/Actual Cost Honour, “Value of SE” INCOSE 2004 Value of Systems Engineering; INCOSE Symposium 6/04 17 Schedule Overrun vs. SE Effort

3.0 90% Assurance (1.6σ) 2.6

2.2 Average Schedule Overrun

1.8

1.4

Actual/Planned Schedule Actual/Planned 1.0 0% 4% 8% 12% 16% 20% 24%

0.6

SE Effort = SE Quality * SE Cost/Actual Cost Honour, “Value of SE” INCOSE 2004 Value of Systems Engineering; INCOSE Symposium 6/04 18 Test Hypothesis: Quality

1.2

Hypothesis

1.0 Quality

0.8 SE Effort

0.6

Development Quality 10.0 (Cost/Schedule Based) 9.0

0.4 8.0 0% 4% 8% 12% 16% 20% 24% 7.0 SE Effort 6.0

5.0

4.0

3.0

Comparative Success 2.0

1.0

0.0 Honour, “Value of SE” 0% 4% 8% 12% 16% 20% 24% INCOSE 2004 SE Effort Value of Systems Engineering; INCOSE Symposium 6/04 19 Conclusions “Value of SE”

„ SE effort improves development quality „ Cost, schedule, subjective „ Hypothesis is supported by the data „ Optimum SE effort is 10-15% or more „ Cost, schedule overruns are minimized „ However, note wide dispersion of data „ Also note few data points at this level; most projects spent far less „ Quality of the SE effort matters „ Lower quality SE reduces effectiveness

Honour, “Value of SE” INCOSE 2004 Value of Systems Engineering; INCOSE Symposium 6/04 20 Honourcode, Inc.

Questions?

Eric Honour Honourcode, Inc. Director, INCOSE SECOE Pensacola, FL, USA +1 (850) 479-1985 [email protected]

Value of Systems Engineering; INCOSE Symposium 6/04 21 Survey Data

Value of Systems Engineering; INCOSE Symposium 6/04 22 Survey Definitions

„ Systems Engineering Cost - all costs to perform traditional SE tasks, no matter who performs them. Typical: „ Technical management and coordination „ Mission and/or need analysis „ System architecting „ System-level technical analysis „ Requirements management „ Risk management „ ...and other tasks associated with these.

Value of Systems Engineering; INCOSE Symposium 6/04 23 Survey Definitions

„ Systems Engineering Quality - subjective evaluation of the overall quality of the systems engineering effort. Scale of 0-10 „ 0 represents SE having no useful value, „ 5 represents a normally effective SE effort, „ 10 represents unexcelled, world class quality.

„ Comparative Success - subjective evaluation of the overall project success against other comparable projects. Scale of 0-10 „ 0 represents a failed project, „ 5 represents a project comparable in success to most other projects „ 10 represents unexcelled, world class success.

Value of Systems Engineering; INCOSE Symposium 6/04 24 Known Limitations

„ Self-reported, unchecked data „ Subjective data items „ No quality controls „ Definition of “Systems Engineering” „ “SE Cost” dependent on respondent perception „ Respondent base not controlled „ Respondents primarily systems engineers

„ Perceptive influences are likely „ Hypotheses well known, widely accepted „ Respondents likely to have subconscious bias

Value of Systems Engineering; INCOSE Symposium 6/04 25