Methodology Tailoring for Development Time Optimization

Anthony Millán JHU/APL

3/20/2019 Methodology Tailoring for Development Time Optimization 1 Agenda

• Background • Definitions • System Complexity, DARPA, and the META Design Flow • Potential to Improve SE Practice • Remaining Problems • Tailoring a SE Methodology for the META Design Flow • Research Approach • META Design Flow Project SDLC • Selecting and Tailoring a Methodology • Tailored Methodology • Methodology Validation • Research Conclusions • Additional Efforts • 2018 INCOSE Systems Engineering in Healthcare • Follow-up Research

3/20/2019 Methodology Tailoring for Development Time Optimization 2 Definitions

• Model-Based Systems Engineering (MBSE) – Systems Engineering approach that focuses on the use of descriptive models to support the definition and capture of system design decisions

• System Development Lifecycle (SDLC) – Stepwise evolution of a new system from concept through development and on to production, operation and ultimate disposal

• Design Flow – Portion of the SDLC that encompasses development of the conceptual and detailed design

• Methodology – Overarching set of Systems Engineering processes iteratively applied throughout the SDLC stages

• Process – Set of organized activities that define Systems Engineering effort focus and transforms inputs into outputs

• Activity – Detailed Systems Engineering task performed as part of a process and adjusted for a specific SDLC stage

3/20/2019 Methodology Tailoring for Development Time Optimization 3 The Challenge: System Complexity

• The value of Systems Engineering (SE) – historically managing complexity and change • “As complexity and change continue to escalate in products, services, and society; reducing the risk associated with new systems or modifications to complex systems continues to be a primary goal of the systems engineer”1

However…

• The systems community continues to face developing challenges regarding complexity • “Mission complexity is growing faster than our ability to manage it… increasing mission risk from inadequate specifications and incomplete verification”2 • “System design emerges from pieces, rather than from architecture… resulting in systems that are brittle, difficult to test, and complex and expensive to operate”2

1INCOSE SE Handbook, INCOSE, 2015 2INCOSE SE Vision 2025, INCOSE, 2014

3/20/2019 Methodology Tailoring for Development Time Optimization 4 The Focus: Military System Complexity

Military system complexity challenges identified by DARPA: • Unexpected interactions • SE has not (fundamentally) changed in 50 years • Unlike other industries, development time of military systems increases proportionally with complexity

Source: Paul Eremenko, Formal Model-Based Design & Manufacture: A Template for Managing Complexity in Large-Scale Cyber-Physical Systems, 2013 3/20/2019 Methodology Tailoring for Development Time Optimization 5 The Source: Systems Engineering

DARPA attributed some of these problems to shortcomings in the application of SE to military system development • Functional decomposition introduces unnecessary boundaries (stovepipes) • Inconsistent optimization goals (cost, SWaP) • Disincentivized design abstraction • Integration results in “brittle” point design architectures • Unknown emergent behaviors • Traditional V&V methods unable to scale with highly complex systems because of the large number of possible system states

As a response, DARPA started the Adaptive Vehicle Make (AVM) Portfolio • Programs: META, iFab, VehicleFORGE • Purpose: Transform design, analysis, verification and manufacture of complex systems to compress the development timelines by at least 5X

META design flow – New system development method proposed by DARPA as part of the META program • Prerequisites: 1. Formalized design language 2. Defined system-level requirements and concept 3. Detailed component model library Source: DARPA - iFAB Foundry / FANG Proposers’ Day

3/20/2019 Methodology Tailoring for Development Time Optimization 7 DARPA’s Solution: META Design Flow

META design flow stages: • Develop an optimized system-level logical architecture taking into account the requirements and the available components • Define physical architecture alternatives from all possible component combinations • Execute the architecture alternatives through automated analysis tools to exclude from consideration poor performers • Increase modeling and analysis fidelity and iterate until final architecture is selected

Source: ISIS/Vanderbilt University - The META Toolchain: Accomplishments and Open Challenges

3/20/2019 Methodology Tailoring for Development Time Optimization 8 Potential to Improve SE Practice

Common MBSE Benefits Additional META Design Flow Benefits • Promotes use of modular, open systems approach • Composing system design from pre-defined lower (MOSA) level models • Promotes design of tightly bound, loosely coupled • Reduced development time for complex systems architectures • Correct-by-construction designs • Shift in SE effort to early stages of development • Optimizing for system complexity and adaptability • Increased level of abstraction in design instead of functional performance metrics • Model-based verification • Increasing return on investment for repeated uses • Faster requirement-based trade studies

The META design flow leverages the benefits of MBSE methods and expands beyond them

3/20/2019 Methodology Tailoring for Development Time Optimization 9 Example: Improving Trade Studies

Traditional Trade Study Process Trade Study Using META Design Flow Entire Component Design Space Entire Component Design Space Candidate Candidate Architectures Architectures

(optimum) 1 (optimum) 1 3 Available Architectures 2 2 Selected Selected Architecture Architecture

1. Component model library continuously updated and expanded 1. Candidate architectures are selected for consideration from SME & SE judgment 2. Unfeasible architectures excluded through requirements-based analysis 2. Final architecture is selected from an analysis of the candidates 3. Final architecture selected through higher fidelity analysis 3/20/2019 Methodology Tailoring for Development Time Optimization 10 Example: Increased RoI

Initial System Development Effort Subsequent Efforts within Domain Entire Component Design Space Entire Component Design Space Candidate Candidate Architectures Architectures 1 3 Available Available Architectures Architectures 2 1 2 Selected Selected Architecture Architecture

1. Component model library continuously updated and expanded 1. Larger component model database results in more 2. Unfeasible architectures excluded through requirements-based available architectures analysis 2. Increased model fidelity accelerates design optimization 3. Final architecture selected through higher fidelity analysis 3/20/2019 Methodology Tailoring for Development Time Optimization 11 Remaining Problems

“If application of the META Design Flow has so many potential benefits, why has it not been broadly applied in fields outside of military system development?” • Limited exposure outside of AVM programs • Constraining prerequisites • Limited theoretical foundation

Research Problem Summary: • The META design flow proposes a transformation of Systems Engineering design methods to reduce the development time of complex and adaptable systems • This approach could improve system development processes in different industries • Several barriers have limited the META design flow’s adoption in other domains • Literature review yielded that additional research on the integration of the META design flow into the broader scope of Systems Engineering methodologies could provide a framework for its incorporation into other domains 3/20/2019 Methodology Tailoring for Development Time Optimization 12 Research Approach

Proposed Research: • Incorporate the META design flow into a tailored end-to-end SE methodology to expand its applicability to other domains

Approach: 1. Study SDLC of a project using the META design flow 2. Study existing methodologies to select one for tailoring 3. Tailor the selected methodology following guidance from the Tailoring Process described in chapter 8 of the INCOSE SE Handbook: • Derive modifications from SDLC analysis and unique domain needs • Modify activities within the original processes • Add new processes, remove processes, and combine processes

3/20/2019 Methodology Tailoring for Development Time Optimization 13 META Design Flow Project SDLC

• By incorporating the META design flow into the SDLC, parts of Concept Definition, Advanced Development and Engineering design stages are combined and all component-level development is removed from SE consideration

3/20/2019 Methodology Tailoring for Development Time Optimization 14 Review of Available SE Methodologies

14 SE and MBSE methodologies were evaluated: Evaluation Criteria: • MIL-STD-449B • Research goal compatibility • IEEE-1220 • EIA/ANSI 632 • META design flow compatibility • ISO/IEC/IEEE 15288 (INCOSE) • Process Structure • Kossiakoff et al. SE Method Traditional SE • Iteration approach • Top-Down Systems Engineering • Recursion approach • Waterfall Software Engineering • Spiral Software Engineering • SDLC relationship • Harmony-SE • Coverage of SDLC stages • INCOSE OOSEM • SDLC dependence/independence • IBM RUP-SE • Vitech MBSE MBSE • Flexibility for tailoring • JPL State Analysis • Scope of process definition • Dori OPM • Detail in guidance

3/20/2019 Methodology Tailoring for Development Time Optimization 15 Top Candidates for Tailoring

1. Kossiakoff et al. SE Method 2. INCOSE Object Oriented SE Method Description Description • Processes structured around SE focus areas (requirements • MBSE methodology that combines object-oriented concepts analysis, functional definition, physical definition and design with traditional SE processes validation) • Development activity focus shifted towards architecture • Activities tailored for SDLC stages and levels of system hierarchy development consistent with MBSE approaches recursion Pros Pros • Very compatible with META design flow • Built-in flexibility for tailoring • Simplified tailoring for MBSE • SDLC independence Cons • Completeness • Limited scope Cons • Incompatible with research goal • Structure incompatible with META design flow • Difficult to convey combination of activities

3/20/2019 Methodology Tailoring for Development Time Optimization 16 Selected Methodology

ISO/IEC/IEEE 15288 methodology used in INCOSE SE Handbook Pros • Built-in flexibility for tailoring through process modification, addition or removal • Structure (provides mechanism to derive process modifications) • Completeness Cons • Low flexibility of activity allocation within processes • SDLC dependence

Credit: Paul Martin, Systems Engineering Professional Preparation Course 3/20/2019 Methodology Tailoring for Development Time Optimization 17 Tailored Methodology

• Proposed methodology for a project using the META design flow (“Vee” view)

3/20/2019 Methodology Tailoring for Development Time Optimization 18 Tailored Methodology (Continued)

• Proposed methodology for a project using the META design flow (META design flow focus view)

3/20/2019 Methodology Tailoring for Development Time Optimization 19 Process Tailoring Matrix

ISO/IEEE/IEC Processes Tailoring Action Tailored Methodology Process Business or Mission Analysis Modified Business or Mission Analysis Stakeholder Needs and Stakeholder Needs and Modified Requirements Definition Requirements Definition System Requirements Definition Modified System Requirements Definition Architecture Definition Combined, Added Modeling Environment Setup Design Definition Combined, Added Architecture Definition and Analysis System Analysis Combined, Added Implementation Removed - Integration Modified Integration Verification Modified Verification Transition Unchanged Transition Validation Unchanged Validation Operation Combined Maintenance Combined Operation, Maintenance and Disposal Disposal Combined

3/20/2019 Methodology Tailoring for Development Time Optimization 20 Methodology Validation

Methodology validation approach combined quantitative and qualitative methods:

• Qualitative: Peer review of the proposed methodology by MBSE and Systems Architecting SMEs through written feedback • Contributing SMEs: Sanford Friedenthal (MBSE), Clyde Smithson (System Architecting) • Open questions – Overall positive evaluation of methodology • Feedback incorporated into final proposed methodology and tailoring discussion

• Quantitative: Validation of the proposed methodology’s viability based on SE community opinion gathered through a standardized survey • 4 categories evaluated: completeness, conformity, feasibility, adaptability • 12 closed questions, 4 open questions (3 and 1 respectively for each category) • Cascade sampling with initial invitation to JHU EP faculty, JHU/APL SE professionals, INCOSE Chesapeake Chapter, INCOSE LinkedIn group, invited peer reviewers • 22 responses, conclusions derived from response trends and observations

3/20/2019 Methodology Tailoring for Development Time Optimization 21 Validation Survey Results

Mostly positive comments regarding conformity and adaptability

Top recurring concerns: • Apparent exclusion of functional analysis in the methodology • Original methodology addresses functional analysis as a set of activities within System Requirements Definition

• Apparent lack of iteration between processes • By definition, processes iteratively applied throughout SDLC stages (not graphically clear)

• Availability and fidelity of required component model library may limit feasibility • This limitation is discussed in the paper • Repeated uses of this methodology mitigate this issue, resulting in increased RoI

3/20/2019 Methodology Tailoring for Development Time Optimization 22 Research Conclusions

• System complexity continues to be a challenge of modern SE efforts

• The META design flow has the potential to help overcome these challenges • In compatible domains, if applied through an end-to-end SE methodology • Leverages and expands benefits of MBSE approaches

• There are limitations and special considerations that should be taken into account in initial applications of the META deign flow • Tailoring discussion provides additional guidance required for application in domains outside of military system development • Tailored methodology provides a starting point for additional project-specific tailoring

• Overall positive results in SE community survey of methodology viability • Suggests tailored methodology could promote broader adoption of the META design flow • Final validation requires additional evidence through case studies of the META design flow applied to different domains

3/20/2019 Methodology Tailoring for Development Time Optimization 23 Additional Methodology Exposure

3/20/2019 Methodology Tailoring for Development Time Optimization 24 Applicability to OR Design

Operating room (OR) design was identified early as a potential candidate for the application for the META Design Flow outside of military system development - Why? 1. The problem is compatible with the META Design Flow: • OR design does not follow top-down development (OR architecture developed first, medical equipment developed specifically for the architecture) • Instead, medical equipment often developed independently to perform specific functions, and integrated into a federated system based on user needs • Modularity and interoperability are virtual necessities of modern medical equipment • Medical equipment functions can be modeled in low fidelity, facilitating automated analysis

2. The META design flow provides specific benefits when applied to this problem: • Shifting the focus of OR design to the system-level architecture reduces unexpected emergent behaviors (such as cable interference, conflicting alerts, display clutter) • Reducing system complexity is by itself a source of value in OR design • Increased system adaptability reduces cost of equipment changes • Increased return on investment for subsequent OR designs (medical equipment models already in place, just change the architecture and optimization function)

3/20/2019 Methodology Tailoring for Development Time Optimization 25 Follow-Up Research

Will continue expanding research on the META design flow and the tailored methodology by exploring topics in four major areas: • Practical Application – Validation of the methodology’s feasibility through case studies • OR (ICU) Room Architecting • System-of-System architecting (for defense) • Missile design optimization • Modern MBSE Tools – Is the state-of-the-art of MBSE sufficiently mature to support the META design flow’s ambitions? • SE Theory – How des the methodology itself evolve to support repeated applications? • Advanced Concepts – How does the META design flow relate to other aspects of SE like agile development, automated design (and manufacture), and design optimization?

3/20/2019 Methodology Tailoring for Development Time Optimization 26 Closing Remarks

Thank you!

• Contact for further discussion: • [email protected] • [email protected]

• Special thanks: • Max Crownover – Master’s Thesis Advisor • Mike Pafford, Kathy Ruben, Larry Strawser – Master’s Thesis Board • Sanford Friedenthal, Clyde Smithson – Methodology Reviewers • Mike Borky, John Labadie – Ph.D. Advisors • Family & friends

3/20/2019 Methodology Tailoring for Development Time Optimization 27 Questions?

3/20/2019 Methodology Tailoring for Development Time Optimization 28