ASSESSING AND MITIGATING VULNERABILITY CHAINS IN MODEL- CENTRIC ACQUISITION PROGRAMS

by

Jack Burnett Reid

B.S. Mechanical Engineering and B.A. Philosophy Texas A&M University, 2015

Submitted to the MIT Institute for Data, Systems, and Society And the Department of Aeronautics and Astronautics in Partial Fulfillment of the Requirements for the Degrees of

Master of Science in Technology and Policy And Master of Science in Aeronautics and Astronautics

at the

Massachusetts Institute of Technology

June 2018

2018 Massachusetts Institute of Technology. All rights reserved.

Signature of Author……………………………………………………………………………………………………... MIT Institute for Data, Systems, and Society Department of Aeronautics and Astronautics May 18, 2018

Certified by……………………………………………………………………………………………………………... Donna H. Rhodes Principal Research Scientist, Sociotechnical Systems Research Center Thesis Supervisor

Certified by……………………………………………………………………………………………………………... Daniel E. Hastings Cecil and Ida Green Education Professor of Aeronautics and Astronautics and Engineering Systems Chief Executive Officer and Director, Singapore MIT Alliance for Research and Technology Thesis Supervisor

Accepted by…………………………………...………………………………………………………………………… Munther Dahleh W. Coolidge Professor of Electrical Engineering and Computer Science Director, MIT Institute for Data, Systems, and Society

Accepted by…………………………………...………………………………………………………………………… Hamsa Balakrishnan Associate Professor of Aeronautics and Astronautics Chair, Graduate Program Committee

This material is partially based upon work by the Naval Postgraduate School Acquisition Research Programs under Grant No. N00244-17-1-0011.

2 ASSESSING AND MITIGATING VULNERABILITY CHAINS IN MODEL-CENTRIC ACQUISITION PROGRAMS

by

Jack Burnett Reid

Submitted to the MIT Institute for Data, Systems, and Society and Department of Aeronautics and Astronautics on May 18, 2018 in Partial Fulfillment of the Requirements for the Degrees of Master of Science in Technology and Policy and Master of Science in Aeronautics and Astronautics

Abstract Acquisition programs increasingly use model-centric approaches, generating and using digital assets throughout the lifecycle. Model-centric practices have matured, yet in spite of sound practices there are uncertainties that may impact programs over time. The emergent uncertainties (policy change, budget cuts, disruptive technologies, threats, changing demographics, etc.) and related programmatic decisions (e.g., staff cuts, reduced training hours) may lead to cascading vulnerabilities within model-centric acquisition programs, potentially jeopardizing program success. Program managers are increasingly faced with novel vulnerabilities. They need to be equipped with the means to identify model-centric program vulnerabilities and determine where interventions can most effectively be taken. In this research, Cause-Effect Mapping (CEM), a vulnerability assessment technique, is employed to examine these vulnerabilities. Using a combination of literature investigation, expert interviews, and usability testing, a CEM is created to represent the novel vulnerabilities posed by model-centric practices in acquisition programs. Particular attention is paid to cybersecurity vulnerabilities, which pose a serious threat to the successful implementation of model-centric practices. From this CEM, key gaps in program manager knowledge and organizational policies are identified and potential responses proposed.

Thesis Supervisor: Donna H. Rhodes Title: Principal Research Scientist, Sociotechnical Systems Research Center

Thesis Supervisor: Daniel E. Hastings Title: Cecil and Ida Green Education Professor of Aeronautics and Astronautics and Engineering Systems

3

God, grant me the insight to find and use models to understand the world around me, The wisdom to acknowledge that they will someday fail, And the strength to rid myself of them when it is apparent they no longer work.

-inspired by Ze Frank & the Serenity Prayer

5

Acknowledgements

Before proceeding onto this work, I would like to thank several individuals. First and foremost is my wife, Rebecca, who has been unfailingly supportive of me throughout the endeavor that is an MIT graduate program. First in a long-distance relationship and then in person, she has consistently buoyed my hopes and sense of self-worth when I needed it most.

Next, I would like to thank Dr. Donna Rhodes, one of my thesis advisors. She has taught me a great deal about how to think critically about my research work and how to present it to an audience. Her level of engagement and enthusiasm for my work has been much appreciated. I regret only that I believe that this thesis insufficiently represents what she enabled me to accomplish. As I transition into a doctoral program and may no longer be working for her, I look forward to continuing to work with her.

Thanks also goes to Prof. Daniel Hastings, for agreeing to serve as my AeroAstro advisor despite not knowing me too well and while working in Singapore. I can only hope that this thesis proves worthy of his trust.

My fellow researchers at SEAri deserve gratitude as well. In particular, I would like to thank Sarah Rovito, Parker Vascik, Shane German, Lucie Reymondet, and Brian Mekdeci (though I never met him). This work is built upon theirs (as a count of the number of times I cite them and the number of hours going over their theses can demonstrate). Lucie in particular set the stage for the emergence chapter, though she never published on the topic. Beyond mere citations, their support was integral as peers, in classes, at conferences, and (in Shane’s case) in diagnosing that, yes, in fact my hand was broken and I needed to go to the doctor.

I would like to thank my fellow housemates at pika. I do not believe that I have ever lived in a true community until I joined this continuing experiment in cooperative living. Thanks for (almost) always having dinner ready at the end of the day and for filling the house with laughter.

Finally, I would be remiss without acknowledging the financial support for this research from the MIT-SUTD Fellowship Program, the Systems Engineering Research Center, and the Naval Postgraduate School. Without their generosity, this work literally would not have happened

7

Table of Contents

ABSTRACT ...... 3

ACKNOWLEDGEMENTS ...... 7

TABLE OF CONTENTS ...... 9

LIST OF FIGURES ...... 13

LIST OF TABLES ...... 15

LIST OF ACRONYMS AND INITIALISMS ...... 17

CHAPTER 1 : INTRODUCTION ...... 19

1.1 MOTIVATION ...... 19

1.2 RESEARCH APPROACH ...... 19

1.3 RESEARCH QUESTIONS ...... 20

1.4 SCOPE ...... 20

1.5 RESEARCH CONTRIBUTION...... 21

1.6 THESIS STRUCTURE ...... 22

CHAPTER 2 : DEFENSE ACQUISITION AND MODEL-CENTRIC ENGINEERING .. 23

2.1 WHAT IS DEFENSE ACQUISITION? ...... 23 2.1.1 Historical Acquisition ...... 23 2.1.2 Acquisition in the 20th Century ...... 24 2.1.3 Problems in Acquisition ...... 25

2.2 WHAT IS MODEL-CENTRIC ENGINEERING/ACQUISITION? ...... 28 2.2.1 Aspects of MCE ...... 29 2.2.2 Examples of MCE Implementation ...... 33 2.2.3 Benefits of MCE ...... 35 2.2.4 MCE Implementation Barriers ...... 36

2.3 PROGRAM VS PROJECT, END-SYSTEMS ...... 37

CHAPTER 3 : VULNERABILITY ASSESSMENT ...... 39

3.1 DEFINITIONS ...... 39

9 3.2 METHODS OF VULNERABILITY, HAZARD AND RISK ASSESSMENT ...... 41 3.2.1 Existing Techniques ...... 41 3.2.2 Existing Frameworks ...... 51

3.3 CYBERSECURITY VULNERABILITIES IN MCE PROGRAMS ...... 59

CHAPTER 4 : APPLYING CEM-VA TO MODEL-CENTRIC ACQUISITION ...... 61

4.1 USING CEM FOR PROGRAMMATIC VULNERABILITY ASSESSMENT ...... 61

4.2 INTERVIEWS ...... 65

4.3 REFERENCE CEM ...... 67 4.3.1 Cybersecurity ...... 76

4.4 USABILITY TESTING ...... 80

CHAPTER 5 : EMERGENCE AND DESIGN PATTERNS ...... 85

5.1 DEFINITIONS ...... 85

5.2 CLASSIFICATIONS OF EMERGENCE ...... 86 5.2.1 Reducibility and Predictability ...... 87 5.2.2 Complexity and Degree of Interactions ...... 90 5.2.3 Expression ...... 92

5.3 DESIGN PATTERNS FOR EMERGENCE ...... 95

CHAPTER 6 : INTERVENTION AND POLICY RECOMMENDATIONS ...... 107

6.1 INTERVENTION RECOMMENDATIONS FOR PROGRAM LEADERS ...... 107

6.2 POLICY RECOMMENDATIONS FOR ACQUISITION ENTERPRISES ...... 109 6.2.1 Clearly Define Ownership over the MCE Environment ...... 109 6.2.2 Treat the Acquisition Program as an Engineered System ...... 115 6.2.3 The DoD should assume a leadership role in encouraging the development and definition of MCE standards within the defense acquisition industry...... 116

CHAPTER 7 : CONCLUSIONS ...... 123

7.1 RESEARCH QUESTIONS AND CONTRIBUTIONS ...... 123

7.2 LIMITATIONS ...... 125

7.3 FUTURE RESEARCH ...... 126 7.3.1 Interviews and Validation from Other Stakeholders ...... 126 10 7.3.2 Interactive Tool ...... 127 7.3.3 Healthcare Industry Comparison ...... 127 7.3.4 Cybersecurity ...... 128 7.3.5 Case Study Applications ...... 128

APPENDIX A (REFERENCE CEM) ...... 129

APPENDIX B (GLOSSARY)...... 139

REFERENCES ...... 143

11

List of Figures

Figure 1-1. Diagram of Research Approach ...... 20 Figure 2-1. Simplified Diagram of the Defense Acquisition Process. From [18] ...... 25 Figure 2-2. Costs escalations over time for major us Defense assets. Reprinted from [1], [2] ... 26 Figure 2-3. Semantic Web Technology compared at various layers of abstraction. From [33] .. 32 Figure 3-1. Simplified Fault-Tree Analysis of the sinking of the Titanic ...... 42 Figure 3-2. Portion of an FMECA. Figure adapted from [74] ...... 43 Figure 3-3. Example STPA Diagram. Image from [79] ...... 45 Figure 3-4. Example CEM for a maritime security system. From [82] ...... 47 Figure 3-5. Example CEM for a supply chain. From [81] ...... 48 Figure 3-6. Trusted Systems and Network Analysis Methodology. From [89] ...... 53 Figure 3-7. Steps for applying CEM to vulnerability analysis. From [70] ...... 55 Figure 3-8. The VAM Vulnerability Matrix. From [91] ...... 57 Figure 3-9. Cyber MAE as applied in the Defense Acquisition Process. From [94] ...... 59 Figure 4-1. System Dynamics Model of Employee Training Rate. Adapted from [115] ...... 63 Figure 4-2. System Dynamics Model of Accumulated Modeling Errors ...... 64 Figure 4-3. Reference CEM ...... 69 Figure 4-4. Types of External Triggering Events ...... 70 Figure 4-5. Section of a matrix representation of the MCE Reference CEM ...... 72 Figure 4-6. Example Vulnerability Chain with Intervention Point (in blue) ...... 75 Figure 4-7. Reference Cybersecurity CEM [Portion of Figure 4-3] ...... 78 Figure 4-8. Reputation Harm Vulnerabilities, section of Figure 4-7 ...... 79 Figure 4-9. Reference CEM used as a basis for usability testing ...... 83 Figure 5-1. Maier’s Types of Emergence with Examples. Adapted from [139] and [43] ...... 89 Figure 5-2. Ferreira’s Emergent Behavior Taxonomy. From [142] ...... 90 Figure 5-3. Bar-Yam’s Four Types of Emergence. From [143]...... 91 Figure 6-1. Excerpt of the Reference CEM highlighting the “active modeling” portion ...... 108 Figure 6-2. Objectives of the DoD M&S Master Plan. Adapted from [175] ...... 111 Figure A-1. Legend and Intervention Points of the Reference CEM ...... 133 Figure A-2. Part A of the Reference CEM ...... 134 Figure A-3. Part B of the Reference CEM ...... 135 13 Figure A-4. Part C of the Reference CEM ...... 136 Figure A-5. Full Matrix Representation of the Reference CEM ...... 137

14 List of Tables

Table 2-1. Examples of benefits and contributions of model curation. From [44] ...... 33 Table 4-1. Intervention Points for Reference CEM ...... 71 Table 4-2. MCE Reference CEM In-Degree/Out-Degree Tabulation ...... 73 Table 5-1. Fromm’s Taxonomy of Emergence [150]...... 93 Table 5-2. Mogul’s Proposed Typology of Emergent Behavior. Adapted from [151] ...... 94 Table 5-3. Mogul’s Proposed Typology of Causes of Emergent Behavior. Adapted from [151] . 95 Table 5-4. Example Design Pattern for Margin. From [82] ...... 96 Table 5-5. Summary information for three types of emergent behavior ...... 99 Table 5-6. Descriptions of three types of causal factors ...... 100 Table 5-7. Example proposed design pattern ...... 101 Table 5-8. Emergent Behavior Cross-Mapping ...... 102 Table 6-1. Summary of Model Package Development Structures ...... 121 Table A-1. Reference CEM Events and Descriptions (Part 1 of 4) ...... 129

15

List of Acronyms and Initialisms

ANSI American National Standards Institute CACE Capability for Accelerated Concurrent Engineering CAD Computer-Aided Design CCMC Community Coordinated Modeling Center CEE Collaborative Engineering Environment CEM Cause-Effect Mapping CEM-VA Cause-Effect Mapping for Vulnerability Analysis CIO Chief Information Officer CNC Computer Numerical Control CONOPS Concept of Operations COTS Commercial Off The Shelf CREATE Computational Research Engineering Acquisition Tools Environment CREATE-AV Computational Research Engineering Acquisition Tools Environment for Air Vehicles CVE Common Vulnerabilities and Exposures DAU Defense Acquisition University DFARS Defense Federal Acquisition Rules Supplement DMbE Digital Model-Based Engineering DMDII Digital Manufacturing and Design Innovation Institute DoD Department of Defense DSM Digital System Model ESD Engineering Systems Division ETA Event Tree Analysis FMEA Failure, Modes, and Effects Analysis FMECA Failure, Modes, Effects, and Criticality Analysis FRAND Fair, Reasonable, And Non-Discriminatory FTA Fault Tree Analysis GE General Electric GSFC Goddard Space Flight Center IDSS Institute for Data, Systems, and Society INCOSE International Council on Systems Engineering IP Intellectual Property ISO International Organization for Standardization JCIDS Joint Capabilities Integration and Development System JPL Jet Propulsion Laboratory KA Knowledge Assessment LS Logistics Support LSA Lead Standardization Activity M&S Modeling & Simulation M&S SC Modeling & Simulation Steering Committee M&SCO Modeling & Simulation Coordination Office MAE Mission Assurance Engineering MBSE Model-Based Systems Engineering MCA Model-Centric Acquisition MCE Model-Centric Engineering

17 MDD Material Development Decision MIT Massachusetts Institute of Technology MITRE [This is not an acronym but is in fact the name of an organization] MSFC Marshall Space Flight Center NASA National Aeronautics and Space Administration NAVAIR Naval Air Systems Command NDA Non-Disclosure Agreements NIMA NASA Integrated Model-Centric Architecture NIPRNet Non-classified Internet Protocol Router Network NIST National Institute of Standards and Technology OWL Web Ontology Language PM Program Manager/Management PPBES Planning, Programming, Budgeting, and Execution System PPBS Planning, Programming and Budgeting System RAND [This is not an acronym but is in fact the name of an organization] RMF Risk Management Framework SCIF Sensitive Compartmented Information Facility SE Systems Engineer/Engineering SERC Systems Engineering Research Center SIPRNet Secret Internet Protocol Router Network SRPO Sounding Rocket Program Office STPA Systems-Theoretic Hazard Analysis SWIFT Structured What If Technique SWT Semantic Web Technology T&E Testing & Evaluation TSN Trusted Systems & Networks UAS Unmanned Aircraft System UEP Undesirable Emergent Property USD (A&S) Under Secretary of Defense for Acquisition and Sustainment USD(AT&L Under Secretary of Defense for Acquisition, Technology, and Logistics USD(R&E) Under Secretary of Defense for Research and Engineering VA Department of Veterans Affairs VAM Vulnerability Assessment & Mitigation

18 Chapter 1: Introduction

The topic of this thesis is the assessment and mitigation of programmatic vulnerabilities in model- centric engineering defense acquisition programs. This chapter lays out the motivation, scope, and contributions of the work, as well as specifying the structure of the rest of the thesis.

1.1 Motivation

Cost of major defense acquisition projects are rising across the military services and delays have become increasingly common [1], [2]. This is in the midst of (and potentially caused by) rapidly rising complexity of defense systems and systems of systems. In an effort to take advantage of modern computing capabilities to address this issue, model-centric engineering has arisen as a potential solution. This concept, which is related to but separate from the successful model-based systems engineering, would involve an unprecedented degree of modeling integration. It has obvious benefits and equally obvious barriers of implementation, both of which are being thoroughly considered by academia, government agencies, professional societies, industry, and working groups [3]–[7]. Beyond this, however, model-centric engineering is sure to introduce new vulnerabilities into the acquisition process. Supporters of model-centric engineering and skeptics alike have been primarily focused on the barriers of implementation. As a result, little attention has been given to these vulnerabilities and how they should best be approached. Furthermore, one of the primary goals of model-centric engineering is to enable the effective design and acquisition of intentionally complex systems that take full advantage of beneficial emergent behavior while suppressing potentially harmful emergent behavior. The model-centric engineering environments themselves are essentially tools for the creation of positive emergence, such as system-level optimization, but are thus potentially susceptible to negative emergence as well. This thesis seeks to address these new vulnerabilities and also provides useful ways to conceptualize both vulnerabilities and emergence.

1.2 Research Approach

This research was characterized by two loops, both of which can be seen in Figure 1-1. In the first, expert interviews fed into an understanding of both the model-centric engineering ideal and the current state of the program management practice. The necessary differences between these two were used to generate the Reference Cause-Effect Mapping which in turn provided insight into

19 model-centric engineering vulnerabilities. These recommendations were then validated by further interviews. The other loop involved the usability testing, which was used to both assist in defining the current state of the practice, and in validating the causal chain concept.

Figure 1-1. Diagram of Research Approach

1.3 Research Questions

This work was guided by the following central research questions:

1. What new programmatic vulnerabilities will model-centric engineering introduce in defense acquisition programs? 2. Are causal chains a useful way to conceptualize and approach these vulnerabilities? 3. What methods can individuals and enterprises use to mitigate these new vulnerabilities?

1.4 Scope

This thesis primarily deals with the topic of vulnerability assessment as related to model-centric engineering acquisition programs, and defense acquisition in particular. It aims to provide a reference model and enable insight into the ways various exogenous factors impact model-centric defense acquisition programs and what can be done to mitigate these impacts. Additionally, it is

20 hoped that this work can help researchers in academia and policymakers in acquisition enterprises determine how best to structure acquisition programs to avoid certain vulnerabilities.

This thesis does not seek to directly tackle all of the challenges facing defense acquisition and bring us into a glorious new era of aligned incentives and good governance. Overcoming all of the challenges facing the defense acquisition system as it transforms into this new paradigm is a mammoth task that will require innumerable researchers, policymakers, industry partners, and uniformed service members to address. Even this is small compared to the major changes occurring in more general engineering and program management practices. While this thesis discusses some of these broader issues, it specifically seeks to contribute new findings that will hopefully play some small part in improving defense acquisition.

1.5 Research Contribution

This research provides several contributions to the body of knowledge concerning vulnerability assessment and the assessment of programmatic model-centric engineering vulnerabilities in particular. First and foremost, it advances the conceptualization of vulnerabilities as causal chains and uses results from usability testing to demonstrate the effectiveness of this conceptualization. The causal chain concept also enables new means of sorting and categorizing vulnerabilities, enabling the identification of effective interventions and heuristics. Second, this work explores and identifies programmatic vulnerabilities in model-centric engineering environments. While significant attention has been paid to the potential benefits of model-centric engineering and to its implementation challenges, much less consideration has been given to the new vulnerabilities that it introduces. This work is a step in that direction. Additionally, this work develops a typology of emergence and proposes a method for the creation of design patterns that jointly provide a common vocabulary for discussing the issues of emergence that can arise both within the model-centric engineering environment and in end-systems.

Tying these contributions together is the Reference Cause-Effect Mapping, which visually displays model-centric engineering vulnerabilities as causal chains, and allows the identification of a number of intervention points available to program managers.

21 Finally, this work provides general recommendations to program managers and several specific policy recommendations for MCE acquisition enterprises. These policies should serve to mitigate and obviate some of the model-centric engineering related vulnerabilities.

1.6 Thesis Structure

Chapter 1 lays out the goals, structure, and contributions of this work. Chapter 2 provides an overview of the defense acquisition system and of model-centric engineering. This includes both their history and their current status as it pertains to this work. Chapter 3 surveys the existing vulnerability assessment techniques and frameworks, including Cause-Effect Mapping. Chapter 4 lays out how Cause-Effect Mapping was applied to the model-centric engineering environment and what methods this research took to gather information and validate results, as well as the results generated by these methods. Chapter 5 explains emergent behavior and the use of design patterns in addressing such behavior, providing a common language for discussing emergence, and its potential relevance to model-centric engineering related vulnerabilities. Chapter 6 discusses general lessons and recommendations for program managers and model-centric engineering enterprises. Chapter 7 recaps the research findings, revisits the research questions, and discusses future work in this vein.

22 Chapter 2: Defense Acquisition and Model-Centric Engineering

Defense acquisition has existed long before model-centric engineering. Similarly, vulnerabilities have existed in both acquisition programs and in end-systems since the beginning of acquisition. Before proceeding into discussing the novel vulnerabilities related to model-centric engineering and how they can be mitigated, this chapter examines the historical development of defense acquisition (Section 2.1) and the transformation that it is undergoing. (Section 2.2).

2.1 What is Defense Acquisition?

Acquisition is the general term used by the United States Department of Defense (DoD) to refer to the lifecycle process of most engineered systems, as opposed to purely commercial off the shelf (COTS) products such as pencils or paper. The Defense Acquisition Glossary defines it as

“The conceptualization, initiation, design, development, test, contracting, production, deployment, integrated product support (IPS), modification, and disposal of weapons and other systems, supplies, or services (including construction) to satisfy DoD needs, intended for use in, or in support of, military missions.” [8]

Acquisition is an integral part of modern warfighting and is becoming even more important as engineered systems grow more complicated, technology advances, and the contexts of operation become more diverse. It is also something that has changed immensely over the decades (and centuries).

2.1.1 Historical Acquisition

For most of history and in most places, equipment, weapons, and military uniforms were provided by the soldiers themselves. In many places, this meant that social and economic class determined the type of soldier a person could be. The aristocracy were the commanders, the wealthy were the armored cavalry, the middle class were infantry, and the poor were often unable to serve at all. During this period, acquisitions largely did not exist. The Roman general Gaius Marius was the first to introduce standardized equipment provided by the state into the Roman Republic in 107 BC [9]. While this change, along with Marius’s other military reforms, is commonly credited as

23 being one of the primary causes for Rome’s later military successes, the practice did not find widespread acceptance in Europe until the Industrial Revolution.

Even when this did become the norm, such as during the 18th and 19th centuries [10], the acquisition process was mostly informal. It relied primarily on handwritten, relatively simple contracts, calling for the delivery of item X in quantity Y for cost Z by a certain date. In the US, these were often directly and specifically ordered by Congressional legislation, with minimal oversight over the actual fulfillment process. The general process was that a military officer or civilian member of the War Department would solicit a few bids, privately evaluate them, and then write a contract. Over the course of the 19th century, as technology improved, Congress got more directly involved in directing the application of these technological gains to military use, including standardized parts in firearms [11]. The implementation of these congressional commands had minimal oversight [12], however, and, in general, the traditional process was largely sufficient until the first World War.

2.1.2 Acquisition in the 20th Century

The world wars of the early 20th century resulted in innumerable lessons for the militaries of the world. Among them were the power of science and engineering in the design of military equipment, the importance of standardization, and the need for an infrastructure to support the design, development, and operation of increasingly complicated systems. These lessons, coupled with the results of the US and British militaries’ experiments in Operations Research, led to the development and application of first Systems Analysis [13] and then Systems Engineering [14] (though the exact timeline is fuzzy and also involved Bell Telephone Laboratories [15]). As the field of systems engineering coalesced, so too did the defense acquisition process, with a unified Department of Defense acquisition policy being put in place in over the course of the 1950’s and 1960’s [16]. This structure continued to evolve over the decades since, gradually becoming more unified [17].

The modern defense acquisition structure includes three interrelated components: the Joint Capabilities Integration and Development System (JCIDS); the Planning, Programming, Budgeting, and Execution System (PPBES); and the Defense Acquisition Process [18]s. Broadly speaking, JCIDS identifies the capabilities needed by the military and validates the requirements

24 for the system to be acquired. PPBES determines and allocates the resources necessary to successfully acquire the system. Finally, the Acquisition Process actually oversees the design, development, production, operation, and disposal of the system.

The Defense Acquisition Process is broken down into three phases (Pre-Systems Acquisitions, Systems Acquisitions, and Sustainment) and has three major milestone decisions (though, confusingly, these do not match up with the three phases). Throughout the process are a number of more minor decision points and reviews. Figure 2-1 shows a simplified timeline of this process. The full process is not easily displayed on a single page [19].

Figure 2-1. Simplified Diagram of the Defense Acquisition Process. From [18]

2.1.3 Problems in Acquisition

The resulting defense acquisition processes enabled the procurement of increasingly complicated and higher-performing assets over the course of the 20th century. They were unable, however, to control rapidly rising unit costs, particularly of major assets, as can be seen for surface naval combatants and fixed-wing fighter aircraft in Figure 2-2. This trend was noticed fairly early on, with Norman Augustine writing wryly in 1986 that “In the year 2054, the entire defense budget will purchase just one aircraft. This aircraft will have to be shared by the Air Force and Navy 3- 1/2 days each per week except for leap year, when it will be made available to the Marines for the extra day.” [20]. While these trends are not unique to defense projects, they are particularly prevalent and severe in this domain [21].

25

Figure 2-2. Costs escalations over time for major us Defense assets. Reprinted from [1], [2]

The underlying causes of these issues are multitudinous. For instance, multiple incentive problems exist. On the military side, acquisition projects are typically managed by uniformed officers. These officers sometimes have little experience in large-scale project management. To further exacerbate matters, they are given little chance to acquire much experience as these posts typically last one to three years, where the acquisition project itself can span a period of five to twenty-five years. Additionally, their future promotion and career status is somewhat dependent on the apparent results in the acquisition project accomplished during their tenure. This can result in hurried

26 prototypes and rushed requirements definition, all of which can be overturned by the next program manager. [22].

On the industry side, different, but still harmful, incentives are pervasive. As mentioned earlier, most large military acquisition contracts are cost-plus, meaning there is little incentive to minimize costs. Additionally, there are typically few companies able to provide a given capability, resulting in monopolistic markets. Even when there is competition, it can be difficult to separate procurement from maintenance and operations, as the costs of switching from one company to another between lifecycle stages can be enormous. Private company careers typically are much longer than the military program managers, resulting in a disparity of experience and knowledge, further skewing incentives [23].

Just as these trends were initially noticed many years ago, efforts to curb the increases date back decades. Such efforts have taken multiple forms, including educative, with the founding of the Defense Acquisition University (DAU) in 1991 [24]; organizational, with the establishment of the Under Secretary of Defense for Acquisition in 1986 [17]; and enterprise-level, with the implementation of the Acquisition Improvement Program reforms following the Packard Commission [17] and the occasional switch to and from fixed cost and cost-plus contracts [24]. While these attempts at reform may have been ameliorative, the increase in costs has only continued to accelerate.

Some current efforts are focused externally to the acquisition programs. For instance, the International Council on Systems Engineering’s (INCOSE’s) Program Management – Systems Engineering Integration Working Group is working on “changing the acquisition game” including the incentives for underestimating costs and downplaying problems [25].

Beyond these attempts to more or less directly address the problematic incentives underlying these issues, others have attempted to work within the system, to provide tools and minor policy changes to curb the symptoms, if not cure the disease. Many of these tools have been developed by the systems engineering community, and include multi-attribute utility theory[26], multi-stakeholder tradespace exploration [27], epoch-era analysis [28], SysML [29], among others. While these tools have made not managed to reign in increasing costs and schedule delays as of yet, they have enable improved operational capability and reliability. There is reason to believe, however, that a more

27 complete embrace of these tools, along with increased model integration and usage, could, in fact, result in significant improvements to cost and schedule as well. This possibility, sometimes referred to as model-centric acquisition, is discussed in 2.2.

2.2 What is Model-Centric Engineering/Acquisition?

The ongoing transformation of acquisition has many aspects: Model-Based Systems Engineering (MBSE) [30], Digital Systems Model (DSM) [31], Digital Twin [32], Digital Model-Based Engineering (DMbE) [4], and Digital Thread [3], among others. Advances in computing, digital workflows, and multidomain-multiscale models are leading to new concepts and approaches for model-centric engineering ([3], [4], [30]–[32]. These terms all refer to different aspects of how these new capabilities may be integrated into modern engineering and acquisition practices. One such term, Model-Centric Engineering (MCE), has been defined as “An overarching digital engineering approach that integrates different model types with simulations, surrogates, systems and components at different levels of abstraction and fidelity across disciplines throughout the lifecycle.” [33]

MCE involves using integrated models across disciplines, subsystems, lifecycle stages, and analyst groups. It uses models as the “source of truth,” thereby reducing document handoff and allowing for more continuous evaluation of tradespaces and system performance. These in turn mean reduced communication time and rework in response to requirement changes, more reuse from project to project, an ability to push designs closer to their limit, and potentially predictive, rather than prescriptive, maintenance cycles.

While many engineering organizations are already testing and applying various aspects of MCE [32], [34], [35], implementation is not without its difficulties. More infrastructure is needed (such as model curation). Increased connectivity means the danger of improper access is heightened. Additionally, even once MCE practices are implemented, difficulties remain, as will be discussed in Chapter 4.

MCE is not solely applicable to defense acquisition. However, this work is primarily focused on MCE in defense acquisition, though some of the findings extend to other domains.

28 2.2.1 Aspects of MCE

MCE is not a singular, specific target to be reached. Rather it refers to the general transformation of the engineering environment and process. This transformation is and will be accomplished with a variety of methods, depending on the enterprise. Several notable aspects of MCE are discussed in this section, or order to provide the reader with a picture of the overall MCE concept.

2.2.1.1 Model-Based Systems Engineering

Model-Based Systems Engineering (MBSE) can be viewed as either the first wave of the broader MCE transformation or as a wholly independent endeavor. MBSE refers to the change of systems engineering methodology to rely on representing requirements and system architecture as models rather than as documents and drawings [36]. The push for MBSE has resulted in the rise of widely used tools such as SysML. The continuing success of MBSE likely contributed to the desire for the broader use of models that is MCE.

2.2.1.2 Digital Systems Models / Digital Thread

The Digital Thread (DTh) is analytic framework for integrating technical data, costs, predictions, and other accumulated knowledge over the course of design, development, and operation of a system. It is intended to provide ready access to usable information for decision makers during the design process. It includes tools such as tradespace analysis and visualization methods. The term “digital thread” is sometimes referred to by other terms, for example, Lockheed Martin Corporation has used the term Digital Tapestry [35]. The DoD defines DTh as:

“An extensible, configurable and component enterprise-level analytical framework that seamlessly expedites the controlled interplay of authoritative technical data, software, information, and knowledge in the enterprise data- information-knowledge systems, based on the Digital System Model template, to inform decision makers throughout a system's life cycle by providing the capability to access, integrate and transform disparate data into actionable information.” [8]

The Digital System Model (DSM) is essentially the proposed product of DTh [37]. It is the integrated model of all technical data out of which individual Digital Twins will be constructed 29 and the technical grounding that the decision-making analytics of the DTh refers to. The DoD defines DSM as:

“A digital representation of a defense system, generated by all stakeholders that integrates the authoritative technical data and associated artifacts which define all aspects of the system for the specific activities throughout the system lifecycle.” [8]

2.2.1.3 Digital Twin

A Digital Twin is an integrated model of an as-built system including physics, fatigue, lifecycle, sensor information, performance simulations, etc. It is intended to reflect all manufacturing defects and be continually updated to include wear-and-tear sustained by the system. The goal is to more effectively and safely manage the individual product as well as to facilitate investigations of potential design or operational changes on the health of the system. The DoD defines Digital Twin more specifically as:

“An integrated multiphysics, multiscale, probabilistic simulation of an as-built system, enabled by Digital Thread, that uses the best available models, sensor information, and input data to mirror and predict activities/performance over the life of its corresponding physical twin” [8]

The Digital Twin can be used to simulate planned missions in order to improve mission design. Additionally, it could be used to predict when maintenance of various components is necessary, thereby having a more individually tailored maintenance schedule.

2.2.1.4 Model Integration and Composability

The descriptor “integrated” is used in different ways by different authors. Some seem to mean that the Digital Twin would be able to simultaneously simulate the system in multiple domains and at multiple scales [3]. Others seem to mean more that the integrated model would track all changes in components and propagate them out to all of the domain models [5]. The former is computationally more difficult but also would provide much greater gains in simulation accuracy.

30 Beyond the computational power required to integrate models of multiple domains and scales, there exists real issues of integration and composability. Different domains rely on fundamentally different assumptions and concepts. Often, even within a single domain, different conceptions of the world exist at different scales. In crack growth, for example, the very point of a crack is best modeled using quantum mechanics, the area surrounding this by molecular dynamics, and the overall material using continuum mechanics. While it is possible to develop algorithms to effectively combine these different scales into one model [38], this is nontrivial, must be done for each such “handoff,” and is epistemologically questionable [39]. Some of these issues are discussed further in Section 2.2.4.2

These technological difficulties are not the only hurtles posed by model integration and composability. Numerous organizational and human factors issues exist as well. Currently, many models are created by individuals or teams of analysts with specific uses in mind. In order to ensure composability, either standards would need to be implemented or models would need to be centrally created. Additionally, serious issues of trust exist when multiple models are composed together into one entity that no individual completely understands.

2.2.1.5 Semantic Web Technologies

Semantic Web Technologies (SWTs) refers to various extensions of the World Wide Web through standards. Several SWTs are of potential relevance to MCE, such as integrating the Web Ontology Language (OWL) with SysML, as proposed by JPL [40]. Figure 2-3 shows a diagram of some various SWTs and their level of abstraction. The general idea behind an integration of OWL and SysML, for instance, would be to make use of OWL’s ability to check consistency, satisfiability, entailments, completeness, and well-formedness. Active work by the Systems Engineering Research Center (SERC) is underway to extend JPL’s work [33].

31

Figure 2-3. Semantic Web Technology compared at various layers of abstraction. From [33]

2.2.1.6 Multidisciplinary Design, Analysis, and Optimization

One of the primary potential benefits of MCE is for a greater degree of system-level analysis and optimization. To accomplish this, it is necessary to be able to design, analyze, and optimize across disciplines. This is what the field Multidisciplinary Design, Analysis, and Optimization (MDAO) aims to do. Active work is being done to integrate single-discipline models, develop generic multidisciplinary models of various useful systems such as UASs [33], and develop better methods of visualization of the large tradespaces that result from such practices as this [41].

2.2.1.7 Model Curation

In order to address these hurtles, some researchers have proposed the creation of a model curation infrastructure [42], [43], that would both ensure that models could be effectively integrated and take whatever steps are necessary for ensuring the proper levels of trust exist in the models. Model curation involves a number of important functions, including setting and administering model- related policies; authenticating, preserving, classifying, and organizing models according to their metadata; and overseeing the data management practices in the enterprise. Some of the benefits and contributions of model curation can be seen in Table 2-1. 32 Table 2-1. Examples of benefits and contributions of model curation. From [44]

2.2.2 Examples of MCE Implementation

Aspects of MCE are being tested and implemented at a variety of points along the acquisition process in different organizations. A few examples of such implementations are discussed in this section.

2.2.2.1 F-35 Joint Strike Fighter

The development of the F-35 as part of the Joint Strike Fighter Program used many aspects of MCE, including “Man-In-The-Loop” simulations [45] and designing the control surface behavior [46], as well as a more general DTh that extended throughout the program [47]. Despite this, however, there was substantial difficulty in effectively modeling the software aspects of the system and this resulted in significant delays [48]. Overall, the F-35 may very well serve as a cautionary tale of the potential consequences of attempting to implement too many aspects of MCE too fast [49].

33 2.2.2.2 US Air Force and Naval Air Systems Command

The US Air Force and the US Naval Air Systems Command (NAVAIR) have been pushing towards further development and use of MCE practices on a variety of fronts. They are two of the primary supporters of the CREATE-AV program. CREATE-AV (Computational Research and Engineering Acquisition Tools and Environments for Air Vehicles) is one of the three parts of the broader CREATE program which over the past decade has sought to provide more integrated engineering toolsets for DoD acquisitions programs. In addition to tool development, the CREATE program has also involved shadowing various acquisition programs, involving multiple branches and sites [50].

The Air Force has been actively seeking to use CREATE-AV to construct a DTh that is continuous throughout the acquisition process [51]. This includes conceptual design (using the DaVinci program of CREATE-AV), detailed wing design (using either the fixed wing Kestrel or the rotary wing Helios), and airframe-propulsion integration (using Firebolt) [52].

Beyond this, NAVAIR has been actively involved with the development of the SWTs discussed in Section 2.2.1.5 [33], a vocal advocate of MCE at government-industry forums [5], and a supporter of using SERC research to develop graphical UAS CONOPS [53].

2.2.2.3 NASA

Within the NASA Jet Propulsion Laboratory (JPL) is Team X, an advanced concept design team charged with rapidly generating space mission concepts. While Team X has existed for more than 20 years, it has been continually improving. It was NASA’s original concurrent engineering team and has recently been a hub MCE testing, including visualization methods for teams of engineers with different disciplines. Additionally, JPL has been involved in developing useful ontologies for SysML and other MBSE processes [54] that have found uses elsewhere [33]. Most of these efforts exist on the early “Pre-Phase A” side of the acquisition process. JPL has even made these ontologies and tools publicly available (and are actively updating them as of time of writing) [55].

Outside of JPL, the NASA Sounding Rocket Program Office (SRPO) has been conducting a multi- year endeavor to develop their MCE capabilities across multiple NASA centers, both in terms of

34 tools and in trained engineers. This includes shadowing existing sounding rocket projects as a means of validating the MCE tools and processes across the entire acquisition lifecycle [56].

At Marshall Space Flight Center (MSFC), effort has been made to apply more MCE practices during the architecture phase, called NIMA (NASA Integrated Model-Centric Architecture [6]. Additionally, the multi-agency partnership known as the Community Coordinated Modeling Center (CCMC), based out of Goddard Space Flight Center (GSFC) seeks to provide coordinated verification of models, access to standardized data formats, and training on various models [57]. While they do not currently seek to set model integration standards in furtherance of MCE, they are a promising potential venue for such activities.

2.2.3 Benefits of MCE

The benefits of increased use of MCE are numerous and impact every stage of the acquisition lifecycle. Some of the most apparent are listed here:

Exploration of System Trade Space: Having an integrated model (whether unified or federated) enables more effective exploration of the system trade space and thus a greater degree of system level optimization [3], [5], [58].

Accelerated Production, Test & Evaluation: By having standards for models that stretch over the acquisition process, redundant effort in re-modeling between groups and lifecycle stages can be eliminated. An example of this would be “Direct CAD-to-CNC” prototyping, [3].

System/Component Life Estimation: Once a detailed integrated model of a system exists, it can be used to simulate operational environments and predict necessary maintenance. If the model was kept updated over the course of the life of the system, as in a Digital Twin, it could thereby enable a switch from prescriptive to predictive maintenance, reducing failure rates and extending the life of the system [59].

Design Reuse: Currently many designed components and subsystems remain locked in documents after their creation. What models exist in standard libraries often only model a particular domain of a component (such as its circuitry or thermal characteristics, for instance). Having an integrated

35 model of the entire system also means having integrated models of components, components that can be easily “drag-and-dropped” into a new system in the future [58].

2.2.4 MCE Implementation Barriers

Introducing MCE into an enterprise’s practices is not a trivial matter. Multiple barriers exist in implementation. Some of these are technical and some are more social or non-technical. Several examples of each category are discussed in this section.

2.2.4.1 Technical

Technical challenges are those that are primarily issues of current technological capabilities and engineering effort. Two particular such changes are:

Lack of standard operation architectures and common standards: Currently there are no extant standards for integrating models across domains, across levels of fidelity, or between organizations. This is a widely recognized problem [3], [5], [31], [58] that is not purely an issue of policy. Rather, significant technical expertise is required to enable data to be passed across domains or scales, due to the significantly different assumptions and conceptions of the physics at play, as discussed in Section 2.2.1.4.

Adequate Computer Processing Power: Even should models be effectively integrated, conducting full simulations of a trade space or operational performance may be difficult with current computing capabilities [3]. Even once simulated, displaying and processing the resulting information in a useful manner is a non-trivial problem [41].

2.2.4.2 Non-Technical

Non-technical challenges are those that are primarily cultural, economic, or social, rather than technology-based. This barriers to MCE implementation are well recognized [3], [4], [31]. Two particular categories are:

Intellectual Property (IP): [31]. IP in this context refers to any piece of information owned or held in some way by a person or organization. In the defense industry, this includes patents, software (or algorithms more generally), material data, business approaches, trade secrets, etc. A

36 key characteristic of IP is that, while information can be easily shared, value exists in information asymmetries (i.e. possessing information or IP that others do not). Since the construction of such a digital framework fundamentally requires the integration of IP from multiple sources, issues of valuation, compensation, and protection of IP arise [31]. The DoD is well aware of the IP issues raised by MCE and has previously discussed potential means around these barriers with representatives of industry [4], [5].

Knowledge Assessment (KA) and Cultural Inertia: KA refers to the assignment of validity to any particular piece of information or expertise. This can be a technical process, such as double- checking statistical methods used to evaluate the results of an experiment, or a more social process, such as deciding which news source to trust. KA issues include “buy-in” (i.e. trust) both by technical staff directly creating the models and by decision-makers in the acquisition and operation processes [60], as well as inter-model validity assessment in the cases of multiple models operating in the same domain (e.g. two crack-growth modeling methods). Some have argued that the issue of cultural inertia may be the primary barrier to implementing MCE [3]. Model Curation is one proposed solution to addressing these KA concerns [44]. Similarly to the IP barriers, the DoD is well aware of these KA challenges and already working to overcome them [4].

2.3 Program vs Project, End-Systems

As a brief aside, it is worth noting the differences between a program and a project. The Defense Acquisition Glossary defines a program as “A directed, funded effort that provides a new, improved, or continuing materiel, weapon or information system, or service capability in response to an approved need”. This is the definition of program that will be used in this work. A project, on the other hand is acknowledged as being synonymous with program in general usage, but more specifically defined as “a planned undertaking having a finite beginning and ending, involving definition, development, production, and logistics support (LS) of a major weapon or weapon support system or systems. A project may be the whole or a part of a program.” [8] MCE as a concept is applicable to both programs and projects. Additionally, many aspects of MCE are equally applicable to both, particularly as many projects exist within a program. Nonetheless, this work will focus on programs. Additionally, the term end-system is used throughout this work to refer to the system that an acquisition program is acquiring.

37

Chapter 3: Vulnerability Assessment

In Chapter 2, the increasing complexity of military end-systems and the resulting changes to the acquisition process were discussed. As this increase in complexity occurs, it becomes increasingly necessary to institute formal methods of assessing risks and vulnerabilities. In the modern era, there are many such methods, each with their own focus, strengths, and weaknesses. Prior to discussing MCE-related vulnerabilities and how they can be assessed in Chapter 4, this chapter defines exactly what vulnerabilities are and lays out several existing methods of vulnerability assessment. At the end of the chapter there is on a specific note about cybersecurity vulnerabilities.

3.1 Definitions

Numerous methods and frameworks for analyzing vulnerabilities, risks, and hazards exist. These three interrelated terms have different definitions depending on the field and on the method of analysis. In this work, a hazard refers to a system or environmental state that has the potential to disrupt the system. Examples include the existence of an iceberg at sea and tired operators. Hazards may not result in system failure, partly depending on the design of the system (such as the inclusion of an automated collision avoidance subsystem).

A vulnerability is the means by which the hazard might disrupt the system, thus it is through the vulnerability that the system is susceptible to the hazard. Vulnerabilities are best expressed as the causal series of events connecting a hazard to system failure. This is a generalization of common, field-specific usages of the term. MITRE’s Common Vulnerabilities and Exposures (CVE) database, for example, defines a vulnerability as “a weakness in the computational logic (e.g., code) found in software and some hardware components (e.g., firmware) that, when exploited, results in a negative impact to confidentiality, integrity, OR availability” [61]. The DoD defines a vulnerability as “The characteristics of a system that cause it to suffer a definite degradation (incapability to perform the designated mission) as a result of having been subjected to a certain level of effects in an unnatural (man-made) hostile environment.” [62] All of these definitions are related in the common dictionary definition: “Capability of or susceptibility to being wounded or hurt.” [63] In all these definitions, the same components can be seen: some structural means or “weakness” that can result in system disruption or “negative impact” if a hazard is present or the vulnerability is “exploited.” For example, the infamous Spectre security vulnerability is described 39 in CVE as “Systems with microprocessors utilizing speculative execution and branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side- channel analysis” [64]. This is a neat summary of the hazard (an attacker), the means (side-channel analysis using speculative execution and branch prediction), and the disruption (unauthorized disclosure of information).

Risk is a measure of the probability of a system disruption and the consequences of that disruption. The common dictionary definition is simply “exposure to the chance of injury or loss.” [65] The DoD however defines it more closely to how engineers and scientists typically use the term: “Probability and severity of loss linked to hazards.” [62] The Defense Acquisition Glossary merely applies this definition to the acquisition domain: “Future event or condition that may have a negative effect on achieving program objectives for cost, schedule, and performance. Defined by 1) the probability (greater than 0, less than 1) of an undesired event or condition, and 2) the consequences, impact or severity of the undesired event, were it to occur.” [8] It is commonly expressed with just a statement of those two components: 1.25 deaths per 100 million vehicle miles. Sometimes risk is instead expressed as a multiplication of likelihood and consequence and can include other components such as detectability.

It should be noted that the above delineations between hazard, vulnerability, and risk are not universally recognized, either across or within disciplines [66], [67]. That said, the definitions stated above are how they shall be used in this work.

Two other relevant but blurry distinctions exist in the literature. The first is that between assessment and analysis. These terms are often used interchangeably, including within the same work (e.g. [68]–[70]). Similarly, this work will make no clear distinction between the two, other than that analysis will be used primarily regarding quantitative methods and assessment will be used primarily for qualitative methods. Since many methods include both quantitative and qualitative aspects, do not take this distinction to be any more than a general connotation.

The other distinction that requires discussion is that of method, framework, and technique. This is a distinction not previously made, to the author’s knowledge, but is worth making here. In this work, the word method will be used according its common usage as “a procedure… or way of doing something,” or “a manner or mode of procedure.” [71] Framework will be used to refer to a

40 method of assessing and analyzing hazards, vulnerabilities, and risks in a comprehensive manner, as well as determining appropriate mitigations and countermeasures. Examples of frameworks include Trusted Systems and Networks Analysis and Vulnerability Assessment & Mitigation Methodology. These and other frameworks are discussed in Section 3.2.2. A technique, on the other hand, will refer to a method for analyzing a particular set of hazards, vulnerabilities, or risks. Examples of techniques include Fault Tree Analysis and the Structured What-If Technique. These and other techniques are discussed in Section 3.2.1.

3.2 Methods of Vulnerability, Hazard and Risk Assessment

Numerous methods of analyzing vulnerabilities, hazards, and risks have been developed over the years. Some were developed with particular issues in mind. Fault-Tree Analysis was invented to be used on the Minuteman Guidance System, for instance [72]. Whatever, their original intent however, many of them have come to be used widely and are often parts of an engineer’s undergraduate education.

In general (though not always), the frameworks consist of four distinct parts.

1. Threat/Hazard Identification and Assessment 2. Vulnerability Identification and Assessment 3. Risk Identification and Assessment 4. Mitigation and Prevention Identification and Assessment

In this way, vulnerability assessment is one part of a more general framework, typically occurring after hazards have been identified but before risks are analyzed [73]. Techniques are often used within frameworks as the means of identifying and/or analyzing specific vulnerabilities or risks. Some existing techniques and frameworks are discussed in this section.

3.2.1 Existing Techniques

In this section, several risk and vulnerability analysis techniques are discussed. Many of these techniques are used as part of the broader frameworks discussed in Section 3.2.2. For an explanation of the distinction between techniques and frameworks, see Section 3.1.

41 3.2.1.1 Fault Tree Analysis and Event Tree Analysis

Fault Tree Analysis (FTA) is a deductive, top-down analysis method where a failure mode is identified and all the possible causes of that event are laid out in sequences until the exogenous hazards are reached. Logic gates are used to connect the various hazards and intermediary events. An example FTA may be seen in Figure 3-1. Probabilities may be assigned to each hazard and thus a cumulative probability of the failure calculated. FTA is thus quite proficient in investigating the cause of failures afterwards, but is limited in its ability to prospectively identify all possible hazards. Additionally, it is somewhat limited by its arbitrary stopping point (i.e. where one chooses to define an event as an exogenous hazard).

Figure 3-1. Simplified Fault-Tree Analysis of the sinking of the Titanic

Event Tree Analysis (ETA) is essentially an inverted FTA. Instead of starting from a failure and working backwards to a hazard, a hazard is selected and logic gates are used to assess potential consequences. This method is useful for predicting potential failures rather than determining the cause of a previous one. It suffers from some of the same limitation as FTA. Additionally, it fails to examine the consequences of multiple concurrent hazards.

42 3.2.1.2 Failure Modes and Effects Analysis (FMEA)

FMEA and the closely related Failure, Modes, Effects, and Critical Analysis (FMECA), are inductive methods, similar to ETA, that seek to tabulate all possible failures and then assessing their severity, probability, detectability, and criticality. It excels at thoroughness but suffers from an inability to easily assess multiple failures simultaneously. Additionally, its tabular format can be difficult to read. An example FMECA can be seen in Figure 3-2.

Function Dispense amount of cash requested by customer

Potential Failure Does not dispense cash Dispenses too much cash Mode Potential Effect(s) of Customer Incorrect entry Discrepancy Bank loses Discrepancy in Failure dissatisfied to demand in cash money cash balancing deposit system balancing Severity Rating 8 6 Potential Cause(s) of Out of Cash Machine jams Power failure Bills stuck Denominations Failure during together in wrong trays transaction Occurrence Rating 5 3 2 2 3 Current Process Internal low-cash Internal jam None Loading Two-person Controls alert alert procedure visual (riffle ends of inspection stack) Detectability Rating 5 10 10 7 4 Risk Priority Number 200 240 160 84 72 Criticality 40 24 16 12 18 Figure 3-2. Portion of an FMECA. Figure adapted from [74]

3.2.1.3 Systems Pathology

Systems Pathology (as it pertains to systems engineering) is an attempt to apply the medical research concept of Systems Pathology [75] to engineered systems. The basic idea is to study the architecture of the system and determine what dysfunctions, or “pathologies,” may occur based on that architecture [76]. Some example pathologies that have been identified, along with the architectures that give rise to them are [77]:

 Cyberpathologies (dysfunctions of feedbacks)  Rheopathologies (dysfunctions of systems flows) 43  Cyclopathologies (dysfunctions of cycling, oscillations)  Heteropathologies (dysfunctions of hierarchy or modular structure)  Hapsopathologies (dysfunctions of network structure or dynamics)  Teratapathologies (dysfunctions in developmental pathologies)  Stathopathologies (dysfunctions in stability states)

This is a relatively new technique that is still actively being developed and formalized [78].

3.2.1.4 Systems-Theoretic Hazard Analysis (STPA)

STPA takes a rather different approach and conceptualizes systems as control loops, as can be seen in Figure 3-3. The goal of STPA is to avoid focusing on exhaustively tabulating all vulnerabilities and attempting to quantitatively calculate probabilities. These can be difficult to do accurately for a system of any significant size or for risks that are sufficiently rare. Instead, STPA attempts to ensure that appropriate monitors and controls are in place for each component of the system (including its operators) so that any hazard is detected and addressed before it can cause a failure. In this way, it seeks to eliminate vulnerabilities while relying primarily on a qualitative, rather than a quantitative, assessment.

44

Figure 3-3. Example STPA Diagram. Image from [79]

The STPA process consists for four iterative steps [79]:

1. Establish the system engineering foundation for the analysis and for the system development 2. Identify potentially unsafe control actions 3. Use the identified unsafe control actions to create safety requirements and constraints 4. Determine how each potentially hazardous control action could occur.

Most of vulnerability analysis methods fail to directly grapple with the problem of blame. Humans, engineers and program managers included, have a tendency to assign blame for a failure to someone or something other than themselves. FTA, ETA, and FMECA can enable this by allowing for an arbitrary “stopping point” (i.e., where the previous step in the causal chain is deemed the initiating hazard). In the Titanic FTA presented in Figure 3-1 of Section 3.2.1.1, for instance, why did the causal deconstruction halt there? Were the designers of the rudder actually at fault? Or

45 were the engineering standards poorly written? Were the owners of the boat at fault for installing too few lifeboats or should the government set a minimum required number of lifeboats? By adjusting the bounds of the analysis, it is easy to place blame on whomever the analyst desires.

STPA avoids this by (a) not assigning a specific “cause” of a failure and (b) by having every part of the system responsible for monitoring the other parts. Despite this, as the creators of STPA themselves acknowledge, the method has been criticized for its lack of a neat, one-page explanation of the causes of an accident [79].

3.2.1.5 Cause-Effect Mapping

Cause-Effect Mapping (CEM), is a vulnerability identification method developed at MIT (where it was originally called “Cause and Effect Mapping”) [80]. CEM has been previously applied to a maritime security system [80] (as seen in Figure 3-4), energy grids [70], and supply chains [81]. It consists of a mapping of causal chains that connect an exogenous hazard to a system degradation or failure, termed a terminal event. Causal chains are a series of events, with each event causing or being an integral part of the cause, or the next “link” in the chain. Each chain represents a vulnerability, sometimes called a vulnerability chain in order to emphasize that vulnerabilities are not discrete events. Terminal events are broadly defined and include any form of value loss.

46

Figure 3-4. Example CEM for a maritime security system. From [82]

Similar to FTA, CEM is easily read in either direction, but it also allows for the simultaneous consideration of multiple failures and multiple hazards. Another difference between the two is that CEM allows for causal loops, such as in Figure 3-4. Unlike STPA, however, it also does not necessitate causal loops can be seen in the supply chain CEM shown in Figure 3-5.

47

Figure 3-5. Example CEM for a supply chain. From [81]

48 The initiating hazards are external to the perspective of the defined user, and are thus sometimes called external triggers. An intermediary event is any unintended state change of a system’s form or operations which could jeopardize value delivery of the program and is situated along a causal chain connecting an external trigger to a terminal event.

A CEM is not created objectively for a system, but for a specific class of decision-maker. The external triggers (referred to as “spontaneous events” in Figure 3-4 and Figure 3-5) are exogenous from the point of view of the designated decision-maker. This means that a CEM created for an operator is different than a CEM created for a program manager. The differences will depend on what actions are available to each role and what hazards are exogenous to their domain. In this way, CEM avoids the “blaming someone else” problem discussed in Section 3.2.1.4 by making all initiating hazards exogenous. The decision-maker only has control over the intermediary events. While she may not be at fault for any of the vulnerabilities, it is still her responsibility to address them.

CEM is fundamentally a qualitative analysis technique, though it can be readily adapted into a quantitative form, by adding probabilities of transition to each intermediary. CEM provides immediate insight into which parts of the system warrant more detailed modeling. For instance, it may be useful to determine the likely time required for a specific vulnerability to complete. The representation of vulnerability as vulnerability chains emphasizes that vulnerabilities are not discrete events. This representation also improves the user’s understanding of the relationships between multiple vulnerabilities. Individuals commonly default to thinking of threats or vulnerabilities as discrete events, rather than fully grappling with their cascading nature. CEM enables this tendency to be overcome.

CEM enables classification of different vulnerability chains (by terminal event, by triggering event or type of triggering event, or by intermediary event) in various, useful ways. For instance, the triggering events in Figure 3-5 were color-coded according to their “domain.” The three domains of triggering event are defined as follows [70]:

 Force Majeure: This is a general term for an event that is the result of actions beyond the possibility of control by not only the CEM-user perspective, but also of their entire

49 organization. In this case, both cyber-attacks and natural disasters are wholly beyond the control of the organizations involved in the supply chain.  Policy: An event that is the result of intentional governmental decisions  Economic/Resource: An event that is the result of general economic or resource- constraint issues, not any one organization.

Note that these domains are not to be considered the sole domains used in CEM-VA, only examples of potential domains. Triggering event domains should be tied to some useful purpose in the assessment of vulnerabilities, such as identifying leading indicators. The domains of external triggers used in Section 4.3, for example, are different.

Additionally, CEM allows immediate identification of potential intervention points at intermediary events where multiple vulnerability chains intersect. It can either be used as a stand-alone vulnerability assessment technique, or as part of a broader assessment framework (see Section 3.2.2.2). Previously, it has been used to assist in generating a tabulation of vulnerabilities [80] and to assist in identifying leading indicators of vulnerabilities [81].

3.2.1.6 Structured What If Technique (SWIFT)

SWIFT is, as the name suggests, a lightweight and fast technique for vulnerability identification by a group. It has been applied in both the chemical process industry and in healthcare [83]. It consists of two primary steps: selection of guide words and repeated questioning. Guide words are a way of framing the relevant vulnerabilities to be considered. In healthcare these can include sets like “time, timing, or speed,” “place, location, site, or environment,” or “person or people.” In chemical processes, they include sets such as “operating errors and other human factors,” and “process upsets of unidentified origin.” Once the SWIFT facilitator(s) has selected the guide words, the participants are then challenged to ask “What if” and “How could” questions that address the guide words for each system, subsystem, and process under consideration. For example, if the guide words were “time, timing, or speed” in a healthcare context, a question might be something along the lines “What if the patient is told about his or her condition while still

50 sedated? How could this happen?” By then answering these questions, participants are identifying vulnerabilities in the system.

3.2.1.7 Delphi Method

The Delphi Method, or Delphi Technique, is a form of collective, expert judgement. It was first developed by the RAND Corporation in the mid-20th century [84] . The process involves selecting a panel of experts in the field being considered and repeatedly surveying them, providing selected information to the panelists between rounds. More specifically, the steps are generally as follows

1. Select a panel of experts. 2. Solicit each expert’s judgement on a question or set of questions (such as the likelihood of failure of a specific project). Allow them access to whatever resources they desire, but do not inform them of the identity of the other panel members or allow them to communicate with one another. 3. Collect the initial responses and solicit a response again. This time, provide some information on the distribution of initial responses and potentially require outliers to justify their deviance. 4. Repeat Step 3, providing various amounts of anonymized data about the distribution of responses and the arguments for particular stances to the panel. 5. Once convergence has been reached, stop the questionnaires.

Note that the convergence in Step 5 is not necessarily to a single answer. Previous uses have resulted in convergence to a bimodal distribution in some cases [84]. The Delphi Method, and variants of it, have found use in applications such as technology forecasting [85], schedule risk assessment [86], and cost prediction [87]. The structure of the technique is such that it is best applied to analyzing specific risks or vulnerabilities, rather than seeking to conduct a comprehensive assessment.

3.2.2 Existing Frameworks

In this section, several risk and vulnerability assessment frameworks are discussed. Many of these frameworks make use of the various techniques discussed in Section 3.2.1. For an explanation of the distinction between techniques and frameworks, see Section 3.1. For a more detailed treatment

51 of these and other frameworks, see Rovito [70]. Of note in these frameworks is that, with the exception of CEM-VA, all were developed with primarily defense applications in mind. As a result, they tend to focus on specifically intentional and malicious threats, which behave quite differently than the primarily unintentional and stochastic hazards conceptualized by many of the techniques discussed in Section 3.2.1. CEM and CEM-VA straddle this boundary, examining both stochastic hazards, such as the Natural Disaster or Resource Demand Restriction of Figure 3-5, and the intentional Rocket Attack of Figure 3-4 and Cyber Attack of Figure 3-5.

3.2.2.1 Trusted Systems and Networks (TSN) Analysis

TSN Analysis is the DoD’s risk assessment framework of choice to “minimize the risk that DoD’s warfighting mission capability will be impaired due to vulnerabilities in system design or sabotage or subversion of a system’s mission critical functions or critical components.” [88] As such, it is an assessment framework focused on preserving end-system integrity, and is particularly concerned with ensuring the integrity of COTS products and the supply chain in general [89]. As shown in Figure 3-6, it consists of several inter-related activities: a criticality analysis that identifies the most crucial parts of the system, a threat assessment to identify likely forms of hazards, a vulnerability assessment, a risk assessment that ties the previous components together, and finally a cost-benefit-based selection of countermeasures / risk mitigations.

52

Figure 3-6. Trusted Systems and Network Analysis Methodology. From [89]

The TSN is a fairly adaptive framework. For instance, it does not specify any particular vulnerability assessment technique to be used, though it does recommend six in particular:

1. Milestone A vulnerability assessment questionnaire 2. Vulnerability databases (such as CVE) 3. Static analyzer tools and other detection techniques 4. Component diversity analysis 5. Fault Tree Analysis 6. Red team penetration testing

Due to this flexibility, success has been had integrating TSN with other vulnerability assessment techniques and frameworks, including CEM-VA [70].

3.2.2.2 Cause-Effect Mapping for Vulnerability Analysis (CEM-VA)

CEM-VA is a vulnerability assessment framework based off of CEM developed by Rovito [70]. It consists of four primary steps, shown below in Figure 3-7. Generating and using a CEM can be

53 done in different ways and to different levels of granularity, depending on the need of the stakeholder. This process can be done with groups, such as project teams, as well as individually. The four steps shown are expanded upon here:

1. The stakeholder consults lessons learned databases, case studies, and other experts to generate hazards, terminal events, intermediary events, causal connections, and interventions. During this step, the system being assessed should be well-defined. 2. The gathered information is used to identify and list the terminal events that would result in immediate value loss. Potential causes of these events are identified and added them to the map. This is repeated using backwards induction for the newly identified events until an external triggering event is reached. 3. Causal connections between each intermediary event are then identified and the CEM is examined to see if there are any additional connections not previously noticed. During this step, the induction process may reverse, moving from external triggering events to and working “forwards” to determine on how they might impact the program. 4. Finally, potential interventions and leading indicators for the various vulnerability chains are identified. Mitigation strategies are planned and implemented.

54

Figure 3-7. Steps for applying CEM to vulnerability analysis. From [70]

The CEM technique and CEM-VA framework have been applied and validated in a variety of contexts. Prior to this work, some methods of validation have included:

 A stochastic, dynamic, discrete-event simulation of a marine security system of systems CEM [82]  Subject-matter expert evaluation of a microgrid electrical distribution system CEM [70]  Subject-matter expert evaluation of a supply-chain variant of CEM-VA [70]

In this work, usability testing of the Reference CEM with program managers and systems engineers was used as the primary source of validation [90], as is discussed in Section 4.4.

All of these have been relatively small-scale, both in terms of numbers of participants and in the CEMs being evaluated. As of the time of writing, no full-scale engineering program has used CEM-VA in any significant way. In general, the previous uses have focused on CEM as a means of generating insight that can then be provided to program managers and systems engineers, rather than as a method to be directly used by program managers. As a result, while CEM and CEM-VA have been demonstrated to be potentially powerful and useful tools, they should not be viewed as mature methods or as capable of supplanting established methods such as FTA or TSN Analysis.

55

3.2.2.3 Vulnerability Assessment & Mitigation (VAM) Methodology

The VAM methodology was developed by the RAND Corporation in 2004 [91]. While it was designed with information systems in mind, it is intended to be broadly applicable and as user- friendly as possible, as evidenced by its freely available Excel tool [92] and its pocket guide aimed specifically at large-scale interagency exercises at the operational and theater levels [93]. It consists of 6 steps:

1. Identify your organization’s essential information functions. 2. Identify essential information systems that implement these functions. 3. Identify vulnerabilities of these systems. 4. Identify pertinent security techniques to mitigate these vulnerabilities. 5. Select and apply techniques based on constraints, costs, and benefits. 6. Test for robustness and actual feasibilities under threat.

Steps 3-6 are intended to be cycled through multiple times. The actual vulnerability identification is guided by a matrix, seen in Figure 3-8. The intent of this matrix is to force the evaluator to consider all permutations of object type and vulnerability type. The VAM methodology also provides for a comprehensive set of mitigation methods for each type of vulnerability. It furthermore has an accounting process for what new vulnerabilities these mitigation methods can introduce.

56

Figure 3-8. The VAM Vulnerability Matrix. From [91]

57 3.2.2.4 Cyber Mission Assurance Engineering (MAE) Methodology

The Cyber MAE Methodology, which was developed by the MITRE Corporation, is specifically oriented at reducing cyber risks in line with NIST’s Risk Management Framework. It is designed to be flexible and capable of evaluating an enterprise, a mission, or an individual system. The methodology consists of five components, as listed below [94].

1. Establishing mission priorities 2. Identifying mission dependencies on cyber resources 3. Performing a mission (or business) impact analysis 4. Performing a threat susceptibility analysis 5. Analyzing alternative cyber risk remediation alternatives for effectiveness, efficiency, and affordability

What is not immediately apparent from these steps is that MAE is focused on assessing vulnerabilities in terms of the types of threat posed. MITRE breaks cybersecurity threats into three main categories: low-end “hackers” or “script kiddies,” mid-range state sponsored that use many of the same techniques as low-end, and high-end that do use lower-end techniques but also will create custom hardware and software, as well as planting or turning insiders in an organization. The lower-end threats are dealt with by tedious but straightforward practices such as end-to-end application of firewalls, anti-virus protection, and vulnerability patching tools. The mid-range and high-end threats are handled using sensitive and classified countermeasures, targeted and deployed according to the Cyber MAE Methodology [95].

When it comes to the acquisition process, Cyber MAE operates across the acquisition lifecycle, as shown in Figure 3-9. For instance, Crown Jewels Analysis is used just after the Material Development Decision (MDD) to identify critical resources and then the threat susceptibility assessment occurs just prior to Milestone A [94].

58

Figure 3-9. Cyber MAE as applied in the Defense Acquisition Process. From [94]

3.3 Cybersecurity Vulnerabilities in MCE Programs

Risk and vulnerability assessment techniques and frameworks have not failed to adapt to novel cybersecurity concerns. As far as techniques are concerned, the aforementioned CVE database has been public since 1999. Quality Assurance testing (essentially the verification and validation of software) has been around since the beginning of commercial software. Software penetration testing (where security experts intentionally seek to break a software product) has been the industry norm for more than a decade [96]. Black-box mutational fuzzing and concolic execution are being used to automatically test for certain types of software vulnerabilities [97]. Formal verification tools, initially limited to pure software domains such as cryptography [98], have been rapidly advancing and finding applications in hardware [99] and business processes [100], as well as fields that straddle the software-hardware-environment boundaries [101]. These techniques just scratch the surface of approaches security researchers and engineers are taking to identify and resolve such technical cybersecurity vulnerabilities.

Beyond these specific techniques, assessment frameworks have progressed as well. As discussed in Section 3.2.2.4, the Cyber MAE Methodology was recently developed with cybersecurity threats specifically in mind [94]. Additionally, the System-Theoretic Process Analysis (STPA) has been adjusted, adapted, and applied to handle cybersecurity vulnerabilities associated with additive 59 manufacturing [102], Internet of Things [103], Air Operations [104], and Mission Operations [105]. More recently, there have also been efforts to combine compiler technology with STPA to automatically detect vulnerabilities in software-controlled systems [106].

While cybersecurity vulnerabilities in operational systems remain alarmingly common, from the trivial [107] to the critical [108], there is some evidence that software is becoming more secure [103]. In many cases, however, the acquisition process itself (as opposed to just the end-system) needs to be protected from both outside threats and endogenous failures. Be it military information or technology-related trade secrets, there is real value in attempting to penetrate much earlier in the life cycle in order to either steal secrets [109], [110] or to disrupt production [111] .

Defense acquisition programs have already instituted a variety of means of ensuring the security of their work. Some of these were originally instituted to address other forms of threats but have turned out to be effective in addressing cybersecurity as well. Examples include relying on the security clearance process, the use of Sensitive Compartmented Information Facilities (SCIFs), restrictions on the use of media storage devices, separate networks such as SIPRNet and NIPRNet that are isolated or semi-isolated from the internet, and general compartmentalization of critical information. Some (non-US) defense agencies have gone so far as to revert to using typewriters where able in order to avoid security breaches and leaks [112].

Unfortunately, many of these historically successful methods are in conflict with the more straightforward implementations of many aspects of an MCE environment. For example, the use of SCIFs has been quite successful in preventing unauthorized access to data. The typical use of a SCIF in the design process, where a small number of engineers work on a task isolated from the outside world, is not directly compatible with an MCE environment structured around model integration and collaboration across teams. While this problem has been previously considered and ways to mitigate this conflict have been proposed (e.g. [31]), no silver bullet resolving these tensions exists. It is likely that the increased use of MCE will result in both the exacerbation of current vulnerabilities and the creation of new ones. Most means of assessing such vulnerabilities are aimed at assisting software and systems engineers to identify and remove cybersecurity vulnerabilities from the end-system. New methods for enabling project and program managers to perform cybersecurity assessments of their enterprise and engineering environment are needed.

60 Chapter 4: Applying CEM-VA to Model-Centric Acquisition

Despite the plethora of vulnerability assessment methods available, analyzing the vulnerabilities in an MCE acquisition process is non-trivial for a number of reasons. For one, no two acquisition programs are alike, so some form of generalized approach must be sought, potentially supplemented with case studies. Second, a fully-fledged MCE environment does not exist yet. As a result, some amount of this vulnerability assessment will be hypothetical and dependent on the final form that MCE takes in the defense acquisition system. Finally, due to the scale and sensitive nature of most (if not all) defense acquisition programs, it is difficult to conduct a thorough vulnerability assessment of a real program for academic research.

Due to the benefits of the causal chain conception of vulnerabilities, CEM-VA was chosen as the means of further studying MCE-related vulnerabilities. To overcome the above difficulties, a variety of approaches were taken. These included interviews with experts, the generation of a Reference CEM, usability testing of the CEM-VA framework, some basic analysis of the Reference CEM, and a discussion of a vulnerability typology. Each of these methods yielded their own insights, and when combined, yet more was learned. In this chapter, the use of CEM for assessing programmatic vulnerabilities is discussed before walking through each of the applied methods in turn. In Chapter 6 some of the broader implications of this research is discussed.

4.1 Using CEM for Programmatic Vulnerability Assessment

The vulnerability assessment techniques and frameworks discussed in Chapter 3 are most commonly applied either to the design or the operation of an engineered system, typically to improve its design or investigate a failure (even the example uses of CEM shown in Section 3.2.1.5 were of end-systems). However, these methods can also be applied to the program itself. Instead of hazards such as “relief valve failure” and “solar flare,” instead we have “hiring freeze” and “unexpected technological hurtle.” It can be difficult to assess likelihoods for such hazards, but even qualitative analysis can be useful. Similarly, terminal events are not “nuclear meltdown” or “loss of communications,” but instead “schedule delay” or “failure during verification/validation.”

A CEM can be used in an acquisition program to assess vulnerabilities in multiple ways and by different individuals. Four uses are described below:

61 (A) By a Program Manager: Assessing potential future vulnerabilities and planning possible interventions (B) By a Program Manager: Determining specific vulnerabilities to address in response to the presence of a specific hazard (C) By the Program Organization: Changing program processes to mitigate or eliminate vulnerabilities (D) By Researchers: Organizing and classifying vulnerabilities into various categories or types

All of these first require the creation of a CEM for the organization’s standard acquisition process or for a particular program. Once this is completed additional steps can be taken including:

1. Identifying notable intermediary events and potential intervention points 2. Conducting more detailed modeling of specific vulnerability chains 3. Classifying vulnerability chains to enable future study and potential mitigation.

While a program manager would be well served by the creation of a CEM specific to their own program, there is some benefit in using a Reference CEM for model-centric programs in general.

Use (A) is most relevant for novice program managers or program managers using MCE for the first time. A senior program manager or team of program managers creates a CEM for their organization’s program process. This CEM can then be provided to the novice for study and reference. The program manager can then learn what can go wrong and how to intervene. In this case, the CEM could be tied to a Lesson’s Learned database, such as NASA’s Lessons Learned Information System [113] . This enables concrete examples and consequences to be linked to each vulnerability. One of the important factors here is that the CEM does not just present potential interventions, but it also places them in the appropriate part of the causal sequence. This enables the program manager to not only know how to intervene, but at what point.

Use (B) is relevant to all program managers, regardless of level of experience. Once a hazard manifests, the program manager examines the CEM to assess potential consequences and options. He can then respond quickly to head off any cascading effects. This may require additional analysis

62 of a specific vulnerability chain or an individual intermediary event. System Dynamics is a method particularly useful for this due to the preexisting models of many organizational phenomena [114]. For instance, the Attrition, Reduced Model Training, and Less Model Expertise can be modeled by adapting the rookie fraction model shown in Figure 4-1 into the more MCE-relevant model shown in Figure 4-2. In this model, it is apparent that a hiring freeze (which would set the “Growth Rate” variable to zero) has no immediate impact, as rookies will continue to develop into experienced employees and model expertise will continue to accumulate. Over time, however, the dearth of new rookies will result in fewer experienced employees, increasing the error rate. This kind of long-term, indirect impacts are likely to become more common with increased use of MCE.

Figure 4-1. System Dynamics Model of Employee Training Rate. Adapted from [115]

63

Figure 4-2. System Dynamics Model of Accumulated Modeling Errors

Use (C) is the traditional use of vulnerability assessment methods: to improve a design or investigate a failure. The program organization can change policies or create infrastructure to either mitigate or wholly eliminate certain vulnerability chains. For example, if the organization elects to only use modeling software produced in-house, vulnerabilities pertaining to the availability of COTS software are no longer relevant. Such a change could be costly though or even introduce new vulnerabilities, so careful analysis is necessary. In this use, the CEM is a visual representation of a risk registry, tabulating all possible hazards to the program and mitigation choices made [116].

In Use (D), CEM is used to organize and classify vulnerability chains. Two obvious classifiers are terminal events and external triggers. Which is used to organize a CEM depends on whether the user wants to examine the causal chains forward or backwards. Beyond this, however, more complicated classifiers are possible. Once these groupings have been identified, they can be considered together and common means of intervention considered.

64 4.2 Interviews

Over the course of this study, a series of interviews was conducted with systems engineers and program managers from a variety of fields, including defense, aerospace, manufacturing, and semiconductors. These interviews sought to provide insight into the following questions, in context of a model-centric enterprise:

1. To what extent are program managers aware of programmatic vulnerabilities? 2. How do program managers conceptualize these vulnerabilities? 3. How do program managers respond to these vulnerabilities? 4. What vulnerabilities are present in MCE programs? 5. What cybersecurity vulnerabilities does MCE pose?

The first three questions provided some useful information regarding the status quo. The fourth and fifth questions were intended to supplement and corroborate a Reference CEM, as is discussed in Section 4.3 The initial questions and prompts asked of the interviewees were as follows, though it should be noted that the interviews were semi-structured, with the bulk of the time being spent on follow-up questions to these.

1. What industries do you have the most experience with? 2. What roles have you filled in those industries? 3. What types of uncertainties (e.g. policy change, budget cuts, disruptive technologies, changing demographics, etc.) are of concern in current or past programs that you have been involved with? 4. What programmatic decisions (e.g. staff cuts, reduced training hours) have you made or observed related to such uncertainties? 5. Have you experienced or do you expect any vulnerabilities or risks that are unique to or specifically exacerbated by model-centric engineering? If so, what mitigations have or will be used to address these? 6. More broadly, how do you or other program managers conceptualize and approach programmatic vulnerabilities? 7. Who do you view as the owner of the model-centric environment used by a program?

65 8. What do you think that researchers could do to improve the ability of program managers to handle vulnerabilities in a model-centric engineering program?

Across all the interviewees’ industries, several facts were clear in the responses. First, many, if not most, programmatic vulnerabilities appear to be triggered by exogenous hazards beyond the control of the program manager. Some, such as poor scope or inadequate budget, can even be present before the first program manager joins a program. In general, program managers are at least aware of the potential for these hazards and in some cases even see them coming. The interview responses showed some variance regarding responses to these hazards, however. Some program managers attempt to use informal practices like preemptive padding of a schedule using a multiplicative factor based on experience. Others used their own spreadsheets and tools for estimating the “real” schedule or cost (that is, the schedule or cost that would result from a potential hazard becoming real). Little to no formal risk or vulnerability assessment would take place however and responses tended to be reactive rather than proactive, contributing to the “program management via crisis management” paradigm. In general, the perception by interviewees was that program managers rely heavily on experience, rather than on formal education. There are attempts, such as by the INCOSE PM-SE Integration Working Group’s Strategic Initiative, to directly address the exogenous hazards [25]. Similarly, others seek to provide risk registries that contain programmatic risks [116], there is a real need for easy-to-use tools for program managers to improve their ability to assess programmatic vulnerabilities and respond to them.

When it came to the topic of cybersecurity vulnerabilities in particular, the interviewees commonly raised the following issues:

 Cybersecurity needs to be thoroughly considered much earlier in the program lifecycle than it commonly is, preferably in the proposal generation stage.  Program managers and systems engineers are sometimes intimidated by cybersecurity issues and thus seek to pass them on to specialists later in the acquisition process.  MBSE and MCE toolset developers have not done a thorough enough job of considering programmatic cybersecurity vulnerabilities, though the tools are typically quite effective at designing for cybersecurity in end systems.

66  Despite the three previous points, according to the interviewees, traditional programmatic cybersecurity defensive practices tend to be quite effective. Interviewees attributed this to the conservative approaches most defense-related engineering groups use as discussed in Section 3.3. The increased use of MCE, particularly for multi-site collaboration could change this.

The above points, which each had a great deal of consensus by the interviewed experts, are clearly nuanced and complicated, with both points of success and failure. In addition to these general lessons, the interviews also raised several of the vulnerabilities and interventions shown in Section 4.3.

4.3 Reference CEM

One of the objectives of this research was to develop a high-level cause-effect map for model- centric programs. The intent is for this CEM to serve as a both a collection of research findings and a reference for validation and further analysis. Additionally, it has the benefit of potentially serving as a standalone resource for such program managers, as well as a basis for organizations to construct their own, program-specific CEM with added detail.

The standard method of using CEM for vulnerability analysis (sometimes referred to as CEM- VA), as discussed in Section 3.2.2.2, was used in this work.

The Reference CEM was generated through a combination of methods. At this time, there is little literature on programmatic vulnerabilities posed by MCE. Most negative case studies, that is those that depict failures [117]–[119], and lessons learned databases [113] are from prior to the rise of MCE and thus deal with general vulnerabilities. Existing case studies that directly deal with MCE tend to be more positive, likely due to the rising popularity of the paradigm [120]–[122]. As a result of these, extrapolations from extant vulnerabilities had to be made, along with hypothetical inversions of the positive instances of MCE. Additional vulnerabilities were contributed by group brainstorming during the in-class activity discussed further in Section 4.4. All of these were supplemented and confirmed by the same interviews discussed in Section 4.2. Otherwise, the generation process followed the steps listed in Section 3.2.2.2.

67 Figure 4-3 shows the full Reference CEM generated. For improved legibility, the Reference CEM can also be seen separated into parts in Figure A-1 through Figure A-4 of Appendix A (Reference CEM). Additionally, a list of the events shown in the Reference CEM, along with descriptions of each one, can be seen in Table A-1 of Appendix A (Reference CEM). This CEM is not intended to be considered a complete or exhaustive account of all MCE-related vulnerabilities. Rather, it is a starting point for further analysis. In practical use of CEM-VA, program managers would create a more detailed CEM of their own MCE environment or use the Reference CEM as a general reference to better understand MCE-related vulnerabilities.

68

Figure 4-3. Reference CEM 69 The higher level of detail in the upper portion of the Reference CEM shown in Figure 4-3, which includes issues such as training, misunderstood model assumptions, and level of trust in the models, represents the increased degree of concern that interviewees had about these issues. In general, the domain of aligning culture and expertise with well-designed and well-documented toolsets was of high priority. Failure to accomplish this had led to significant problems in past projects, but was viewed as a surmountable difficulty moving forward.

The arrangement of the vulnerability chains is not random. External triggers that result in similar vulnerability chains are grouped together. By “similar,” we mean that these vulnerability chains either involve many of the same intermediary events or that they involve the same part of the program. For instance, most of the intermediary events involving model curation and trust are located close to one another in the center-top of Figure 4-3.

Additionally, as discussed in Section 3.2.1.5, the external triggers were classified into three different domains. These three domains, which are shown in Figure 4-4 are defined as follows:

 Force Majeure: This is a general term for an event that is the result of actions beyond the possibility of the program enterprise (not just the program manager) to influence. Thus it includes both malicious action and general, unforeseeable events such as Technological Change.  Policy: An event that is the result of intentional decisions made at the organizational or enterprise level. In the case of a government-run program, this includes oversight from Congress and the general public. Non-government organizations may still be impacted indirectly by such oversight, but their proximal triggering event would be different.  Private Sector: Any event that is the result of the actions of one or more private-sector firms outside the program enterprise.

Figure 4-4. Types of External Triggering Events

70 Once the Reference CEM was created, intervention points were identified. These are indicated as the small blue circle in Figure 4-3 and are detailed in Table 4-1 below.

Table 4-1. Intervention Points for Reference CEM

Point Number Intervention Action

1 Initiate Internal Assessment and a PR Strategy

2 Initiate various non-monetary benefits (such as allowing pets at work or a 9/80 schedule) to encourage employees to stay

3 Seek to share resources and employees with other programs

4 Hire employees with prior experience with the new software

5 Compartmentalize sensitive information

6 Obfuscate sensitive data with false or misleading information

7 Create documentation and curation processes within the program

8 Institute handover periods to benefit from contractor expertise

9 Reevaluate the training regime and needed fields of expertise

10 Increase the amount of testing conducted

11 Increase the use of contractors/consultants (including previous employees) to maintain expertise level

12 Reevaluate the requirements with the client and other stakeholders

13 Design for modularity to minimize impact on system

14 Negotiate with client / end-user to see if they are able to pay for the software

15 Maintain isolated but readily accessible back-ups of data

16 Conduct reviews/comparisons of models between lifecycle stages

17 Use multiple, independent simulations or component checkers

18 Maintain isolated, independent backup equipment that can be switched to while primary equipment is being evaluated

19 Conduct regular “red-team” / penetration test exercises

71 In order to better assess the relative importance of these interventions, the Reference CEM was transformed into a matrix representation, a portion of which can be seen in Figure 4-5 (the full matrix can be seen in Figure A-5 of Appendix A).

Figure 4-5. Section of a matrix representation of the MCE Reference CEM

Once the matrix was complete, it was then used to calculate the in-degree and out-degree of each event, including the external triggers and the terminal events. The results of this basic graph theory metric are shown in Table 4-2.

72 Table 4-2. MCE Reference CEM In-Degree/Out-Degree Tabulation

Event In-Degree Out-Degree Public Scrutiny 0 1 Congressional Scrutiny 0 1 Budget Cut 0 3 Economic Crunch 0 1 Hiring Freeze 0 1 List of Approved Modeling Software Changes 0 1 Modeling Software Becomes More Expensive 0 1 Modeling Software No Longer Supported 0 2 Strategic Realignment 0 1 Technological Change 0 1 Foreign Actor Action 0 1 Unexpected Technological Hurtle 0 2 Accidental/Malicious Data Release 0 2 Cyberattack Attempt 0 1 Sufficient IP Rights Not Acquired in Contract 0 1 Pressure to Streamline Acquisition Process 2 1 Reduction of "Upkeep" and Indirect Costs 1 3 Diminished Enterprise Model Curation Capability 1 3 Poor Model Documentation 1 2 Reduced Model Training 2 1 Delay in Integrating New Models into Suite 1 1 Layoffs/Incentivized Retirement of Experienced Staff 2 1 Unnoticed Modeling Software Flaw 1 1 Attrition 1 1 Less Model Expertise 4 3 Reduced Modeling Speed 1 1 Misunderstood Model Assumptions 3 1 Gap in Subject-Domain Knowledge 1 1 Inaccurate Simulations / Performance Predictions 4 1 New Software Created or Selected 1 1 Inability to Oversee Contractor Models/Designs 1 2 Over-Trust in Model 1 1 Preferred Modelling Software/System Unavailable 4 1 Under-Trust in Model 2 1 Under-Use of Model 1 1 Software Goes Without Updates 1 1 Needs Change 3 2 Missing/Improper Requirements 2 2 Change of Modeling Domain 1 1

73 Table 4-2. Continued from previous page

Event In-Degree Out-Degree Change of Modeling Software Needed 1 1 System Design Changes to Work Around Lack of Technology 1 1 Development Stall Due to Lack of Critical Technology 1 1 Theft/Exposure of Information 2 1 Countermeasures/Competitors Developed 1 1 Increased Testing Required by Customer 1 1 Current or Future Contracts Canceled/Avoided by Customer 1 1 Increased Cybersecurity Vulnerabilities 2 1 Reputation Harm 2 3 Industry Partners Unwilling to Share Information 1 1 Inability to Fully Integrate Models 1 1 Reduced Confidence in System 1 1 Successful Cyberattack 2 6 System Integrity Reevaluation 1 1 Loss of Historical Data 1 1 Inability to Reuse Models in Future Projects 2 1 Unnecessary Rework 1 1 Weak Security Controls 1 5 Documentation Tampering 1 1 Modeling Tampering 1 1 Physical Component Substitution 1 1 Malicious Insertion 1 1 Component Tampering 1 1 Delays 7 0 Failure During Verification/Validation 2 0 Over-conservative System 2 0 Field Failure 1 0 System Not Acquired 1 0 Loss of Technological/Strategic Advantage 1 0 Loss of Contract 1 0 Compromised System 3 0

From this table, it is apparent that Delays is the terminal event with the greatest in-degree. This is likely a product of two factors. First, due to modern systems engineering techniques, it is relatively rare for a program to fail outright. Instead, delays (and related budget overruns) are much more common. Second, program managers and systems engineers are perhaps more concerned with

74 delays than with other types of failures. Since the Reference CEM was partially based on interviews and usability testing involving program managers and systems engineers, it is likely to emphasize their concerns, much as the Penfield’s Homunculus represents the human body parts in proportion to their sensitivity [123].

Three intermediate events are tied for greatest in-degree: Less Model Expertise, Inaccurate Simulations / Performance Predictions, and Preferred Modelling Software/System Unavailable. Furthermore, it should be noted that these three events are closely tied together and exist along the several of the same vulnerability chains, additionally amplifying their likelihood. One such vulnerability chain is shown in Figure 4-6. Note that this is not the only vulnerability that includes all three, much less two of the three. This chain can be expressed verbally as the following:

A particular piece of simulation software that your company has used on similar projects in the past is licensed from another company. The license contract is up for renewal soon and the price might go up significantly. This could result in the preferred modeling software being unavailable for use in this program and a new software would need to be selected, one that your team has less (or no) experience with. Due to this lack of experience with the new software, assumptions underlying the model may be misunderstood by your analysts and thus inaccurate simulation results generated. This may not be noticed until either verification or validation when the system or subsystem does not behave according to the predicted performance levels.

Figure 4-6. Example Vulnerability Chain with Intervention Point (in blue)

75 This particular chain only has one identified intervention point (one that was generated during the usability testing discussed in Section 4.4). This means that the program manager needs to recognize when the external trigger is imminent or happening and act quickly to avoid serious disruption to the system. Alternately, it means that the program manager should invest time at the beginning of the program into identifying additional possible points of intervention along this vulnerability chain, in the event that it becomes relevant.

The intermediate event with the greatest out-degree, Successful Cyberattack is discussed further in Section 4.3.1.

While this analysis was quite simple, more sophisticated applications of graph theory and probabilistic modeling can be conducting using the Reference CEM if more information about a specific system is known. For instance, if probabilities, likelihoods, or time scales of each event transition are known, techniques such as Markov Chain Modeling, Monte Carlo Analysis, and Bayesian Networks can be brought to bear, weighting each arc of the graph instead of treating them equally.

4.3.1 Cybersecurity

The Successful Cyberattack event was the intermediary event with the greatest out-degree, as can be seen in Table 4-2. For this reason, it is worth discussing the cybersecurity-relevant portions of the reference CEM in more detail.

Some of the vulnerabilities and interventions shown in Figure 4-7 below (which shows a portion of Figure 4-3) are not unique to MCE environments. Some of the vulnerabilities will simply be exacerbated by increased use of MCE environments and processes. Some of the interventions will require new, creative means of implementing MCE. For instance, Intervention Point #1 in Figure 4-7 is Compartmentalize sensitive information. Clearly this is already done with the use of SCIFs and the Need-To-Know (NTK) information framework. However, such methods may not be feasible if the benefits of model integration and collaboration offered by MCE are desired. Instead new methods must be developed [124]. An example of one such possibility is the Federal Drug Administration’s (FDA) Sentinel Initiative, which involves querying a distributed system and receiving anonymized, aggregate data back [125]. Such a system may allow modeling software to

76 communicate across domains and locations, while still ensuring that even if one location is breached, only some information is exposed.

This Reference CEM does omit vulnerabilities and interventions that are present without MCE and are entirely unchanged by its adoption. For example, practices like the security clearance system and restricting the use of digital storage media will remain effective interventions that are not significantly impacted by MCE environments.

77

Figure 4-7. Reference Cybersecurity CEM [Portion of Figure 4-3]

78 One set of cybersecurity-related vulnerabilities that came up repeatedly in both the interviews and was observed in the usability testing data set were those that passed through the Reputation Harm intermediary event, as shown in Figure 4-8. Despite the frequency that the potential for this vulnerability was raised, few interventions were proposed for post-breach. This suggests that program managers and systems engineers could use more training in how to respond to breaches, particularly prominent ones, instead of just how to prevent them. While in the private sector there is evidence suggesting that the reputation harm incurred by a prominent breach does not significantly impact the firm [126], contractors to the government are known to suffer significant financial penalties due to breaches, even when such a breach is unrelated to their government duties [127], [128]. In a defense acquisition environment, there is thus significant incentive to having program managers (and the enterprise as a whole) well prepared to respond to major breaches.

Figure 4-8. Reputation Harm Vulnerabilities, section of Figure 4-7

CEM-VA is intended to supplement, rather than replace, existing vulnerability assessment frameworks, particularly when it comes to cybersecurity. In this way, it can help fulfill the requirements set by NIST’s Risk Management Framework (RMF) [129] and the DoD’s Defense Federal Acquisition Rules Supplement (DFARS) [130]. These regulations have shifted how government contractors handle cybersecurity. Previously, one-time assessments were completed and defensive practices instituted. Now the process is more dynamic. Contractors have to 79 continuously assess threats and develop countermeasures as they arise, both with regards to the end-system and to the enterprise. The Reference CEM can potentially assist in this, by serving as a reference that can be revisited as new threats arise.

4.4 Usability Testing

While several potential use cases were proposed in the previous section, due to the scale, duration, cost, and uniqueness of major MCE programs, it is difficult to systematically test the utility of either the Reference CEM or of CEM-VA in general. Some simple usability testing was explored in the interviews with experts. Usability was also considered through analyzing results of a scenario-based exercise on vulnerability analysis that had been generated in a graduate class activity involving techniques for investigating enterprises.

The classroom activity involved approximately 40 students from various backgrounds, most having prior experience in either industry or the military as systems engineers and/or program managers. The students were randomly divided into six groups of 5-7 students each, and each group was provided with the same “context” for the activity, as follows:

You are a project manager for a vehicle manufacturer. Your current project is designing a lightweight troop transport vehicle for the US military. It has a variety of high-tech components, including encrypted radio and satellite communication systems, an explosives detector, and night vision cameras. The design and testing process will take multiple years. Your company considers this a major project in terms of the resources put into it, the revenue received for it, and the potential for future military contracts. The military, as part of the contract, specified that the design and production process should predominately rely on models (sometimes called model-centric engineering) rather than written specifications.

Each group was provided with 1 or 2 selected external triggers to respond to. They were asked to discuss and record the potential negative impacts these hazards may have on the engineering environment and how they might act to mitigate these consequences. They were permitted to display these impacts and mitigations in any format they chose. The hazards provided were as follows: 80 1. An unrelated military project (at another company) to design a next-generation missile defense system has ended up in the national news. That system has gone significantly over budget, has been repeatedly delayed, and still looks like it is a long way off from being completed. Public accusations of mismanagement and waste are being made, including frivolous travel and lavish company events. Congress and the Department of Defense are now carefully scrutinizing all other major projects for potential mismanagement or waste, including your project. 2. After recent elections, there is significant political pressure on Congress to reduce federal spending. As a result of this, they are making significant cuts to many agencies and programs, including the military. The decrease in government spending is likely to impact your company’s other projects and may impact yours as well. 3. Government intelligence officials inform you that your company, and perhaps even your project, is likely to be the target of cybersecurity attacks based out of another nation. 4. This is your first project of this type (your prior experience was in designing civilian vehicles). You now have to choose whether your project will use the set of modeling software that you are accustomed to from the civilian projects or to use another set, more commonly used on military projects, but that you are unfamiliar with. 5. A recent economic downturn, coupled with a government that is cutting back on its military spending, has resulted in your company declaring a hiring freeze for an indeterminate amount of time. 6. Another project at your company is threatening to miss its deadlines. In order to get it back in line, its project manager is requesting that personnel from other projects, such as yours, be reallocated to hers. 7. The design requirements from the military that you are currently working with were put together with the idea of using this vehicle in a currently ongoing conflict. However, US involvement in this conflict is winding down and the military is currently unsure where this vehicle will be used in the future. The context (and thus the requirements) may change during the design process. Furthermore, your models were created with the current context in mind. 8. A recent economic downturn, coupled with a government that is cutting back on its military spending, has resulted in your company providing incentives for early retirement

81 to the more experienced, higher paid employees. You have no intention of retiring early yourself, but some of those working on your project might accept the offer. 9. In order to minimize rework and redundancy, your company has recently started an initiative pushing for increased reuse of components, designs, and models from one project to another. Your previous project also involved a night vision camera, but in a very different application context. 10. A particular piece of simulation software that your company has used on similar projects in the past is licensed from another company. The license contract is up for renewal soon and the price might go up significantly. You are uncertain if your company’s executives will approve the license renewal.

After a period of 20 minutes, the students were taught about causal chains and the use of CEM as a technique for investigating enterprise vulnerabilities. Each group was instructed to re-write the previously identified vulnerabilities and interventions as causal chains, and map them on the provided Reference CEM (Figure 4-9), as well as coming up with new ones. After another period of 20 minutes, their results were collected, a group debrief was given, and students shared general feedback on the class activity.

82

Figure 4-9. Reference CEM used as a basis for usability testing 83

Several useful pieces of information were garnered through analyzing the documented results of the usability testing that had been conducted. In the out-briefing material, the participants expressed unanimous agreement that using CEM and conceptualizing programmatic vulnerabilities as causal chains was helpful, though the perceived degree of usefulness varied from “slightly” to “extremely”. Additionally, team out-briefs reported on four primary forms of how the CEM helped in their assigned scenario:

1. Identifying high priority intervention points: (70%) 2. Identifying new vulnerabilities: (55%) 3. Understanding the causal path / Reframing the concept of vulnerabilities: (45%) 4. Understanding interrelationships between vulnerabilities: (40%)

These reported benefits were answers to an open-ended query, not selections in response to a multiple-choice question. The relative importance of this first point was corroborated by the group outputs that were generated (i.e. the actual vulnerabilities and mitigations identified). It was clear that in this instance, when provided with a Reference CEM, the groups tended to focus on identifying where and how to intervene in the vulnerability chains. During the first round (without the CEM), however, most of the groups presented their vulnerabilities and interventions as unordered lists of short phrases, typically unpaired (i.e. vulnerabilities were not matched with interventions). These short phrases were typically isolated events such as “team feeling more cautious” and “reputation damages.” The ultimate outcomes of these were assumed rather than explicitly stated. Once the CEM was introduced in the second round, the matching of interventions to vulnerabilities appeared to become much clearer, and most groups also identified additional interventions.

84 Chapter 5: Emergence and Design Patterns

The use of CEM in this work has been structured as a linear, causal network structure. That is, no feedback loops were used and each vulnerability chain was presumed to operate on a straightforward ABCD model. While CEM can accommodate feedback loops (see Figure 3-4 for an example), even then it presumes a direct and straightforward causal relationship between each intermediary event. This can sometimes be an oversimplification of reality.

One of the primary goals of MCE is to enable the design of more complicated end-systems that take full advantage of complex, non-linear behaviors that only emerge on the system level. These behaviors are commonly called emergent behavior or emergence. They are characterized by what Heise calls “mutual causation” [131] across multiple scales of the system. MCE not only enabled “design for emergence” in the end-system, but it also poses the potential of introducing novel forms of emergent behavior into the acquisition program itself. MCE results in more integrated models, reduced communication times, and increase re-use. All of these can contribute to feedback loops and cascading behavior, either beneficial or harmful to the program. For this reason, it is worthwhile expanding our understanding of emergent behavior and developing a framework for concretely grappling with it in engineered systems.

This work takes a step in that direction. This chapter surveys several of methods of definition and classification of emergence, discusses the different problems they seek to address, and proposes the development of a set of emergent behavior design patterns based upon a certain form of typology. The expectation is that these design patterns will enable practicing systems engineers and operators of systems to anticipate potential emergent behavior, more quickly identify it, and have ready access to tools to deal with it. This typology and its associated design patterns will thus serve as a supplement to the Reference CEM developed in Chapter 4.

5.1 Definitions

Classification, typology, and taxonomy are three terms used throughout this work and this chapter in particular. In common parlance. these terms are commonly used interchangeably. This work, however, will use Marradi’s definitions [132]. Classification is the generic term for sorting a set by some defined metric, either quantitative or qualitative. Systems of classification or merely classification will be used as a generic term, encompassing both typology and taxonomy, as well 85 as any other form of classification. Typology refers to a system of classification in which multiple means of division are simultaneously applied. As an example of this, when discussing government systems, a theocratic, authoritarian government has two means of classification applied and is the same as an authoritarian, theocratic government. A taxonomy, on the other hand, arises when multiple means of division are consecutively applied. The modern system of sorting living species, the evolutionary taxonomy, is, as the name suggests, a taxonomy. Thus Homo sapiens is not the same as a sapiens Homo.

The word emergence as used by systems engineers and scientists can be vague. This is likely because it is intended to refer to a wide variety of behaviors that are uncommon, hard to predict, often lack defined structure, and span multiple scales. Nonetheless, these behaviors do seem to share some intuitive similarities that tease at potential generalizable theories regarding them. Many have attempted to more clearly define what emergence should mean, as to enable productive investigation into them and allow those generalized theories to be generated, tested, and proven. Due to the disparate forms of emergence expressed, a natural way to approach this is not to deal with emergence as a whole, but to break it down into categories that can be more readily conceptualized and investigated. Thus classifications of emergence can be found in virtually any discussion of emergent phenomena.

Emergence in this work is not synonymous with complex systems. Complex systems are usually defined in terms of their scale, number of interacting parts, and non-linear behavior [133]. The former typically describes behavior, while the latter more typically (though not always) describes some formal aspect of the system itself. Clearly the two are closely linked with many complex systems exhibiting emergent behavior. Complex Adaptive Systems, for example, are noted for performing a certain emergent form of adaptation (as the name suggests) [134]. Due to this closeness, as well as the fact that neither emergence nor complex systems have definitive, universally agreed upon definitions, and this chapter discusses many of these various definitions and means of classification, some overlap may occur.

5.2 Classifications of Emergence

While numerous systems of classification of emergence exist, they tend to be based on one of several differentiating metrics. Most common are the degree of reducibility, predictability, or inter-

86 component interaction. Additionally, some others have based their classification on the expression of the emergent behavior. Examples of each of these metrics are discussed in this section.

5.2.1 Reducibility and Predictability

Many past classifications of emergence have been taxonomies based on the degree of complexity, reducibility, or predictability of the system or behavior. Philosophers in particular tend to favor the latter two measures when constructing taxonomies. Reducibility (i.e. whether system-level emergent behavior is merely the result of complicated interactions of components or is fundamentally other from the components) is commonly used to differentiate nominal and strong emergence in an ontological sense (here ontological refers to the fundamental nature of what emergence is, regardless of whether or not we can ever fully understand it).

Nominal emergence is any form of system property that results from the combined properties of its components [135]. This includes things as simple as the fact that a watch can tell time due to its combination of gears and springs, all the way up to things as complicated as financial markets.

Strong emergence on the other hand refers to a behavior exhibiting “irreducible causal powers,” [135] that is, it is an entity in its own right, not explainable in terms of its components, though it may still be dependent on them. The distinction here can be subtle and is not especially relevant here and thus will not be discussed further. Barnes [136] has an excellent explanation for interested readers. It should be noted that while some philosophers argue that conscious thought or life fits into the strong emergence category, scientists typically (but not universally) hold that the category is an empty set, as any case of strong emergence would be unable to be explained or replicated by the modern scientific method.

Predictability is used by philosophers to refer to what is theoretically predictable rather than what is practically or currently predictable. It is often used to subdivide nominal emergence into further categories like weak emergence in an answer to an epistemological question (here epistemological refers to what can be known about emergence, regardless of what it genuinely is). Weak emergence is system-level behavior that, while caused by properties and interactions of its components (thus making it fit into the nominal category), is not easily explainable by them and requires simulation of potentially extremely high complexity in order to be predicted [135], [137].

87 Engineers typically are not overly interested in the ontological question of emergence and often dispense with reducibility as a metric. Additionally, rather than consider predictability in terms of the theoretical (e.g. you have unlimited computing power and infinite computational time), it is instead preferred to think of predictability in a more practical or current sense. When Bjelkemyr et al. use the weak and strong categories for example, weak emergence is defined as that “which can be predicted by experience or extensive modeling and simulation.” Strong emergence, on the other hand, describes properties and behaviors that are displayed by the system, but can’t be reduced to known interactions or components [138]. This shift means that as our scientific understanding and modeling capabilities increase, a behavior could shift from one category to another, unlike in the philosophical typology. Additionally, this typology implies that there is no behavior that engineers and scientists cannot tackle.

Having only two categories in a taxonomy is somewhat limiting, however, and thus Maier expanded these two categories into simple, weak, strong, and spooky emergence, while making the emphasis on simulation and prediction even more explicit [139], as seen in Figure 5-1. Basic systems, like mechanical watches, were downgraded to the simple category. The weak category was reserved for behavior that is consistently predictable using simulations of equal complexity to the actual system, but not with more abstract models. Most systems studied using agent-based models would fall into this category. Strong emergence is now a behavior whose general causal chain can be traced down to the component level, but simulations fail to consistently replicate. An excellent example of this is financial bubbles, which are easily explained on evening news, but resist accurate real world predictions and diagnoses. Lastly spooky emergence refers to all system- level behavior that is flatly inexplicable and unpredictable using current scientific knowledge, such as conscious thought.

88

Figure 5-1. Maier’s Types of Emergence with Examples. Adapted from [139] and [43]

Note that as these taxonomies have progressed, the emphasis on modeling and prediction has grown in importance. Some go as far as to say that emergence should be defined subjectively as whatever behavior is unexpected to an observer informed of the basic rules of the system [140], [141]. This makes the category that a behavior fits into not merely dependent on current scientific knowledge and engineering methodologies, but on the individual observer of the system. Under such definitions it may occur that, for example, a specific behavior would not be emergent to an experienced system designer, but would be emergent to a novice system operator. This concept is potentially problematic for discussing emergence in a productive manner as certain complexity scientists have noted [134].

Alternately, one taxonomy embraces the subjectivity and uses it as a basis of classification. Ferreira et al. define a taxonomy based on two dimensions: expected/unexpected and desirable/undesirable, as seen in Figure 5-2. They use this taxonomy to narrow in on UEPs (Undesirable Emergent Properties) for risk assessment [142].

89

Figure 5-2. Ferreira’s Emergent Behavior Taxonomy. From [142]

5.2.2 Complexity and Degree of Interactions

Not all engineering-specific systems of classification have focused on predictability, however. Complexity is commonly measured using the amount of interaction within the system with extra weight going to unique interactions. It is another popular metric for distinguishing categories of emergence. Bar-Yam of the New England Complex Systems Institute based his emergence taxonomy on degree of interactions both among the components of the system, and between the system and its operating environment [143]. This system of classification results in four types, as seen in Figure 5-3:

 Type 0: This refers to the comparison of the properties of the system with each of its components in isolation. Examples of this include much of contemporary physics, as physicists seek a unified theory of constituent matter and energy.  Type 1: This is the type of emergent behavior that results in a system due solely to the position, momenta, and interactions of its individual, but otherwise identical components. Examples include the pressure of a gas, which results from the dynamic motion of individual particles. Which view is more useful depends on the scale under consideration.  Type 2: This is the type of emergent behavior that results when a system is considered in all of its possible states, rather than in just an individual state. An example of this would be any system with a system-level constraint that does not apply to an individual component. An example of this includes the lift to weight ratio of an aircraft. No individual component is bound to any mass, drag, or lift requirement, but collectively, the aircraft must be able to generate more lift than it has weight.

90  Type 3: This is the type of emergent behavior that can only be observed when a system and its environment are taken together. For instance, a key cannot be fully described without describing the lock that it matches. Similarly, the behavior of proteins and DNA cannot be fully described without knowing the details of their environment in a living creature.

Figure 5-3. Bar-Yam’s Four Types of Emergence. From [143]

Szabo et al. took the bold step of proposing a quantitative measure of emergence based upon interaction and possible system states [144]. They and others have used grammar systems to formally define and quantify emergence in terms of interaction [145], [146]. Szabo’s definition of emergence is essentially

퐿휉 = 퐿푤ℎ표푙푒 − 퐿푠푢푚

Where 퐿휉 is the set of emergent property states, 퐿푤ℎ표푙푒 is the set of possible system states due to agent-to-agent and agent-environment interactions, and 퐿푠푢푚 is the sum of all individual agent behaviors, without considering inter-agent interactions. They applied this to a simulation of a group of birds that collectively flock together, providing them with various advantages. This is accomplished even though the individual birds just follow three rules: (1) separation between birds, (2) alignment to steer towards the average heading of neighbors, and (3) cohesion to steer towards the average position of neighbors [144].

Bar-Yam argued that these forms of classification enable a better understanding of the possible states of a system and more easily identify behavior dependencies [143]. A robust, formal 91 mathematical definition of emergence would certainly lend towards more accurately determining which aspects of systems lead to emergent behaviors. On the other hand, such a definition may exclude behaviors commonly considered to be emergent and have, as of yet, only been demonstrated on either rather basic systems or systems with few (or one) unique components. Thus the feasibility of applying these definitions to more complicated designed systems, much less the multidomain systems of systems that are the common subject of an MCE acquisition program, remains unproven. Note that these taxonomies should not be confused with similar ways of classifying complex systems by degree of complexity, such as Sheard’s complexity typology [147]. While such systems of classification are quite useful, they focus on describing the system as a whole, rather than specific behaviors which is the subject of this chapter.

5.2.3 Expression

The systems of classification discussed in Sections 5.2.1 and 5.2.2 (all of which are taxonomies) have many uses. For one, they serve to clarify what is meant by “emergent behavior” so that it can have genuine meaning in the systems engineering context other than “I didn’t see that coming” (excluding the aforementioned exception that does precisely that). Additionally, they help inform design themes for handling emergence, such as robustness, communication, and learning [148]; define the risk of undesirable emergence [142]; and inculcate the general mindset that a systems engineer should have when confronting emergence [149]. That said, these taxonomies are aimed more towards the experienced systems engineer and the development of theory. They do not lend well towards concretely explaining the concept of emergence to a layperson, nor do they immediately suggest concrete ways of either reducing the likelihood of a specific undesirable emergent behavior occurring or diagnosing and addressing such behavior once it has started occurring. Other forms of classification, based more on the type of expression of the emergent behavior can better serve this purpose.

Fromm [150], for example, developed a taxonomy that still uses the basic simple-to-strong structure proposed by Bjelkemyr et al. [138] and Maier [139], but also adds subcategories based on traits of the behavior exhibited, as can be seen in Table 5-1. Fromm’s taxonomy is quite useful for illustrating emergence with real world examples and for discussing potential causes (he does so mostly in terms of feedback loops across different spatial and temporal scales).

92 Table 5-1. Fromm’s Taxonomy of Emergence [150]

Type Subtypes Examples a) Simple a) Mechanical Watch, Steam Type I: Simple/Nominal Intentional Engine Emergence without top-down b) Simple b) Pressure, Temperature, Slope of feedback Unintentional a Sand-pile a) Flocking, Self-Organization of Type II: Weak Emergence a) Stable the Internet with top-down feedback b) Instable b) Financial Bubbles, Demographic Clustering

a) Stripes, Spots, Bubbling a) Zebra Stripe Formation, Type III: Multiple Emergence b) Tunnelling, Financial Market Cycles with many feedbacks b) Natural Evolution, Adaptive Adaptive Systems, Scientific Revolutions Emergence Type IV: Strong Emergence None a) Life, Culture

Mogul [151], on the other hand, chose to dispose of the complexity hierarchy taxonomy altogether, instead creating a typology using descriptions of behavioral patterns, such as “unwanted synchronization” or “livelock,” to create categories. Additionally, Mogul used the same structure to make a typology of causes of emergent behavior. The typology of behaviors can be seen in Table 5-2 and the causes in Table 5-3. It should be noted that Mogul did not intend this list to be exhaustive, but rather to serve as a starting point to be developed further [151]. While Mogul’s typologies are restricted to undesirable emergent behavior (which he called “emergent misbehavior”), they could readily be expanded to include desirable behavior as well. Additionally, this “emergent misbehavior” of Mogul’s appears to be a subset of Troncale’s system pathologies [76], which are discussed further in Section 3.2.1.3.

93 Table 5-2. Mogul’s Proposed Typology of Emergent Behavior. Adapted from [151]

Behaviour Description Examples Types Thrashing When competition over a resource Computers when insufficient working results in switching costs between memory is allocated for the task at the sharing parties dominating hand, a notable instance of this was the actual useful work IBM/370 mainframe computers [152] Synchronization When components whose time- The frequency of pedestrians steps on varying behaviour is normally the London Millennium footbridge uncorrelated becomes correlated. [153]; router protocol message synchronization [154] Oscillation When the system periodically Gliders in Conway’s Game of Life, switches between states due to PsEPR herding [151] some feedback loop. Deadlock When progress stalls due to a Mars Pathfinder software failure [155] circular set of dependencies Livelock When progress stalls because Two individuals “dancing” in a hallway multiple components keep while trying to get around one another, adjusting in such a way as to Ethernet Capture Effect [156] remain counteracting one another

Phase change When the system switches Northeast Blackout of 2003 [82], [157]; between states in an erratic or transition from synchronized traffic irreversible manner flow to a traffic jam [158]

94 Table 5-3. Mogul’s Proposed Typology of Causes of Emergent Behavior. Adapted from [151]

Type of Cause Description Unexpected Resource The system designer assumed that separate components had access Sharing to separate resources when in fact the resources are shared and insufficient Massive Scale The number of communicating components in the system is large enough to give rise to complex global behavior, even if individual components have simple behaviors

Decentralized Control Distributed systems that lack central controls, and hence suffer from incomplete knowledge and delayed information, can exhibit oscillations and chaos

Lack of Composability When components are not composable (i.e. lack strictly defined interfaces), modularity may not simplify behavior Misconfiguration Beyond literal errors, in complex systems, it is often too difficult for operators to understand global consequences of local configuration choices Unexpected Inputs or While not all undesired behavior in response to an unexpected input Loads or load is emergent, sometimes implementers draw the boundaries of “the system as a whole” too close to what they are responsible for implementing. Information Sharing Latency makes a system harder to understand and harder to control, Delays which can lead to oscillations and chaos.

Unlike many of the other systems of classification which were developed as taxonomies, Mogul’s typology does not seek to place an individual instance of emergent behavior into certain type to the exclusion of others. Rather multiple behavior types may be exhibited in one instance of emergent phenomena. While the matched frequency of pedestrian’s steps on the London Millennium Footbridge is listed as an example of synchronization in Table 5-2, the overall behavior of the system can be characterized as an oscillation caused by an unexpected feedback loop. Applying these “tags” on cases of emergent behavior gives system designers and operators another handle on which to conceptualize their own work, by enabling quick comparisons between similar cases to be made.

5.3 Design Patterns for Emergence

Once these various systems of classification were understood, an expression-based typology that built upon Mogul’s work was used to generate and compile a set of design patterns. This design

95 patterns are intended to enable the quick identification of emergent behavior, to find analogous cases in other systems, and to provide insight into possible avenues to address the mitigation or inducement of such behavior in the future. While these design patterns have relevance to designing with emergence in general, it has particular relevance to the MCE environment and to vulnerability assessment, as will be discussed further in later on.

In this work, design patterns are used to refer to a subset of design principles. The latter refers to an abstract rule that produces concrete solutions to design problems. The design patterns are a subset of these aimed at specific problems, apply to the end-system itself (as opposed to the design process), and have some degree of provability or verifiability. Design patterns can be contrasted with design heuristics, the other subset of design principles, which are general, unverifiable rules- of-thumb usually aimed at the design process. Design patterns and heuristics have previously been used by Mekdeci [82] as well as by Sheard and Mostashari [159] for similar purposes in the field of survivability and resilience. Closer to this work, German developed a set of interview-based design heuristics for human-model interaction and model-centric decision-making [160]. An example of a design pattern that arose based off the design principle “Margin” in Mekdeci’s work is shown in Table 5-4. Though design patterns have been developed for numerous fields and predate the cited researchers [161], this work uses an altered version of Mekdeci’s general format for describing them.

Table 5-4. Example Design Pattern for Margin. From [82]

96 When formulating these design patterns, there are certain caveats that should be kept in mind. First, though design patterns are commonly referred to as general solutions to common problems based on a certain design principle, it is not claimed that these patterns “solve” emergence or the vulnerabilities that they can introduce. While this work does not endorse the definition of emergence as the unexpected as some others have done [140], [141] (See Section 5.2.1), most, if not all, forms of relevant emergence cannot be consistently predicted or designed for. Rather, through observing certain similarities in circumstances surrounding emergent behavior and the forms that certain behaviors take, these design patterns aim to expand the ability to at least anticipate it to some degree. So in this context, the phrases “suggested action to alleviate” and “suggested action to encourage” are used instead of “solution.”

Similarly, while Mogul referred to his latter typology as a “taxonomy of causes” [151], it is likely that the explicit cataloging of strict causal factors of emergent behavior is overly ambitious for what systems scientists are currently capable of (or may ever be capable of). Instead this research seeks to catalogue common circumstances that are conducive to emergent behavior. For instance, a system merely being of massive scale (one of Mogul’s listed causes) does not guarantee emergent behavior, nor can it often be truly stated as the cause of the emergent behavior. Rather, systems of massive scale tend to be more likely to exhibit emergent behavior, and accordingly this work uses the phrase “causal factor” to describe these traits. It is defined here in the following way: A causal factor is any aspect of the system which, when removed or changed, is likely to reduce the occurrence of emergent behavior, or, when induced, is likely to increase the occurrence of emergent behavior. Clearly this limits the testability of the design patterns. Nonetheless, it is expected that many of these patterns will be noncontroversial for experienced systems engineers and appear reasonable upon reflection. Moreover, we believe that these patterns will prove useful to practicing systems engineers, particularly those with limited experience with emergent phenomena.

The first step towards developing the proposed design patterns is to generate a typology of emergent behavior, a set of common causal factors, and suggested actions to either alleviate or induce the behavior. The suggested actions will form the basis of each design pattern. The required pieces of information should not be listed independently, but instead linked to one another along with concrete examples. Table 5-5 shows a summary of such information for three types of

97 emergent behavior: Synchronization, Phase Change, and Oscillation. A key element is to keep each type well-documented both in its examples and in its suggested actions. For example, Over- Design Capacity has been found to be a feasible action in a variety of domains: from traffic jams, where the transition from synchronized flow to congestion cannot occur over a certain density [162], [163], to online communication, well-illustrated by the PsEPR herding case where the initial herding behavior and oscillation from one server to the next cannot occur unless the number of clients is above a certain threshold [151]. Note that the common causal factor Density exists in both of these examples as well. These can be found both in field specific research (as the above examples were) or in more generalized discussions of emergent behavior, such as De Wolf’s and Holvoet’s proposal of two specific design patterns for dealing with emergence resulting from decentralised coordination: gradient fields and market-based control [164]. This general system of cataloging is not without precedence and has been used previously to link cause, effect, solution, and examples of different types of perturbations [80].

98 Table 5-5. Summary information for three types of emergent behavior

Emergent Behavior Synchronization Phase Change Oscillation Description Components of the system The system switches The system or some of Emergent or of the environment between two or more set of components Behavior whose time-varying states in an irregular, periodically switch behavior is expected to be unintended, or irreversible back and forth uncorrelated become way between two or more correlated states in some repeating pattern. Consequences Can increase the load on a Can cause the system to The rapid state component or the system act in unpredictably, changes can inhibit as a whole, alternately making control and productive function or may result in greater monitoring more difficult. cause undue loading efficiency of a system that Can also dramatically to some parts of the is normally random increase or decrease system. system performance. Common Unexpected Inputs; Density; Unexpected Feedback Causal Decentralized Control; Perturbation/Nucleation Loops, Frequency Factors Frequency Matching; Site Matching Unexpected Feedback Loops, Density Suggested Over-Design Capacity; Remove Perturbations; Decoupling; Induce Action(s) to Induce Randomness; Over-Design for Capacity Randomness Alleviate Introduce Perturbations

Suggested Minimize Perturbations; Induce Perturbations; Add Introduce Time Action(s) to Market-Based Control; Energy Delays Encourage Gradient Fields Example Frequency of pedestrian Spontaneous Traffic Vortex Induced steps on the Millennium Jam[158]; Laminar vs Vibration, London footbridge [153]; router Turbulent fluid flow; Millennium protocol message Cascading Network Footbridge Vibration synchronization [154]; Failure [82], [157]; [166]; PsEPR herding PsEPR herding [151]; Locust Swarms[165] [151]; Gliders in Laminar fluid flow Conway’s Game of Life

While Table 5-5 contains descriptions and potential consequences of each emergent behavior, it lacks clear explanations of the common causal factors and the suggested actions. Some of the former are more clearly defined in Table 5-6, while the latter will be more fully described in the proposed design patterns, discussed later in this section.

99

Table 5-6. Descriptions of three types of causal factors

Causal Factor Explanation Examples Density As the energy or material density of a Spontaneous Traffic Jam[158]; system increases, feedback loops are Laminar vs Turbulent fluid accelerated and strengthened, nonlinear flow; Locust Swarms[165]; behavior becomes more apparent, and PsEPR herding [151] likelihood of a perturbation increases. Frequency Two or more known feedback loops or Vortex Induced Vibration, Matching oscillations that were expected to have London Millennium Footbridge different frequencies are instead found to Vibration [166] have mutually reinforcing frequencies. Perturbation / There exists some temporally and spatially Laminar vs Turbulent fluid Nucleation limited variation of input to the system, flow; Spontaneous Traffic Site either endogenous or exogenous, that can Jam[158]; Cloud seeding [167] agitate local nonlinearities, potentially initiating a chain-reaction across the system

Once the above catalog has been created (or at least a substantial start has been made), it is then possible to use the suggested actions as design patterns. Table 5-7 shows an example of such a design pattern for “Over Design Capacity.” In a full digital catalog, the various causal factors, suggested actions, and examples could be linked to the expanded explanations in order to facilitate easy exploration by the reader. Similarly, the related patterns could also be linked.

100 Table 5-7. Example proposed design pattern

Over-Design Capacity / Increase Capacity Emergent Behavior Types Synchronization; Phase Change Addressed Context Emergent behavior, potentially exacerbated by the material or energy density in the system, results in higher than expected load on some component or the system as a whole

Action Allow for extra load capacity than would be necessary should no emergent behavior occur. Unintended Consequences May result in induced demand; will likely require additional resources (particularly physical space) that may be costly or not be available

Example Increased memory capacity resulted in trashing behavior like that seen in the IBM/370 become less common [152]; opening the shoulder of a highway as an additional lane can (at least temporarily) reduce traffic [168]

Related Patterns Margin, Heterogeneity, Physical Redundancy

While this format is useful for a reader to quickly reference information about specific behaviors, it is limited in its ability to search in the opposite direction (e.g. a designer has noticed that their system-as-designed contains some unexpected feedback loops and want to understand what kind of behaviors might occur if this not addressed before implementation). To serve this need, a cross- mapping table, relating causal factors and suggested actions to the three forms of emergent behavior discussed here has been generated and is shown in Table 5-8.

Neither the design patterns nor the cross-mapping table are intended to be exhaustive. Instead they are intended to serve as a structure for evolving a full catalog.

101 Table 5-8. Emergent Behavior Cross-Mapping

Common Causal Factors

Unexpected Perturbation / Frequency Unexpected Decentralized Massive Density Feedback Nucleation Site Matching Inputs Control Scale Loops

Over Design Synchronization Synchronization Synchronization Synchronization Synchronization Capacity (Margin) Induce Synchronization Synchronization Synchronization Synchronization Synchronization Randomness Oscillation Oscillation Induce Synchronization Phase Change Synchronization Synchronization Synchronization Synchronization Perturbations Phase Change Oscillation Oscillation Decoupling Oscillation Remove Phase Change Phase Change

Suggested Actions to Alleviate Actions Suggested Perturbations

There are four primary intended users of these design patterns: systems architects/engineers (the people designing and creating the system), system operators (the people running the system on a day-to-day basis), analysts (those charged with diagnosing problems and rectifying them), and students.

Designers can use these patterns to help anticipate emergent behavior and adjust their plans to either encourage or discourage the development of emergent behavior once the system is put into operation. Furthermore, designers can use the listed examples as generative metaphors in order to draw inspiration [169].

Both operators and analysts can use these patterns to more quickly identify and react to emergent behavior in their systems. This is line with the Cynefin framework’s suggest course of action “Probe-Sense-Respond” in the complex domain [170]. By showing system operators and analysts the common causal factors and common suggested actions to alleviate, they more quickly identify the correct elements of their system to probe, know what they should be looking to sense, and develop a response. As discussed previously, the historical taxonomies have focused on the systems architect’s role in working with emergence and been less relevant to the analyst or operator. Previously, this need has been noted by De Wolf and Holvoet, who attempted to remedy it by applying design patterns to a subset of emergent behaviors, specifically decentralized coordination [164].

102 Lastly, “students”, be they grade school or university students or practicing engineers learning about systems engineering, can use these design patterns to get a more concrete grasp on what emergence means in applied use. Emergence has highly varied meanings across fields (and even within them). Additionally, it has a common English definition of “becoming visible over time” [171]. As a result, the concept can be difficult to teach to students. This is particularly true for those who are not native English speakers as it requires a non-literal interpretation of the colloquial meaning of “emergence.” Anecdotal experience suggests this difficulty in explanation is eased through the use of examples from a variety of domains, particularly those with personal or professional familiarity to the student. The proposed catalog of design patterns would formalize this process and allow for more detail and clarity in the explanation. Once a student has been familiarized with these various types of emergent behavior and examples of them, they will be better prepared to consider the systems of classification proposed by Bar-Yam [143], Maier [139], and the others, as well as the more general modes of thought useful for tackling emergence, such as those discussed by Keating [148].

Program managers can fit in several of these user categories. They are often working closely with the systems engineers and thus need an understanding of what vulnerabilities undesired emergence can introduce into a system. Program managers are often the individuals to whom analysts are reporting to, so developing a common language around emergence can facilitate communication. Finally, program managers can be students themselves depending on their current career stage.

Beyond emergent behavior in end-systems, as MCE progresses and increasingly integrated modeling environments are constructed, program managers should be vigilant for the potential of emergent behavior arising within the acquisition environment. One could even argue that many of the benefits of MCE, such as system-level optimization, are themselves positive forms of emergence. Also possible, however, is the manifestation of negative emergence such as the thrashing or herding examples discussed earlier.

Combining the classification of vulnerability chains with the typology introduced in this chapter has the potential to aid in the identification of new design patterns and heuristics to help accommodate and enable this emergent behavior. The classification of vulnerability chains in the Reference CEM discussed in Section 4.3 already enables the application of heuristics. For instance,

103 German identified 29 heuristics for use by decision-makers (such as program managers) in MCE environments [160]. He grouped them into six categories

1. Designing Models for Human Use 2. Using Models in Decision-Making 3. Sociotechnical Considerations 4. Context and Assumptions 5. Transparency and Trust 6. Mitigating Biases

If a program manager is concerned about a vulnerability chain that passes through the Misunderstood Model Assumptions event, she may reference the heuristics in the Context and Assumptions category and use these heuristics herself or for communicating with other team members in order to seek to combat the event. For instance, if she is concerned that due to Poor Model Documentation, her analysts are at risk of misunderstanding model assumptions, she may remind either herself or her analysts that “Before any model can be useful, its capabilities and limitations must be revealed and understood,” and require additional testing and verification of the model to ensure that the capabilities and limitations are identified. Similarly, if the program manager is concerned about one of the vulnerability chains that have to do with trust, such as those that pass through Over-Trust in Model or Under-Trust in Model, she may reference the Transparency and Trust category of heuristics. The usefulness of classifying vulnerabilities to discern similarities and connections is corroborated by the usability testing discussed in Section 4.4.

This linking of groups of vulnerability chains to design heuristics is just one example of a how vulnerability classification systems can enable the application of design principles. This chapter also laid out how the classification of emergent behavior can lead to the generation and application of design patterns. As MCE is implemented and harmful emergent behaviors manifest within the program, the emergent behavior typology presented here can similarly be used to diagnose these new vulnerabilities and identify actions to intervene in them. In this way, the Reference CEM and the emergent behavior typology can supplement one another. The former handles the vulnerabilities that can be expressed as linear causal chains, while the latter addresses those involving tight, multi-scale feedback loops. 104 It should be noted that, while most of the examples of emergent behavior cited in this chapter are those of designed systems, the natural world exhibits innumerable instances of emergent behavior that are also worth considering. Such cases have been thoroughly studied by organizations like the Sante Fe Institute and often demonstrate aspects of direct relevance to system designers, such as evolved modularity [172].

105

Chapter 6: Intervention and Policy Recommendations

Over the course of Chapter 4 and Chapter 5, two approaches to MCE-related vulnerabilities were explored: the Reference CEM and emergent behavior design patterns. Various specific vulnerabilities were identified, along with potential interventions. The different approaches taken in this work also resulted in more general recommendations and lessons learned for both individuals in defense acquisition programs and for MCE enterprises. This chapter presents these recommendations.

6.1 Intervention Recommendations for Program Leaders

In general, the intervention points identified in the Reference CEM of Section 4.3 tend to be in the first half of the vulnerability chains, with several being immediately after an external trigger. This means that program managers, systems engineers, and other individuals serving leadership roles in defense acquisition programs must remain vigilant for potential or imminent external triggers and be ready to respond as soon as, or even in advance of, their manifestation.

The Reference CEM can be used to guide the program manager in his dedication of attention to various vulnerabilities. For instance, it should be noted that within the “active modeling” set of intermediate events (inside the blue box of Figure 6-1) there are relatively few intervention points identified, despite the high number of vulnerability chains that pass through that section of the Reference CEM. The primary intervention point identified in that section, #7, reads as “Create documentation and curation processes within the program.” (Table 4-1). This dearth of intervention points represents the unfamiliarity of program managers with MCE processes and how to intervene in them. This suggests that further work would be useful in identifying potential interventions in that section and educating program managers about their availability. That said, mere awareness of these vulnerabilities could enable program managers to maintain vigilance as they cascade and intervene in more ad-hoc ways.

107

Figure 6-1. Excerpt of the Reference CEM highlighting the “active modeling” portion

On the other hand, certain vulnerability chains have multiple intervention points identified at multiple stages. For instance, several of the vulnerability chains that pass through the Needs Change event have three intervention points each (and the others have at least two). This means that the program manager may not need to be as concerned about these vulnerabilities, due to the multiple options of intervention available and the fact that several are positioned multiple events into the chain, giving significant time for response.

An experienced program manager will likely find some of the listed intervention points to be common sense. For instance, one of the interventions (#12) following the Needs Change event is “Reevaluate requirements with the client and other stakeholders.” This degree of occasional obviousness is not unique to CEM, but is true of all vulnerability assessment techniques. The point of these techniques is not just to identify new vulnerabilities and interventions, but to consistently track and list them so that all options are available. After all, even experienced pilots still use a checklist (and surgeons really should be [173]).

108 Finally, many of the MCE-related vulnerabilities displayed in the Reference CEM can be more generally combated by using MCE decision heuristics [160] and by developing a common vocabulary to discuss the emergent behavior of both the end-systems and the MCE environment itself. The typology and design patterns laid out in Section 5.3 can help accomplish this, thereby supplementing the CEM and provide for multiple branches of attack on MCE-related vulnerabilities.

6.2 Policy Recommendations for Acquisition Enterprises

While the programmatic vulnerabilities discussed in this work are most directly the responsibility of program managers (and secondarily of systems engineers), there are several policy lessons to be taken for acquisition enterprises. Numerous policy changes have been proposed by others regarding the MCE transformation and acquisition reform in general (see Section 2.1.2 for a discussion of past changes). This work did not identify any wholly novel policy recommendations, but over the course of the CEM generation, interviews and usability testing, three particular recommendations were given additional support. These are:

1. The acquisition enterprise should clearly define ownership and responsibility over the MCE environment. 2. The acquisition enterprise should treat the acquisition process as an engineered system in its own right, subject to the same means of analysis and design as the actual end-systems being acquired. 3. The DoD should assume a leadership role in encouraging the development and definition of MCE standards within the defense acquisition industry.

In this section, these three recommendations are discussed at greater length.

6.2.1 Clearly Define Ownership over the MCE Environment

The US Defense Acquisition System is governed by the DoD 5000 Series Directives [174]. For many years, in both official doctrine and elsewhere, the DoD has recognized the importance of MCE, even before that term was commonly used, as well as the general importance of modeling & simulation (commonly referred to as M&S in DoD directives). For instance, in the 2006 Acquisition Modeling and Simulation Master Plan, it is recognized that “MBSE calls for

109 automated engineering tools, which are modeling environments, to analyze requirements, develop architectures, and specify constraints…M&S-enabled CEEs [Collaborative Engineering Environments] provide a means to share authoritative information across an acquisition enterprise and use interoperable modeling environments, models, simulations, and distributed environments to communicate, design, assess, immerse warfighters, integrate, and test.” [175] This passage seems well in line with the goals of MCE, which include a “single, authoritative source of truth,” [176] and a desire to integrate “different model types… across disciplines throughout the lifecycle.” [33] This recognition of importance is also supported in official doctrine, as DoD Directive 5000.59 states in Part 4.1 that “M&S is a key enabler of DoD activities,” and in Part 4.3 that “M&S Management shall develop plans, programs, procedures , issuances, and pursue common and cross-cutting M&S tools, data, and services to achieve DoD’s goals.” [177]

The Master Plan set forth five objectives for Acquisition M&S, along with specific actions for achieving these objectives and who was responsible for each action, as seen in Figure 6-2. Among these are several that are still relevant today. For instance, A1-2 reads “Promote model-based systems engineering (MBSE) and M&S-enabled collaborative engineering environments (CEEs) at both the program and joint capability.” Within this action it is stated that “DoD must maintain awareness of emerging tools and processes to enable MBSE, lessons learned in commercial and defense applications, and trends in the systems engineering community to exploit MBSE.” This goal has been largely effective, which is why the DoD is now poised to take advantage of emerging MCE tools and processes beyond MBSE. Another is A2-2, which reads “Support development of open commercial and non-proprietary standards for systems engineering, such as OMG’s Systems Modeling Language (SysML) and ISO Standard 10303 AP-233.” While these standards have been well defined and adopted by the DoD [178] other model integration standards remain to be established and will need DoD leadership, as discussed in Section 6.2.3

Similarly, the problem of re-use (or lack thereof) is recognized in A2-5: “Although many M&S resources are reusable for various acquisition-related purposes, relatively little reuse occurs… Standard templates of descriptive information (metadata) about such resources would aid in automated searches for such resources.” While subsequent actions based on this Master Plan led to an expansion of the DoD Metadata Registry and its ultimate replacement with the DoD Data Services Environment [179], this issue of enabling user-friendly reuse of models remains.

110 Some of the Master Plan actions were specific and near-term, such as A1-4: “Establish policy to require documented M&S planning as part of the Systems Engineering Plan, T&E Strategy, and T&E Master Plan.” Others seem more general and long-term, such as A1-2. Despite this, however, all of the listed actions have specified target completion dates between 2006 (the same year that the Master Plan was issued) and 2011, with the majority (including both A1-4 and A1-2) timed for completion in 2007. Subsequent updates to M&S policy in the DoD have primarily come from elsewhere, including the Modeling & Simulation Steering Committee (M&S SC).

Enhance the technical Provide necessary Improve model and Improve model and framework for Shape the workforce policy and guidance simulation capabilities simulation use modeling and

Figure 6-2. Objectives of the DoD M&S Master Plan. Adapted from [175]

DoD Directives are the official source of policy within the DoD. As mentioned early, the 5000 Series Directives set policy for the acquisition system and process, and define responsibilities within in. Several key divisions of responsibility are worth nothing here.

The Under Secretary of Defense for Acquisition, Technology, and Logistics (USD(AT&L)): Directive 5134.01 states the high level description of the USD(AT&L)’s relevant responsibilities in 3 as, “The USD(AT&L) is the Principal Staff Assistant… for all matters relating to the DoD Acquisition System… modeling and simulation; systems engineering…” Furthermore, it goes onto

111 say that the USD (AT&L) must “Develop policies, plans, and programs to coordinate, harmonize, and rationalize DoD M&S, including the DoD Modeling and Simulation Master Plan and Investment Plans. Ensure that DoD M&S investments support operational needs and the acquisition process; the development of common tools, methodologies, and databases; and the establishment of standards and protocols promoting interoperability, data exchange, open system architecture, and software reusability.” [180]

In addition to these broad oversight roles over M&S, the USD(AT&L) is specifically charged in 5000.70 E2.1.a to “develop and implement the DoD M&S discovery metadata search capability.” [181] This authority is not complete however, as E2.3.b of Directive 8230.02 states that the “In coordination with the DoD CIO, the USD(AT&L) provides guidance to program managers and Program Executive Officers to evaluate and approve system or program implementation of data sharing practices.” [182]

Note, that as of February 2018, this role no longer exists. Instead, it has been split into two roles, Under Secretary for Acquisition & Sustainment (USD (A&S)) which the last USD (AT&L) has filled, and Under Secretary for Research & Engineering (USD(R&E)) which has been filled by a former NASA Administrator. The complete separation of the role is expected to take another two years, so exactly what responsibilities and authorities will fall to each cannot be said yet [183].

The Director of the Modeling & Simulation Coordination Office (M&SCO): Directive 5000.70 in E2.2.f.1 states that the Director shall “serve as the DoD lead standardization activity (LSA) for managing M&S standards and methodologies.” [181] The M&SCO, which is established by Directive 5000.59 [177], sits under the USD(AT&L) and seeks to provide support in “governance, operations, and technical collaboration” both within and without the DoD [184].

The DoD Chief Information Officer (CIO): Directive 8320.02 in E2.1.a states that the CIO “guides and oversees matters related to the sharing of data, information, and IT services to ensure interoperability down to the technical level internally within DoD and externally with mission partners.” As mentioned earlier however, the CIO must coordinate with the USD (AT&L) according to E2.3.b.

112 The Program Manager (PM): Directive 5000.01 (The Defense Acquisition System) states in E1.1.29 that “The PM shall be the single point of accountability for accomplishing program objectives for total life-cycle systems management, including sustainment.” [174] Directive 5000.02 (Operation of the Defense Acquisition System) further clarifies in E3.9 that “The Program Manager will integrate modeling and simulation activities into program planning and engineering efforts. These activities will support consistent analyses and decisions throughout the program’s life cycle. Models, data, and artifacts will be intergraded, managed, and controlled to ensure that the products maintain consistency with the system and external program dependencies, provide a comprehensive view of the program, and increase efficiency and confidence throughout the program’s life cycle.” [185]

What is notable among the above divisions of responsibility is the separation between the program manager and the broader efforts of modeling coordination. The above quote from Directive 5000.02 is the entirety of Section 9 on Modeling and Simulation. Of note in that passage is that it only discusses the program at hand, with no reference to the broader modeling environment or the possibility of reuse. Similarly, the various directives such as 5000.59 and 5000.70 that define that broader modeling environment make no reference to program managers. All responsibility is given to USD(AT&L), the CIO, and the Director of the M&SCO. Yet some amount of coordination across programs is clearly required. This point has been repeatedly acknowledged by the DoD. A1-4 of the Acquisition M&S Master Plan contains a statement that “M&S Planning must be explicitly address at the joint capability level – not just the individual program level.” [175]

Many of the vulnerabilities analyzed in Chapter 4 and the barriers to implementing MCE discussed in Section 2.2.4 are at least partially attributable to this unclear ownership over the modeling environment. Currently the program manager is responsible for the modeling environment of her program and for the successful completion of her program. The possibility of model reuse, of assumption documenting, and other forms of long-term benefits are largely beyond her purview. If either the program manager was made responsible for such aspects, or a coordinating body such as the M&SCO was given more direct authority in individual programs, some of the vulnerabilities could be obviated. For instance, a program manager would not need to concern with themselves with any of the Software Changes category of external triggers. Furthermore, many of the vulnerabilities passing through the Over-Trust in Model and Under-Trust in Model intermediate

113 events could be avoided if concrete standards for the modeling environment were in place and accepted.

The DoD then would be well served by linking these two categories of responsibility. The infrastructure and expertise for a cross-program, coordinated modeling environment is already present. Similarly, the DoD has fantastic program management experts. All that is needed is to connect the two. The clearest way to accomplish this is through a series of updates to the directives.

 In 5000.02, the responsibility of the program manager should be extended beyond the program’s lifecycle to include ensuring that any models generated are readily accessible and usable for reuse in future projects.  In 5000.59 and 5000.70, the relationship between the USD (AT&L) and the M&SCO with the program manager should be made clear. This could include requiring program managers to abide by any model integration standards that the M&SCO applies or by clearly stating the role the program manager is to play in furthering the modeling coordination goals of the agency.

The ongoing USD (AT&L) split makes this a particularly good time to implement some of these changes, with the USD (A&S) taking most of the responsibilities discussed here. The stated plans (and the job title) suggest that this new role will focus more on sustainment of programs and systems [186], which is well in line with the above recommendations.

While these recommendations would be helpful in continuing progress, the DoD’s directives are already ahead of many other groups when it comes to establishing a coordinated modeling and simulation regime. NASA’s Technical Standard for Models and Simulations, for instance, make no mention of MBSE, model integration, interoperability, or coordination (other than a reference to the DoD’s M&SCO). Program managers are only called upon to “identify and document the parties responsible for complying with the requirements in this NASA Technical standard.” In fact, it explicitly allows much of modeling and simulation to be exempted from the standard entirely when it states “All M&S used to support critical decisions, as determined by the outcome of compliance with [M&S 6] are in scope of this NASA Technical Standard. Beyond that, at the discretion of the program/project management and the Technical Authority, any other M&S may be deemed in scope of this NASA Technical Standard.” [187]

114 This is not to say that NASA is not making significant strides when it comes to MCE. Several examples of NASA’s leadership in the field are discussed in Section 2.2.2.3. Additionally, the NASA Systems Engineering Handbook devotes significant space to discussing the Capability for Accelerated Concurrent Engineering (CACE) environment, which has significant overlap with MCE. The Handbook describes CACE as a “core multidisciplinary engineering team working with a stakeholder team using well defined processes in a dedicated collaborative, concurrent engineering facility with specialized tools.” [188] This description is clearly in line (though not identical) to most definitions of MCE. Additionally, the Handbook recognizes the need for more integration. In 7.3.7 it states that, “As the Agency increasingly engages in multi-Center programs and projects, the need for interoperability among different tools, and different versions of the same tool, becomes critical.” However, it provides little guidance on achieving this goal. Thus, while NASA is exercising a real leadership role in testing and applying MCE tools and processes, it appears to lag behind the DoD in instituting requirements or standards in policy.

6.2.2 Treat the Acquisition Program as an Engineered System

During the second half of the 20th century, policy analysts, researchers, and decision-makers began explicitly considering various policy and social issues from a systems-theoretic conceptual point of view [189]–[191], basing these metaphors and models upon Wiener’s cybernetics [192] and Simon’s The Sciences of the Artificial [193]. This was conceived as a way of ‘rationally’ approaching policy in “an objective, logical, complete and thoroughly professional way.” [191] While the popularity of this approach may have waned some, it continues exert a strong pull, as evidenced by the creation of the Engineering Systems Division (ESD) at MIT [194] and, more recently, the Institute of Data, Systems, and Society (IDSS) [195].

During this same stretch of time, the tools and methods for designing complex systems have improved immensely, including many beyond the MCE concept discussed in this work. Some of these tools have found their way into policy applications. Data science, for instance, has become quite popular, as evidenced both by IDSS’s focus [196] and by the profligate use of data analytics in the criminal justice system [197]. MBSE and MCE tools have found less extensive use in policy applications. The DoD is not unfamiliar with the use of systems metaphors for policy design. In fact, the DoD was among the first to make such applications, including with the Planning,

115 Programming and Budgeting System (PPBS) (the predecessor to the contemporary PPBES) [198], which later spread across industry and government [189].

It is proposed that the DoD should thus leverage its experience with systems-theoretic policy design and its significant experience with systems engineering to bring MBSE and MCE tools to bear on the acquisition process itself, by treating it as an engineered system. The US Department of Veterans Affairs (VA) has been among the first to apply MBSE to policy construction [199], [200], perhaps due to their close relationship with the DoD. Such a conceptual viewpoint could be leveraged to more extensively identify and mitigate vulnerabilities within the acquisition process. This potential benefit is not purely hypothetical, as using metaphors to other systems has already enabled new approaches to cybersecurity vulnerabilities [201], [202].

CEM-VA could be one tool used in such an approach to redesigning the acquisition process, but it would not be the only one.

6.2.3 The DoD should assume a leadership role in encouraging the development and definition of MCE standards within the defense acquisition industry.

The DoD must ensure that MCE standards are developed that suit the needs of the defense acquisition process. In the modern era, IP is often as least as valuable as physical capital. This is particularly true in technology-centric industries, including defense. The advantages that the US (as well as a number of other nations) holds is not merely in terms of physical or human labor resources, but also in terms of intellectual capital, including both the skill levels of the employees and the value of their intellectual output [203]. As a result, the US has developed many robust mechanisms for protecting intellectual property. Some are enforced by the government, such as patents, copyrights, and trademarks. Others are privately enforced, such as non-disclosure agreements (NDAs). Trade secrets have both government and private enforcement mechanisms.

The MCE environment will doubtlessly include highly valuable IP. As the integration of all “authoritative data and associated artifacts,” possession of such MCE-related artifacts as the DSM could allow for full understanding of the system and its individual subsystems. Many aspects of manufacture and assembly would likely be either explicitly included in or could be indirectly

116 inferred from the DSM. The simulation and design assessment capabilities of DSM and the MCE environment as a whole significantly enhance the ease of understanding the system.

If one firm owns the system being designed, from start to finish, then implementing MCE would be a relatively simple matter as far as IP is concerned. If the same firm also operated the product, then the development and use of a Digital Twin would likewise be a straightforward IP matter. This is not typically the case in the defense realm. The DoD sets requirements and is the operator of the end product. Multiple teams of firms then bid on the project. Even after one team has been selected, this team will often use other firms as suppliers. These teams are not fixed alliances. On one project two defense contractors may be cooperating, while on another simultaneous project they may be competitive rivals. As a result, these firms may be unwilling to share the detailed IP required for the DSM. One approach to addressing this issue in the past has been through a combination of non-disclosure agreements (NDAs) and a design process where subsystem interfaces are rigidly defined but the actual subsystems themselves are largely treated as black boxes [204]. This allows one firm to maintain control over their IP, while delivering only the final product to the larger working group. Overcoming the design limitations that this approach introduces should be a major goal of DoD, particularly since the DoD has repeatedly maintained that the contractor should retain as much intellectual property as possible [5], [205], [206].

Patents are one potential tool for safeguarding the interests of DSM stakeholders. Jointly developed technologies for simulation and model integration methods could be protected under patents shared by the developers. Under the Cooperative Research Act and its amendment, the Standards Development Organization Advancement Act, cooperative research towards developing a DSM standard would potentially be protected from antitrust litigation, as it is a research area of national interest [207]. This is a ready extension of activities that the Digital Manufacturing and Design Innovation Institute (DMDII) is already pursuing [208].

Copyright has limited application for the protection of DSM. While specific presentation of facts, such as experimental results or material properties, are subject to copyright, the facts themselves are not. Thus a restatement in a different format is typically enough to circumvent copyright. As a result, copyright in general cannot be relied upon for protecting such information. Similarly, while computer programs, such as modeling software, are copyrightable, the fundamental basis of these

117 (e.g., specific mathematical methods) are not. In fact, mathematical equations are not even patentable, though some applications of algorithms are. This results in variation in the usefulness of copyright as it pertains to software. In cases where the specific software package is itself valuable (e.g., Microsoft Word) copyright is a sufficient tool of protection. In other cases, where the value is not in the software package but in the technical underpinnings, such as a proprietary modeling program, copyright may not sufficiently guard the IP. Furthermore, while copyright might protect the modeling software itself, it would not protect the models made of the end-system, which are valuable in their own right.

Currently, technology and methods not patented or copyrighted are typically protected via trade secrets. While trade secrets enjoy some legal protections [209], these protections are contingent on the IP being kept secret. Unless the DoD is the sole holder of the MCE-related IP, in order for the MCE environment to effectively operate, information will need to be shared among firms. One of the more common methods for handling inter-firm sharing of trade secrets is the NDA. While useful, the NDA has certain drawbacks when it comes to MCE. For one, they are time-consumptive to complete, which could be an issue during an iterative design process. Additionally, the level of detail of information being shared under most proposed versions of MCE is quite high. Firms may not feel that NDAs are substantial enough to protect this information, particularly as the difficulties in discovering violations of NDAs might be difficult or impossible, limiting enforceability of such contracts [210]. Excessive use of NDAs could limit the reuse of knowledge generated by one project in another project, one of the proposed goals of MCE.

In order to resolve these issues, the DoD needs to take a leadership role, rather than waiting for private sector solutions to materialize on their own. The DoD already acknowledged the need for improved model integration standards in the 2006 Acquisition M&S Master Plan, where in A2-3.1 is written “Lack of agreement within the Department of Defense on a common distributed simulation method increases the complexity, time, and cost of composing distributed live-virtual- constructive environments.” [175] This does not necessarily mean that the DoD should necessarily seek to command and control the private sector. In fact, three major options exist when it comes to establishing standards for the MCE environment.

118 Standards organizations exist in many industries. Some are private groups, such as W3C or IEEE, while others are governmental, such as ANSI or ISO. Depending on the group and the specific standard, compliance may be either voluntary or mandatory. Either way, a common reason for the existence of standards is interoperability of products between firms. While other reasons for standards exist, including health and safety, these are less relevant to the topic of MCA. In general, the development of interoperability standards is motivated by the firms themselves, seeking to eliminate competition in certain dimensions (such as size of customer-base and location of market) and facilitate competition based on the products themselves. In order to avoid violation of antitrust laws, these standards are typically open, meaning that any firm may develop products under the standard. If there is protected IP, such as patents involved in the complying with a standard, then this IP must be licensed under fair, reasonable, and non-discriminatory (FRAND) terms [211]. Standards are typically formalized through a consensus method, though the exact definition of consensus varies from one standards organization to another.

In the DoD context, there is little incentive for the industry firms to develop specialized MCE interoperability standards as the size of their customer base is fixed at one, the DoD. Thus, interoperability will not allow an expansion of market. The firms may develop their own, limited, proprietary versions of MCE tools in order to develop better products and thus better compete [47], but they will not likely reach out to other firms without external incentives. The power held by being the sole customer, though, does allow the DoD significant leeway in providing those external incentives. If the DoD defines specific interoperability requirements of modeling software or even requires the use of specific modeling software, these requirements will be followed by the industry.

It should be noted that as the DoD would the motivating force behind the standardization effort, the DoD is not bound by any consensus method of DSM development, thought it may wish to do so in order to make use of industry expertise and to promote goodwill with the industry. Additionally, due to the aforementioned exemptions in antitrust laws, it may be possible that the IP used to develop DSM does not need to be licensed under FRAND terms. This would mean additional options are open to the DoD regarding the enabling structure for the DSM.

With this in mind, DoD could standardize the model interfaces and model-centric decision-making processes. This heterogeneous structure would allow each firm to create its own modeling

119 packages (or license those of others as it chose). This option does not directly resolve any IP- sharing issue nor does it resolve the model validity issues. Unless the software packages are permanently licensed or outright purchased, this option raises the possibility of the DoD not having perpetual access to all components of the model of their system, especially if a firm goes out of business or fails to maintain its own modeling packages. It should be mentioned that this is currently the structure favored by the DoD, due to its combination of modularity and openness [212].

Alternately, the DoD could develop its own modeling packages and processes for each part of the MCE environment and then mandate that industry partners use these packages. This could be an extension, though a highly ambitious one, of the CREATE program [51]. This homogenous structure would eliminate any IP disputes over the modeling packages themselves and eliminate any KA issues of resolving inter-model differences. Furthermore, this would alleviate fears of losing access to models of older systems since the completed MCE environment would not be reliant on any privately-held software. Doing this, however, would fail to take advantage of the large amount of intellectual capital present in the private defense industry and would require the DoD to continually maintain and improve these software packages.

The third option is a hybrid of the first two, being privately-developed like the first option, but homogenous like the second. Here the industry partners would be relied upon to generate the modeling packages and these packages would be required to interface with one another, but the DoD would select specific modeling packages to be used for each portion of the DSM, likely in some sort of periodic bidding process (an acquisition project in its own right). The DoD would not just buy or license the modeling software for itself, but also for each other industry partner, otherwise the owner of the selected software package would have effective monopoly power over other members of the industry. This option does not address perpetual access issue any more than the second option, but does resolve the inter-model comparison issue while still relying on the industry to generate and improve modeling packages.

These options, along with their pros and cons, are summarized in Table 6-1.

120 Table 6-1. Summary of Model Package Development Structures

Pros Cons DoD Developed Reduces IP disputes Does not utilize industry expertise Can maintain access Requires DoD to maintain and Eliminates inter-model update comparisons

Heterogeneous, Privately- Fully leverages competitive Does not resolve IP disputes Developed industry directly

Minimizes DoD effort Requires inter-model comparison

Potential lack of continued access

Homogenous, Privately- Reduces IP disputes Introduces temporary Developed monopolies Partially leverages competitive industry Potential lack of continued access Reduces DoD effort Does not fully leverage Eliminates inter-model industry expertise comparisons Does not minimize DoD effort

In the MCE acquisition context, the government serves several roles: customer (thereby supplying financial incentive), the standards enforcer (regulatory incentive), and neutral mediator (serving as non-competing party with whom information can be shared). This combination of roles grants the DoD significant (but not complete) power over the ultimate shape of MCE acquisition. At the same time though, it places upon the DoD significant responsibility in working to make that shape a reality.

121

Chapter 7: Conclusions

In 2002, a study of 258 major transportation infrastructure projects found that almost 9 out of 10 of them went beyond the planned cost, even after disregarding additional costs caused by delays [213]. Cost of major defense acquisition projects are rising across services [1], [2]. This is clearly a major issue both in and outside of the defense sector. While the benefits of MCE will help address this issue as well as enable the design of more complex systems and systems of systems, it will also introduce new vulnerabilities, as has been shown in this work. Already cybersecurity attacks on aircraft manufacturing [111] and additive manufacturing processes [214] have been demonstrated. As MCE is further developed, these and other vulnerabilities will only increase in significance. This thesis examines how vulnerabilities can be conceptualized, how emergent behavior can be classified and integrated into the design process, what vulnerabilities MCE will introduce, and how to approach their mitigation. This chapter reviews the contributions of this research, discusses its limitations, and plots a course for future work.

7.1 Research Questions and Contributions

The research contribution of this thesis is an examination of vulnerabilities and vulnerability assessment as they pertain to model-centric environments. This led to the development of a Reference CEM and emergent behavior design patterns that aim to provide further understanding and interrogation of such vulnerabilities, including the socio-technical factors involved in human- model interaction.

As first stated in Section 1.3, this work was guided by the three central research questions. Those research questions are revisited and answered here.

1. What new programmatic vulnerabilities will model-centric engineering introduce in defense acquisition programs?

For all of its benefits, MCE will introduce vulnerabilities that are either non-existent or less severe in the historic, document-centric acquisition system. Chief among these are issues of cybersecurity, model assumptions, and model trust. While these vulnerabilities pose a challenge, they have thus far been overshadowed by the difficulties of implementing MCE. This work has provided additional insight into the topic, both through the use of CEM and through an examination of

123 emergent behavior, one of the potential sources of MCE-related vulnerabilities. The Reference CEM provides an additional tool to program managers and researchers as an entry point to analyze MCE-related programmatic vulnerabilities in various contexts. The emergent behavior typology provides a way of identifying and classifying the vulnerabilities introduced by the integrated nature of MCE environments. The identified vulnerabilities were generated and validated through a combination of expert interviews, usability testing, and literature case studies.

2. Are causal chains a useful way to conceptualize and approach these vulnerabilities?

Investigation suggests that program managers currently do not formally grapple with programmatic vulnerabilities as they do with vulnerabilities in the end-system. Many program managers mentally conceptualize vulnerabilities as discrete events with vague likelihoods Conceptualizing vulnerabilities as causal chains instead encourages a more structured and detailed consideration of programmatic vulnerabilities. This work made it clear that conceptualizing vulnerabilities as causal chains is not only viable, but quite useful for improving the ability to recognize the similarities and connections between vulnerabilities as well as to identify potential means of intervention.

3. What methods can individuals and enterprises use to mitigate these new vulnerabilities?

While the vulnerabilities identified in Chapter 4 are significant, individuals and MCE enterprises are not without tools to address them. Via the usability testing, the Reference CEM has been demonstrated to be useful for identifying and prioritizing interventions in MCE-related vulnerabilities. It can also be used to categorize vulnerabilities and promote understanding of the underlying connections between them. Similarly, the general CEM-VA framework has been adapted to apply to programmatic assessment.

Additionally, this work has proposed the use of emergent behavior design patterns as a method to approach vulnerability mitigation. While the understanding of complexity and emergent behavior has improved immensely over the past several decades, much of this understanding has not made it into the design and acquisition process. Furthermore, much of it remains sequestered in field- specific discussion (such as traffic in transportation engineering). The expression-based typology presented in this work can be coupled with design patterns and heuristics to enable comparison and discussion across fields and across levels of expertise, thereby enabling various parts of an 124 acquisition program to effectively communicate and collaborate. Furthermore, this typology can be used to identify and intervene in vulnerabilities introduced by emergent behavior, supplementing the CEM-VA framework.

Beyond the methods available to individuals involved in a defense acquisition program, three primary enterprise-level policy recommendations have been proposed: clearly defining ownership over the MCE environment, treating the acquisition process as an engineered system, and, in the DoD’s case, assuming a leadership role in establishing MCE standards. These policies have the potential to address many of the identified vulnerabilities and would either mitigate or wholly eliminate them.

7.2 Limitations

The limitations of CEM and CEM-VA have been discussed previously in in Section 3.2.2.2. With regards to this work in particular, additional limitations exist. For one, no two acquisition programs are alike, making the development of a generic Reference CEM difficult. Second, a fully-fledged MCE environment does not exist yet. The creation of CEM is typically based on extrapolation from knowledge derived from existing systems and on modeling of hypothetical future systems [82]. As a result, for unprecedented situations, such as an MCE acquisition environment, it may be incomplete. The degree to which the actual vulnerabilities present in a complete MCE defense acquisition process matches those identified in this work will depend on the specific form of that MCE environment.

That said, there is reason to believe that the general structure of an MCE environment will be similar to what has been laid out in this work and that many, if not all, of the identified vulnerability chains and interventions will be applicable. Much of MCE development and application is taking the form of extending the successes that MBSE brought to the systems engineering domain. As a result, the general structure and division of authority is being inherited from MBSE. NASA has laid out such a division, with the enterprise being responsible for developing the “modeling infrastructure” including standards and repositories [215]. As part of this, NASA’s JPL is developing their MCE environment out of Team X, which focuses on mission concept design [54]. A joint effort by NAVSEA and Huntington Ingalls Newport News Shipbuilding to develop an integrated model has followed an ownership division, data exchange structure, and IP protection

125 structure well in line with the concepts discussed in Section 2.2 [5].Similarly, the main commercial tools being developed in this field are based in MBSE. Vitech’s GENESYS and CORE, which was among the first software to implement MBSE principles, is continuing to grow in a quest to further the “model-centric transformation” and enable “virtual prototyping.” [216] Similarly SERC has made use of Analytical Graphics, Inc.’s STK, a multidiscipline modeling software, and joined it with the Unity gaming engine to develop an “Integrated Concept Engineering Framework.” [217] These efforts, along with those discussed in Section 2.2.2, suggest that MCE will develop out of the systems engineering community. This means that the final form of MCE is likely to be largely shaped by the preferences and goals of the same professional and research communities cited throughout this work. As a result of this, there is reason to believe that the Reference CEM fully reflects the general MCE-related vulnerabilities that will be present in the future.

One final limitation is that, while integration of CEM with various (more complicated) graph theory analysis methods have been discussed both here and previously [70], this has not yet been accomplished. These limitations are not to be viewed as insurmountable barriers to the use of CEM, but instead as potential avenues for future research. Several such avenues are discussed in the next section.

7.3 Future Research

Several directions of future research have been identified and are being pursued. Some of these are aimed at addressing the limitations of the previous section. Others seek to take advantage of opportunities that have presented themselves.

7.3.1 Interviews and Validation from Other Stakeholders

The interviews and usability testing in this work were performed with program managers and systems engineers (the people who “live in” in the MCE environment). In the future, additional viewpoints should be sought from MBSE and MCE tool developers, as well as from the leaders of enterprise MCE environments. The former may yield additional potential MCE-related vulnerabilities that are ‘invisible’ to the users of the MCE tools, particularly on the cybersecurity front, and possibility identify ways to intervene in or preemptively eliminate some of the vulnerabilities identified in the Reference CEM. This would be particularly useful in filling in the “gap” of interventions in the active modeling section of the Reference CEM noted in Section 6.1. 126 The latter would be useful for two reasons. First, it would provide insight into the feasibility of implementing the policy recommendations discussed in Section 6.2 and may uncover new policy options for preempting addressing some of the vulnerabilities. Second, it would potentially enable the creation of new CEMs, aimed at enterprise leaders rather than program managers, as there are doubtlessly organizational vulnerabilities that threaten the success of acquisition programs just as there are programmatic vulnerabilities.

7.3.2 Interactive Tool

An interactive version of the CEM, which enables easy sorting and adding vulnerabilities, is desired. This would make the method more accessible, similar to how NIST’s Cybersecurity Assessment Tool [218] makes the RMF [129] more approachable to small manufacturers. Additionally, it could serve as a platform for future usability testing of CEM in MCE programs. Such interactive tools have previously been found to be useful in both systems engineering [219] and in system project management [220] and would likely be similarly useful in using CEM for vulnerability assessment.

Such a tool would also be useful for developing larger, more complicated CEMs without losing legibility. Additionally, it would facilitate the integration of CEM with Dynamic Bayesian Networks and other graph theory techniques, as first proposed by Rovito [70].

7.3.3 Healthcare Industry Comparison

There is some indication that program managers may be well served by observing fields that have somewhat analogous aspects with defense acquisition, in order to derive helpful metaphors [201] or lessons learned [221].

The healthcare industry shows promise for such an analogy to cybersecurity in MCE environments. The healthcare industry deals with sensitive information, computer equipment, and high pressure environments. All of these are present at numerous stages of operation. Patient records have to be transferred from one system to another and be available to medical practitioners, but have to be restricted to authorized personnel. Researchers need to be able to query systems in order to provide improved medical treatment, but cannot violate individuals’ privacy. They must

127 do all this and more while under constant threat of cyberattack, as recent events have shown [222]– [224].

Engineers and researchers have made significant headway in making medical devices more interoperable with one another, particularly when it comes to sharing data securely [225]. Increasingly, model-based methods are being used to assess and design medical systems [226]. An example of this is the FDA’s Sentinel Initiative, which seeks to enable active querying of medical data while preserving individual privacy [125].

All of these endeavors are strikingly similar to the challenges currently faced in defense acquisition. This suggests that there may be benefit in conducting a systematic comparison of the two fields, perhaps via the use of comparative CEMs.

7.3.4 Cybersecurity

As mentioned in Section 7.3.1, one of the benefits of interviews with MCE tool makers would be the identification of additional cybersecurity vulnerabilities and interventions. Beyond this, additional work remains to be done to address cybersecurity vulnerabilities. Potential next steps include applying CEM to some of the CVE entries to see if CEM is similarly useful for visualizing non-programmatic cybersecurity vulnerabilities.

7.3.5 Case Study Applications

As discussed in Section 2.2.2, several groups, including SRPO, NAVAIR, the US Air Force, and JPL’s Team X are all actively developing, testing, and applying MCE practices in their programs. It is hoped that CEM-VA and/or the Reference CEM could be tested in one of these (or another) real-world contexts. This would not only enable the gathering additional data that could contribute to a more detailed and robust Reference CEM, but it would provide full scale validation of the framework.

128 Appendix A (Reference CEM)

This appendix contains detailed information about the Reference CEM presented in Section 4.3.

Table A-1. Reference CEM Events and Descriptions (Part 1 of 4)

Event Name Event Type Event Description Accidental/Malicious Data External Trigger Either accidentally or intentionally, some amount of sensitive data Release involving the program has been released to individuals or groups not cleared to access such information

Budget Cut External Trigger Current or projected funding for this program or for the organization as a whole is being reduced

Congressional Scrutiny External Trigger Congress has become increasingly concerned with the management or status of this program or of defense programs in general Cyberattack Attempt External Trigger Some individual or group seeks to disrupt or surveil the program using a cyberattack Economic Crunch External Trigger Reduced consumption, reduced willingness to lend, raising unemployment or other forms of economic recession or depression are occurring Foreign Actor Action External Trigger A foreign actor has taken some significant action that impacts the intended use of the system that the program is acquiring

Hiring Freeze External Trigger The organization has ceased all new hiring for some period of time List of Approved Modeling External Trigger The organization maintains a list of software approved for use in Software Changes programs and has changed (added and/or removed) certain software from this list Modeling Software Becomes External Trigger Some modeling software used by the program which is purchased More Expensive or licensed from an external provider has become more expensive in the upcoming version/renewal Modeling Software No Longer External Trigger The maintainer/developer of some modeling software used by the Supported program has ceased issuing updates and/or new versions Public Scrutiny External Trigger The public and/or news media has become increasingly concerned with the management or status of this program or of defense programs in general Strategic Realignment External Trigger The strategic interests of the client or other stakeholders in the program have changed Sufficient IP Rights Not Acquired External Trigger A previous project with relevant components or systems that could in Contract be reused did not acquire sufficient intellectual property rights to make those components available for reuse Technological Change External Trigger A significant new technology has been developed or put into application that impacts the program in some way Unexpected Technological Hurtle External Trigger During the program, some desired technology either is unexpectedly unavailable or is taking an unexpectedly long time to develop

129 Table A-1. Reference CEM Events and Descriptions (Part 2 of 4)

Event Name Event Type Event Description Attrition Intermediary Event Reduction in experienced staff and/or total staff available to the program Change of Modeling Domain Intermediary Event The program must now model the system, subsystem, or component in a different environment or in a different manner than was previously done Change of Modeling Software Intermediary Event The program must now change the modeling software used Needed Component Tampering Intermediary Event A digital or physical component of the system is maliciously altered to harm the program Countermeasures/Competitors Intermediary Event Other individuals or organizations are able to either develop Developed countermeasures to the system being acquired or developing competing systems Current or Future Contracts Intermediary Event The program is either cancelled by the client or future programs Canceled/Avoided by Customer are never initiated Delay in Integrating New Models Intermediary Event Increased time is required to integrate a new modeling software or into Suite model into the MCE environment Development Stall Due to Lack of Intermediary Event A critical technology for the program is not available and there is Critical Technology not means of altering the design to circumvent the need for the technology Diminished Enterprise Model Intermediary Event Reduced ability of the enterprise or program to track, integrate, Curation Capability document, and improve the modeling environment Documentation Tampering Intermediary Event Documentation of assumptions or other important aspects of models is maliciously altered to harm the program Gap in Subject-Domain Intermediary Event Among program staff there is a lack of knowledge on a particular Knowledge subject area of relevance to the program

Inability to Fully Integrate Intermediary Event One or more models in use by the program cannot be integrated Models into the MCE environment and thus remain "stove-piped" Inability to Oversee Contractor Intermediary Event Program staff no longer have the ability to effectively understand Models/Designs and vet the models and/or designs provided by the contractor Inability to Reuse Models in Intermediary Event Models from previous or current programs are not available to be Future Projects used in future programs Inaccurate Simulations / Intermediary Event Models and simulations do not accurately reflect real-world Performance Predictions behavior or performance Increased Cybersecurity Intermediary Event The program and/or MCE environment has become more Vulnerabilities susceptible to cyberattacks Increased Testing Required by Intermediary Event The client increases the amount of testing required during Customer verification to demonstrate system readiness Industry Partners Unwilling to Intermediary Event Subcontractors, suppliers, or other members of industry are Share Information unwilling to share information with the program due to concerns that it will be misused or inappropriately shared Layoffs/Incentivized Retirement Intermediary Event Staff experienced with the MCE environment or having general of Experienced Staff relevant experience are laid off or incentivized to retire Less Model Expertise Intermediary Event Among program staff there is either a reduced total amount of experience or reduced average amount of experience with the MCE environment

130 Table A-1. Reference CEM Events and Descriptions (Part 3 of 4)

Event Name Event Type Event Description Loss of Historical Data Intermediary Event Data from current or previous programs has been lost or corrupted such that it is unavailable for future reference or reuse Malicious Insertion Intermediary Event A new digital or physical object is inserted into the model or system with intent to harm system integrity Missing/Improper Requirements Intermediary Event The current set of requirements do not sufficiently match stakeholder needs, either due to a lack of relevant requirements or due to a conflict between the requirements and needs Misunderstood Model Intermediary Event Some member of the program staff misunderstands or fails to Assumptions consider one or more of the assumptions underlying the model in such a way as to potentially impact the program Modeling Tampering Intermediary Event The models themselves are maliciously altered to harm the program Needs Change Intermediary Event The needs of the client stakeholder(s) are changing in a way that impacts the program New Software Created or Intermediary Event A new modeling or simulation software must be created or Selected selected (if COTS) and integrated into the MCE environment Over-Trust in Model Intermediary Event One or more members of the program staff trust the current state of the model or results of simulation when these do not represent reality in some significant manner Physical Component Substitution Intermediary Event A physical component in the system being acquired is substituted, threatening system integrity Poor Model Documentation Intermediary Event The model documentation is missing relevant information or presented in an inaccessible manner Preferred Modelling Intermediary Event The preferred modeling software or system, either due to prior Software/System Unavailable experience or particular relevance to the program, is unavailable for use in this program Pressure to Streamline Intermediary Event External pressure, either from the organization, client, or other Acquisition Process stakeholder, is exerted on the program manager to minimize costs, timing, and "bloat." Reduced Confidence in System Intermediary Event Members of the program, organization, or clients have reduced confidence in the ability of the MCE environment to be able to operate effectively and securely Reduced Model Training Intermediary Event Reduction in training in the use of the MCE environment is available to program personnel Reduced Modeling Speed Intermediary Event Time required to effectively use the modeling software is increased Reduction of "Upkeep" and Intermediary Event Reduction or elimination of "unessential" procedures, tests, and Indirect Costs personnel. Particularly likely targets include any procedures aimed at improving reuse or other programs Reputation Harm Intermediary Event The professional, political, or public reputation of the program, organization, or client has become significantly harmed Software Goes Without Updates Intermediary Event Software important to the MCE environment goes without updates or patches for longer than ideal, resulting in lack of capability or security concerns Successful Cyberattack Intermediary Event A cyberattack was not repelled or stopped until after it significantly impacted the program

131 Table A-4 Reference CEM Events and Descriptions (Part 4 of 4)

Event Name Event Type Event Description System Design Changes to Work Intermediary Event A critical technology for the program is not available and thus the Around Lack of Technology design of the system must be altered in order to circumvent the need for the technology System Integrity Reevaluation Intermediary Event The program or organization must reevaluate the integrity of the MCE environment and/or the system being designed to ensure that it has not been compromised Theft/Exposure of Information Intermediary Event Sensitive or otherwise non-public information is either intentionally stolen or otherwise made available to those uncleared to possess it Under-Trust in Model Intermediary Event One or more members of the program staff do not place do not believe the current state of the model or results of simulation when these do represent reality to a sufficient extent Under-Use of Model Intermediary Event The program does not make full effective use of the MCE environment available Unnecessary Rework Intermediary Event Work previously accomplished in the current program or a previous one must be done again when it should not have to be Unnoticed Modeling Software Intermediary Event Some modeling bug, error, or other form of inaccuracy develops Flaw and goes undetected by the model curation staff and/or the program Weak Security Controls Intermediary Event No or few security controls are in place to prevent physical or virtual security compromises Compromised System Terminal Event The system is put into operation, but suffers from lack of integrity

Delays Terminal Event Acquisition of the program is delayed and/or increased costs incurred Failure During Terminal Event The system fails during verification and/or validation, prompting Verification/Validation redesign to occur or cancellation of the program Field Failure Terminal Event The system passes verification or validation but fails during operation Loss of Contract Terminal Event The program is cancelled

Loss of Technological/Strategic Terminal Event The system does not provide the advantage or superiority that it Advantage was intended to Over-conservative System Terminal Event The system is "over-engineered" and thus more costly and/or has reduced performance that should be possible System Not Acquired Terminal Event The program is cancelled and the system is not acquired

132

Figure A-1. Legend and Intervention Points of the Reference CEM

133

Figure A-2. Part A of the Reference CEM

134

Figure A-3. Part B of the Reference CEM

/

135

Figure A-4. Part C of the Reference CEM

136

Figure A-5. Full Matrix Representation of the Reference CEM 137

Appendix B (Glossary)

This section provides a centralized list of some of the terms used in this work and their definitions. The definitions provided here are, unless otherwise noted, how the terms are used in this work specifically. They should not be interpreted as having a universal or normative claim. The indicated sections for each term are where the term first appears or is defined, not the complete set of locations that it occurs. That is to say, this is not an index.

Term Definition Section

acquisition The conceptualization, initiation, design, development, test, 2.1 contracting, production, deployment, integrated product support (IPS), modification, and disposal of weapons and other systems, supplies, or services (including construction) to satisfy DoD needs, intended for use in, or in support of, military missions.

analysis An evaluation, either quantitative or qualitative; synonymous with 3.1 assessment.

assessment Synonymous with analysis. 3.1

causal chain A series of events, with each event causing or being an integral part of 3.2.1.5 the cause, or the next “link” in the chain.

causal factor Any aspect of the system which, when removed or changed, is likely 5.3 to reduce the occurrence of emergent behavior, or, when induced, is likely to increase the occurrence of emergent behavior.

classification A generic term for sorting a set by some defined metric, either 5.1 quantitative or qualitative. Includes both taxonomies and typologies.

complexity A measure of a systems scale, number of interacting parts, and non- 5.2.2 linear behavior.

design heuristic A subset of design principles that are general, unverifiable rules-of- 5.3 thumb usually aimed at the design process.

design pattern A subset of design principles that are aimed at specific problems, 5.3 apply to the end-system itself, and have some degree of provability or verifiability.

design principle An abstract rule that produces concrete solutions to design problems. 5.3

139 Digital Systems A digital representation of a defense system, generated by all 2.2.1.2 Model stakeholders that integrates the authoritative technical data and associated artifacts which define all aspects of the system for the specific activities throughout the system lifecycle.

Digital Tapestry Synonymous with Digital Thread. 2.2.1.2

Digital Thread An extensible, configurable and component enterprise-level analytical 2.2.1.2 framework that seamlessly expedites the controlled interplay of authoritative technical data, software, information, and knowledge in the enterprise data-information-knowledge systems, based on the Digital System Model template, to inform decision makers throughout a system's life cycle by providing the capability to access, integrate and transform disparate data into actionable information.

Digital Twin An integrated multiphysics, multiscale, probabilistic simulation of an 2.2.1.3 as-built system, enabled by Digital Thread, that uses the best available models, sensor information, and input data to mirror and predict activities/performance over the life of its corresponding physical twin.

emergence A macroscale behavior that is in some way linked to a microscale 5.1 behavior. Often is difficult to predict and lacks a defined structure.

end-system The system that an acquisition program is acquiring. 2.3

epistemology The study of the what is known and how it is known. 5.2.1

external trigger A hazard external from the perspective of a defined user; sometimes 3.2.1.5 referred to as a spontaneous event.

framework A method of assessing and analyzing hazards, vulnerabilities, and 3.1 (in vulnerability risks in a comprehensive manner, as well as determining appropriate assessment) mitigations and countermeasures.

hazard A system or environmental state that has the potential to disrupt the 3.1 system.

intermediary Any unintended state change of a system’s form or operations which 3.2.1.5 event could jeopardize value delivery of the program and is situated along a causal chain connecting an external trigger to a terminal event. intervention point A means of disrupting or mitigating a vulnerability chain. 3.2.1.5

method A procedure of way of doing something. 3.1

140 model-centric An overarching digital engineering approach that integrates different 2.2 engineering model types with simulations, surrogates, systems and components at different levels of abstraction and fidelity across disciplines throughout the lifecycle.

ontology The study of the fundamental nature of reality. 5.2.1

program A directed, funded effort that provides a new, improved, or continuing 2.3 materiel, weapon or information system, or service capability in response to an approved need.

project A planned undertaking having a finite beginning and ending, 2.3 involving definition, development, production, and logistics support (LS) of a major weapon or weapon support system or systems. A project may be the whole or a part of a program. Sometimes used interchangeably with program.

risk A measure of the probability of a system disruption, the consequences 3.1 of that disruption, and sometimes other factors such as detectability or criticality.

system of A specific method of classification 5.1 classification

taxonomy A system of classification in which multiple means of division are 5.1 consecutively applied.

technique A method for analyzing a particular set of hazards, vulnerabilities, or 3.1 (in vulnerability risks. assessment)

terminal event Any form of system degradation or failure, or of value loss. 3.2.1.5

typology A system of classification in which multiple means of division are 5.1 applied simultaneously and in a non-hierarchal manner.

vulnerability The means by which the hazard might disrupt the system. 3.1

vulnerability A conceptualization and representation of a vulnerability as a causal 3.2.1.5 chain chain.

141

References

[1] M. V Arena, O. Younossi, K. Brancato, I. Blickstein, and C. a Grammich, Why has the cost of fixed-wing aircraft risen? Santa Monica, CA: The RAND Corporation, 2008. [2] M. V. Arena, I. Blickstein, O. Younossi, and C. a. Grammich, Why Has the Cost of Navy Ships Risen? Santa Monica, CA: The RAND Corporation, 2006. [3] T. D. West and A. Pyster, “Untangling the Digital Thread: The Challenge and Promise of Model-Based Engineering in Defense Acquisition,” INSIGHT, vol. 18, no. 2, pp. 45–55, 2015. [4] B. Puchek et al., “Digital Model-based Engineering: Expectations , Prerequisites , and Challenges of Infusion.” Office of the Deputy Assistant Secretary of Defense, Washington D.C., 2017 [Online] Available: https://www.acq.osd.mil/se/docs/DMbE-20170524- 0602.pdf Accessed Dec./12/2017. [5] M. Clifford, M. Blackburn, D. Verma, and P. Zimmerman, “Model-Centric Engineering – Insights and Challenges: Primary Takeaways from a Government-Industry Forum.” Systems Engineering Research Center, Hoboken, NJ, 2016. [6] J. P. Hale, “NASA Integrated Model-Centric Architecture.” Huntsville, AL, 2014 [Online] Available: https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20140006456.pdf Accessed Apr./23/2018. [7] T. J. Bayer, M. Bennett, C. L. Delp, D. Dvorak, J. S. Jenkins, and S. Mandutianu, “Update - Concept of operations for integrated model-centric engineering at JPL,” in IEEE Aerospace Conference Proceedings, 2011. [8] Department of Defense, “Glossary: Defense Acquisition Acronyms and Terms.” Defense Acquisition University Press, Fort Belvoir, VA, 2015. [9] E. Hildinger, Swords Against the Senate: The Rise of the Roman Army and the Fall of the Republic. Da Capo Press, 2002. [10] C. D. Hazen, The French Revolution and Napoleon. New York City, NY: H. Holt and Company, 1917. [11] C. H. Fitch, Report on the Manufacture of Fire-arms and Ammunition. Washington D.C.: United States Government Printing Office, 1882. [12] P. Baida, “Eli Whitney’s Other Talent,” American Heritage, Rockville, MD, 1987, http://www.americanheritage.com/content/eli-whitney’s-other-talent?page=show. [13] J. A. Stockfisch, “The Intellectual Foundations of Systems Analysis.” The RAND Corporation, Santa Monica, CA, 1987. [14] Systems Engineering Book of Knowledge, “History of Systems Science,” 2017. [Online] Available: http://www.sebokwiki.org/wiki/History_of_Systems_Science#Open_Systems_and_Gener al_Systems_Theory Accessed May/06/2018. [15] A. D. Hall, A Methodology for Systems Engineering, 4th ed. Princeton, NJ: D. Van Nostrand Company, Inc., 1965.

143

[16] E. V. Converse, History of Acquisition in the Department of Defense: Rearming for the Cold War 1945-1960, vol. 1. Washington D.C.: Historical Office of the Office of the Secretary of Dense, 2012. [17] J. R. Fox, Defense Acquisition Reform, 1960-2009. An Elusive Goal. Washington D.C.: Center of Military History, 2011. [18] AcqNotes, “Acquisition Process Overview,” 2010. [Online] Available: http://acqnotes.com/acqnote/acquisitions/acquisition-process-overview Accessed Apr./21/2018. [19] AcqNotes, “Defense Acquisition Life Cycle Compliance Baseline (Pre‐Tailoring): Acquisition & Procurement Milestones, Phases and Decision Points.” [Online] Available: http://acqnotes.com/wp-content/uploads/2014/09/Interactive-Lifecycle-Wall-Chart-Jan- 18.pdf Accessed Apr./21/2018. [20] N. R. Augustine, Augustine’s laws, 6th ed. Reston, VA: American Institute of Aeronautics and Astronautics, Inc., 1997. [21] R. S. Lineberger and A. Hussain, “Program management in aerospace and defense: Still late and over budget.” Deloitte Development LLC, 2016. [22] United States Government Accountability Office, “Defense Acquisitions: Department of Defense Actions on Program Manager Empowerment and Accountability,” GAO-08-62R. Washington D.C., pp. 109–364, 2007 [Online] Available: https://www.gao.gov/assets/100/95239.pdf Accessed Apr./21/2018. [23] W. P. Rogerson, “Economic Incentives and the Defense Procurement Process,” Journal of Economic Perspectives, vol. 8, no. 4, pp. 65–90, Nov. 1994. [24] United States Army Center of Military History and Industrial College of the Armed Forces, Providing the Means of War: Perspectives on Defense Acquisition 1945-2000. Washington D.C.: US Department of Defense, 2005. [25] C. Wasson and C. Neill, “Strategic Technical Planning WG: Progress and Status,” in INCOSE International Symposium, 2017. [26] J. Ramón San Cristóbal Mateo, “Multi-Attribute Utility Theory,” in Multi Criteria Analysis in the Renewable Energy Industry, London, UK: Springer, 2012, pp. 63–72. [27] M. E. Fitzgerald and A. M. Ross, “Recommendations for Framing Multi-Stakeholder Tradespace Exploration,” in INCOSE International Symposium, 2016. [28] A. M. Ross and D. H. Rhodes, “Using Natural Value-Centric Time Scales for Conceptualizing System Timelines through Epoch-Era Analysis,” in INCOSE International Symposium, 2008. [29] J. Holt and S. Perry, SysML for Systems Engineering: A model-based approach, 1st ed. London, UK: The Institution of Engineering and Technology, 2008. [30] C. Piaszczyk, “Model Based Systems Engineering with Department of Defense Architectural Framework,” Systems Engineering, vol. 14, no. 3, pp. 305–326, 2011.

144

[31] J. B. Reid and D. H. Rhodes, “Digital System Models : An investigation of the non- technical challenges and research needs,” in Conference on Systems Engineering Research, 2016. [32] E. H. Glaessgen and D. Stargel, “The Digital Twin paradigm for future NASA and US Air Force vehicles,” in Structual Dynamics and Materials Conference, 2012, pp. 1–14. [33] M. Blackburn et al., “Transforming Systems Engineering through Model-Centric Engineering: Technical Report SERC-2017-TR-110.” Systems Engineering Research Center, Hoboken, NJ, 2017. [34] T. Kellner, “Wind in the Cloud? How the Digital Wind Farm Will Make Wind Power 20 Percent More Efficient,” GE Reports, General Electric, Oct-2015, http://www.gereports.com/post/119300678660/wind-in-the-cloud-how-the-digital-wind- farm-will/. [35] Lockheed Martin, “Digital Tapestry,” 2015. [Online] Available: http://www.lockheedmartin.com/us/what-we-do/emerging/advanced- manufacturing/digital-tapestry.html Accessed Mar./22/2017. [36] S. Friedenthal, R. Griego, and M. Sampson, “INCOSE model based systems engineering (MBSE) initiative,” in INCOSE International Symposium, 2007. [37] P. Zimmerman, “MBSE in the Department of Defense.” Office of the Deputy Assistant Secretary of Defense for Systems Engineering, Greenbelt, MD, 2015. [38] J. Broughton, F. Abraham, N. Bernstein, and E. Kaxiras, “Concurrent coupling of length scales: methodology and application,” Physical Review B - Condensed Matter and Materials Physics, vol. 60, no. 4, pp. 2391–2403, 1999. [39] J. R. Bursten, “Surfaces, Scales, and Synthesis: Scientific Reasoning at Nanoscale,” University of Pittsburgh, 2015. [40] J. S. Jenkins and N. F. Rouquette, “Semantically-Rigorous Systems Engineering Modeling Using SysML and OWL,” in International Workshop on Systems & Concurrent Engineering for Space Applications , 2012. [41] M. D. Curry, “Design as a Search Problem: Interactive Visualization for System Design Under Uncertainty,” Massachusetts Institute of Technology, 2017. [42] D. Rhodes and A. Ross, “Interactive Model-Centric Systems Engineering (IMCSE) Pathfinder Workshop Report,” Systems Engineering Research Center, Systems Engineering Advancement Research Initiative, 2015. [43] L. Reymondet, “A Framework for Sense-Making of Complex Sociotechnical Systems,” Massachusetts Institute of Technology, 2016. [44] L. Reymondet, A. M. Ross, and D. H. Rhodes, “Considerations for Model Curation in Model-Centric Systems Engineering,” in IEEE International Systems Conference, 2016. [45] B. Evans, “Modeling and Simulation Applied in the F-35 Program.” Lockheed Martin, Fort Worth, TX, 2011. [46] D. W. Nixon, “Flight Control Law Development for the F-35 Joint Strike Fighter.” Lockheed Martin, 2004. 145

[47] Lockheed Martin, “The Digital Thread Key to F35 Joint Strike Fighter Affordability,” 2010. [Online] Available: https://www.onlineamd.com/article/amd-080910-f-35-joint- strike-fighter-digital-thread Accessed Sep./22/2015. [48] Government Acountability Office, “Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities.,” Report to Congressional Committees. Washington D.C., 2014. [49] Department of Operational Test & Evaluation, “F-35 Joint Strike Fighter FY16 Operation.” US Department of Defense, Washington D.C., 2017. [50] S. Arevalo et al., “A new DoD initiative: the Computational Research and Engineering Acquisition Tools and Environments (CREATE) program,” Journal of Physics: Conference Series, vol. 125, 2008. [51] E. M. Kraft, “HPCMP CREATE TM-AV and the Air Force Digital Thread,” SciTech. American Institute of Aeronautics and Astronautics, Kissimmee, FL, 2015. [52] G. L. Roth, J. W. Livingston, M. Blair, and R. Kolonay, “CREATE-AV DaVinci: Computationally Based Engineering for Conceptual Design,” in AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition, 2010. [53] M. Blackburn et al., “Transforming Systems Engineering through Model-Centric Engineering: Technical Report SERC-2018-TR-103.” Systems Engineering Research Center, Hoboken, NJ, 2018 [Online] Available: http://www.sercuarc.org/wp- content/uploads/2014/05/A013_SERC-RT-170_Technical-Report-SERC-2018-TR- 103.pdf Accessed Apr./25/2018. [54] D. A. Wagner, M. B. Bennett, R. Karban, N. Rouquette, S. Jenkins, and M. Ingham, “An ontology for state analysis: Formalizing the mapping to SysML,” IEEE Aerospace Conference Proceedings, 2012. [55] N. Rouquette and S. J. I. Herzig, “JPL’s Integrated Model-Centric Engineering,” GitHub, Inc., 2018. [Online] Available: https://github.com/JPL-IMCE Accessed Apr./23/2018. [56] J. Knizhnik, “FY17 Model Based Sounding Rocket Kick Off.” National Aeronautics and Space Administration, 2016. [57] A. Chulaki and M. Kuznetsova, “About CCMC,” Community Coordinated Modeling Center. [Online] Available: https://ccmc.gsfc.nasa.gov/about.php Accessed Apr./23/2018. [58] Model Based Engineering Subcommittee, “Final Report of Model Based Engineering (MBE) Subcommittee,” National Defense Industrial Association, 2011. [59] E. J. Tuegel, A. R. Ingraffea, T. G. Eason, and S. M. Spottswood, “Reengineering Aircraft Structural Life Prediction Using a Digital Twin,” International Journal of Aerospace Engineering, vol. 2011, 2011. [60] S. E. German and D. H. Rhodes, “Model-centric decision-making: exploring decision- maker trust and perception of models,” in Conference on Systems Engineering Research, 2017. [61] The MITRE Corporation, “Terminology,” Common Vulnerabilities and Exposures, 2015. [Online] Available: https://cve.mitre.org/about/terminology.html Accessed Feb./20/2018.

146

[62] US Department of Defense, DoD Dictionary of Military and Associated Terms. Washington D.C., 2017. [63] Dictionary.com, “vulnerable.” [Online] Available: http://www.dictionary.com/browse/vulnerability?s=t Accessed Oct./25/2017. [64] The MITRE Corporation, “CVE-2017-5753,” Common Vulnerabilities and Exposures, 2017. [Online] Available: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017- 5753 Accessed Feb./20/2018. [65] Dictionary.com, “risk,” 2018. [Online] Available: http://www.dictionary.com/browse/risk?s=t Accessed Feb./23/2018. [66] Y. Y. Haimes, “On the Definition of Vulnerabilities in Measuring Risks to Infrastructures,” Risk Analysis, vol. 26, no. 2, pp. 293–296, Apr. 2006. [67] J. Johansson and H. Hassel, “An approach for modelling interdependent infrastructures in the context of vulnerability analysis,” Reliability Engineering & System Safety, vol. 95, no. 12, pp. 1335–1344, Dec. 2010. [68] T. Glade, “Vulnerability Assessment in Landslide Risk Analysis,” Die Erde, vol. 134, no. 2, pp. 123–146, 2003. [69] P. E. Waggoner et al., “A framework for sustainability science: a renovated IPAT identity.,” Proceedings of the National Academy of Sciences of the United States of America, vol. 99, no. 12, pp. 7860–5, Jun. 2002. [70] S. M. Rovito, “An Integrated Framework for the Vulnerability Assessment of Complex Supply Chain Systems,” Massachusetts Institute of Technology, 2016. [71] Dictionary.com, “method,” 2018. [Online] Available: http://www.dictionary.com/browse/method?s=t Accessed Apr./21/2018. [72] C. A. Ericson, “Fault Tree Analysis,” in Hazard Analysis Techniques for System Safety, Hoboken, NJ, USA: John Wiley & Sons, Inc., 2005, pp. 183–221. [73] W. Kröger and E. Zio, Vulnerable Systems. London: Springer London, 2011. [74] N. Tague, “Failure Mode Effects Analysis,” ASQ, 2004. [Online] Available: http://asq.org/learn-about-quality/process-analysis-tools/overview/fmea.html Accessed Jan./17/2017. [75] J. Costa, “Systems pathology: A critical review,” Molecular Oncology, vol. 6, no. 1, pp. 27–32, Feb. 2012. [76] L. Troncale, “Would A Rigorous Knowledge Base in Systems Pathology Add Significantly to the Systems Engineering Portfolio?,” in Proceedings of the Conference on Systems Engineering Research, 2011. [77] L. Troncale, “Systems Processes and Pathologies: Creating An Integrated Framework for Systems Science,” in INCOSE International Symposium, 2013, pp. 1330–1353. [78] H. L. Davidz, “Systems Engineering Pathology : Leveraging Science to Characterize Dysfunction,” in Conference on Systems Engineering Research, 2017. [79] N. Leveson, “An STPA Primer.” Cambridge, MA, 2013.

147

[80] B. Mekdeci, A. M. Ross, D. H. Rhodes, and D. E. Hastings, “A taxonomy of perturbations: Determining the ways that systems lose value,” in 2012 IEEE International Systems Conference, Proceedings, 2012, pp. 507–512. [81] S. M. Rovito and D. H. Rhodes, “Enabling Better Supply Chain Decisions Through a Generic Model Utilizing Cause-Effect Mapping,” in 2016 Annual IEEE Sytems Conference, Proceedings, 2016. [82] B. Mekdeci, “Managing the Impact of Change Through Survivability and Pliability to Achieve Viable Systems of Systems,” Massachusetts Institute of Technology, 2013. [83] J. Prous, “Beyond FMEA: The structured what-if technique (SWIFT),” American Society for Healthcare Risk Management, vol. 31, no. 4, pp. 23–29, 2012. [84] O. Helmer, “Analysis of the future: The Delphi Method.” Rand Corporation, Santa Monica, CA, 1967. [85] J. M. English and G. L. Kernan, “The prediction of air travel and aircraft technology to the year 2000 using the Delphi method,” Transportation Research, vol. 10, pp. 1–8, 1976. [86] T. R. Browning, “Sources of schedule risk in complex system development,” Systems Engineering, vol. 2, no. 3, pp. 129–142, 1999. [87] R. Valerdi, “Convergence of expert opinion via the wideband delphi method: An application in cost estimation models,” in INCOSE International Symposium, 2011. [88] US Department of Defense, “DoD Instruction 5200.44.” Washington D.C., 2017 [Online] Available: http://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/520044p.pdf Accessed Apr./21/2018. [89] Deputy Assistant Secretary of Defense for Systems Engineering and Department of Defense Chief Information Officer, “Trusted Systems and Networks (TSN) Analysis.” Washington D.C., 2014 [Online] Available: www.acq.osd.mil/se Accessed Apr./21/2018. [90] J. B. Reid and D. H. Rhodes, “Assessing Vulnerabilities in Model-Centric Acquisition Programs Using Cause-Effect Mapping,” in Acquisition Research Symposium, 2018. [91] P. S. Antón, R. H. Anderson, R. Mesic, and M. Scheiern, Finding and Fixing Vulnerabilities in Information Systems: The Vulnerability Assessment and Mitigation Methodology. Santa Monica, CA: The RAND Corporation, 2004. [92] RAND Corporation, “Finding and Fixing Vulnerabilities in Information Systems: VAMM Tool License Agreement,” 2003. [Online] Available: https://www.rand.org/pubs/monograph_reports/MR1601/download1.html Accessed Apr./21/2018. [93] C. M. Schnaubelt, V. Larson, Eric, and M. E. Boyer, Vulnerability Assessment Method Pocket Guide. Santa Monica, CA: RAND Corporation, 2014. [94] MITRE Corporation, “Cyber Mission Assurance Engineering: A Risk-Based, Threat- Informed Approach to Address Advanced Adversaries.” Bedford, MA, 2013 [Online] Available: https://www.mitre.org/sites/default/files/publications/cyber-mae.pdf Accessed Apr./25/2018.

148

[95] The MITRE Corporation, The MITRE Systems Engineering Guide. Bedford, MA: MITRE Corporate Communications and Public Affairs, 2014. [96] B. Arkin, S. Stender, and G. McGraw, “Software penetration testing,” IEEE Security and Privacy, vol. 3, no. 1, pp. 84–87, 2005. [97] E. Schwarz, “Automating Vulnerability Discovery in Critical Applications,” Software Engineering Institute, 2018. [Online] Available: https://www.sei.cmu.edu/research- capabilities/all-work/display.cfm?customel_datapageid_4050=6487 Accessed Mar./27/2018. [98] C. A. Meadows, “Formal verification of cryptographic protocols: A survey,” in International Conference on the Theory and Application of Cryptology, 1994, pp. 133– 150. [99] C. Kern and M. R. Greenstreet, “Formal verification in hardware design: a survey,” ACM Transactions on Design Automation of Electronic Systems, vol. 4, no. 2, pp. 123–193, Apr. 1999. [100] S. Morimoto, “A Survey of Formal Verification for Business Process Modeling,” Springer, Berlin, Heidelberg, 2008, pp. 514–522. [101] M. Kamali, L. A. Dennis, O. McAree, M. Fisher, and S. M. Veres, “Formal verification of autonomous vehicle platooning,” Science of Computer Programming, vol. 1, pp. 1–19, 2016. [102] G. Pope and M. Yampolskiy, “A Hazard Analysis Technique for Additive Manufacturing,” in Better Software East Conference, 2016. [103] G. Pope, “A Hazard Analysis Technique for the Internet of Things ( IoT ) and Mobile,” in STAMP Workshop, 2017. [104] W. E. Young, “A System Saftey Approach to Assuring Air Operations Against Cyber Disruptions,” in STAMP Workshop, 2013. [105] W. E. Young and R. Porada, “System-Theoretic Process Analysis for Security (STPA- SEC): Cyber Security and STPA,” in STAMP Workshop, 2017. [106] G. Pope, “Combining STPA with Compiler Technology to Identify Vulnerabilities and Hazards in Software-Controlled Systems,” in STAMP Workshop, 2018. [107] S. Hanselman, “Everything’s broken and nobody’s upset,” Hanselman Blog, 2012. [Online] Available: https://www.hanselman.com/blog/EverythingsBrokenAndNobodysUpset.aspx Accessed Mar./27/2018. [108] S. Gressin, “The Equifax Data Breach: What to Do,” Federal Trade Commission Consumer Information, 2017. [Online] Available: https://www.consumer.ftc.gov/blog/2017/09/equifax-data-breach-what-do Accessed Mar./27/2018. [109] N. Raymond, “U.S. charges Chinese-Canadian citizen with trade secret theft,” Reuters2, 31-Aug-2017, https://ca.reuters.com/article/topNews/idCAKCN1BB2K8-OCATP.

149

[110] J. Hanna, C. Smythe, and C. Martin, “China’s Sinovel Convicted in U.S. of Stealing Trade Secrets,” Bloomberg, New York City, NY, 24-Jan-2018, https://www.bloomberg.com/news/articles/2018-01-24/chinese-firm-sinovel-convicted-in- u-s-of-trade-secret-theft. [111] N. Statt, “Boeing production plant hit with WannaCry ransomware attack,” The Verge, New York City, NY, Mar-2018, https://www.theverge.com/2018/3/28/17174540/boeing- wannacry-ransomware-attack-production-plant-charleston-south-carolina. [112] C. Irvine and T. Parfitt, “Kremlin returns to typewriters to avoid computer leaks,” The Telegraph, Jersey, UK, 11-Jul-2013, https://www.telegraph.co.uk/news/worldnews/europe/russia/10173645/Kremlin-returns- to-typewriters-to-avoid-computer-leaks.html. [113] NASA Office of the Chief Engineer, “NASA Public Lessons Learned System,” NASA, 1994. [Online] Available: https://llis.nasa.gov/ Accessed Jul./13/2017. [114] E. Rouwette and N. Ghaffarzadegan, “The system dynamics case repository project,” System Dynamics Review, vol. 29, no. 1, 2013. [115] J. D. Sterman, “Coflows and Aging Chains,” in Business Dynamics: Systems Thinking and Modeling for a Complex World, Boston, MA: Irwin McGraw-Hill, 2000, pp. 469–512. [116] D. Hall, “Risk Identification Challenge,” in INCOSE 2018 International Workshop, 2018. [117] Software Engineering Institute, “Acquisition Archetypes: Firefighting.” Pittsburgh, PA, 2007. [118] Software Engineering Institute, “Acquisition Archetypes: Longer Begets Bigger.” Pittsburgh, PA, 2009. [119] Software Engineering Institute, “Acquisition Archetypes: Everything for Everybody.” Pittsburgh, PA, 2008. [120] M. Martz and W. L. Neu, “Multi-Objective Optimization of an Autonomous Underwater Vehicle,” Oceans 2008, Vols 1-4, p. 1042–1050\r2248, 2008. [121] R. A. Conigliaro, A. A. Kerzhner, and C. J. J. Paredis, “Model-Based Optimization of a Hydraulic Backhoe using Multi-Attribute Utility Theory,” SAE International Journal o Materials and Manufacturing, vol. 2, no. 1, pp. 298–309, 2009. [122] J. Maley and J. Long, “A Natural Approach to DoDAF,” Blacksburg, VA, 2005. [123] S. Prince-James, “The Original Homunculus Company,” 2017. [Online] Available: https://www.sharonpricejames.com/the-original-homunculus-company.html Accessed Apr./20/2018. [124] J. B. Reid and D. H. Rhodes, “Applying Cause-Effect Mapping to Assess Cybersecurity Vulnerabilities in Model-Centric Acquisition Program Environments,” in Acquisition Research Symposium, 2018. [125] Office of Surveillane and Epidemiology, “The Sentinel Initiative.” Food and Drug Administration, Silver Spring, MD, 2010. [126] R. Lange and E. W. Burger, “Long-term market implications of data breaches, not,” Journal of Information Privacy and Security, vol. 13, no. 4, 2017. 150

[127] S. Overly, “IRS temporarily suspends contract with Equifax,” Politico, Arlington County, VA, Oct-2017, https://www.politico.com/story/2017/10/12/irs-equifax-contract- suspended-243732. [128] S. Braun, “OPM plans to terminate contracts with USIS,” Federal News Radio, Washington D.C., 10-Sep-2014, https://federalnewsradio.com/management/2014/09/opm- plans-to-terminate-contracts-with-usis/. [129] R. Ross, K. Dempsey, V. Y. Pillitteri, J. Jacobs, and N. Goren, “Risk Management,” NIST2, 2016. [Online] Available: https://csrc.nist.gov/projects/risk-management/risk- management-framework-(RMF)-Overview Accessed Mar./29/2018. [130] Manufacturing Extension Partnership, “DFARS Cybersecurity Requirements,” NIST, 2017. [Online] Available: https://www.nist.gov/mep/cybersecurity-resources- manufacturers/dfars800-171-compliance Accessed Mar./29/2018. [131] D. R. Heise, Causal Analysis. New York City, NY: John Wiley & Sons, Inc., 1975. [132] A. Marradi, “Classification, Typology, and Taxonomy,” Quality and Quantity, vol. 24, no. 2, pp. 129–157, 1990. [133] System Engineering Body of Knowledge, “Complexity,” 2017. [Online] Available: http://www.sebokwiki.org/wiki/Complexity Accessed May/02/2018. [134] J. H. Holland, “Complex Adaptive Systems and Spontaneous Emergence,” in Complexity and Industrial Clusters, A. Q. Curzio and M. Fortiz, Eds. New York: Physica-Verlag, 2002, pp. 25–34. [135] M. Bedau, “Downward Causation and the Autonomy of Weak Emergence,” Principia, vol. 6, no. 1, pp. 5–50, 2002. [136] E. Barnes, “Emergence and Fundamentality,” Mind, vol. 121, no. 484, pp. 873–901, 2012. [137] W. Seager, “Emergence and Efficiency,” in The Mind as a Scientific Object, 1st ed., C. Erneling and D. Johnson, Eds. New York, New York, USA: Oxford University Press, 2005, pp. 176–190. [138] M. Bjelkemyr, D. Semere, and B. Lindberg, “Definition, classification, and methodological issues of system of systems,” System of Systems Engineering—Principles and Applications, 2008. [139] M. W. Maier, “The role of modeling and simulation in system of systems development,” in Modeling and simulation support for system of systems engineering applications, L. B. Rainey and A. Tolk, Eds. Hoboken, NJ: Wiley, 2015, pp. 11–44. [140] E. M. A. Ronald, M. Sipper, and M. S. Capcar, “Design, Observation, Surprise! A Test of Emergence,” Artificial Life, vol. 5, no. 3, pp. 225–239, 1999. [141] B. E. White, “On Interpreting Scale (Or View) and Emergence in Complex Systems Engineering,” in 1st Annual IEEE Systems Conference, 2007, pp. 1–7. [142] S. Ferreira, M. Faezipour, and H. W. Corley, “Defining and Addressing the Risk of Undesirable Emergent Properties,” in IEEE International Systems Conference, 2013. [143] Y. Bar-Yam, “A Mathematical Theory of Strong Emergence Using Multiscale Variety,” Complexity, vol. 9, no. 6, pp. 15–24, 2004. 151

[144] C. Szabo, Y. M. Teo, and G. K. Chengleput, “Understanding Complex Systems: Using Interaction as a Measure of Emergence,” in Proceedings of the 2014 Winter Simulation Conference, 2014, pp. 207–218. [145] A. Kubík, “Toward a Formalization of Emergence,” Artificial Life, vol. 9, pp. 41–65, 2003. [146] N. A. Baas and C. Emmeche, “On Emergence and Explanation,” Intellectica, vol. 25, no. 2, pp. 67–83, 1997. [147] S. Sheard, “A Complexity Typology for Systems Engineering,” in INCOSE International Symposium, 2010. [148] C. B. Keating, “Emergence in System of Systems,” in System of Systems Engineering: Innovations for the 21st Century, M. Jamshidi, Ed. Hoboken, New Jersey: John Wiley & Sons Inc., 2009, pp. 169–190. [149] G. R. Mcconnell, “Emergence : Still a Challenge for the Systematic,” in INCOSE International Symposium, 2008, pp. 720–734. [150] J. Fromm, “Types and Forms of Emergence,” Arvix, 2005. [151] J. C. Mogul, “Emergent (mis)behavior vs. complex software systems,” in Proceedings of the 2006 EuroSys conference on - EuroSys ’06, 2006, vol. 40, no. 4, pp. 293–304. [152] P. J. Denning, “Thrashing: its causes and prevention,” in Proceedings of the Fall AFIPS Joint Computer Conferences, 1968, pp. 915–922. [153] B. Eckhardt, E. Ott, S. H. Strogatz, D. M. Abrams, and A. Mcrobie, “Modeling walker synchronization on the Millennium Bridge,” Physical Review E, vol. 75, 2007. [154] S. Floyd and V. Jacobson, “The synchronization of periodic routing messages,” IEEE/ACM transactions on networking, vol. 2, no. 2, pp. 122–136, 1994. [155] G. E. Reeves, “What Really Happened on Mars? The Authoritative Account,” Microsoft Research, 1997. [Online] Available: http://research.microsoft.com/en- us/um/people/mbj/Mars_Pathfinder/Authoritative_Account.html Accessed Feb./15/2017. [156] K. K. Ramakrishnan and H. Yang, “The Ethernet capture effect: Analysis and solution,” in Proceedings of the Conference on Local Computer Networks, 1994, pp. 228–240. [157] R. J. Glass, W. E. Beyeler, and K. L. Stamber, “Advanced Simulation for Analysis of Critical Infrastructure: Abstract Cascades, the Electric power grid, and Fedwire 1 Advanced Simulation for Analysis of Critical Infrastructure 1,” Albuquerque, NM, 2004. [158] D. Helbing, “Traffic and related self-driven many-particle systems,” Reviews of Modern Physics, vol. 73, no. 4, pp. 1067–1141, Dec. 2001. [159] S. Sheard and A. Mostashari, “A Framework for System Resilience Discussions,” in INCOSE International Symposium, 2008, pp. 1243–1257. [160] E. S. German, “An Investigation of Human-Model Interaction for Model-Centric Decision-Making,” Massachusetts Institute of Technology, 2017. [161] C. Alexander, S. Ishikawa, and M. Silverstein, A pattern langugage: towns, buildings, construction. Oxford University Press, 1977.

152

[162] S.-I. Tadaki et al., “Critical Density of Experimental Traffic Jam,” in Traffic and Granular Flow ’13, M. Chraibi, M. Boltes, A. Schadschneider, and A. Seyfried, Eds. New York City, NY: Springer International Publishing, 2014, pp. 505–511. [163] B. S. Kerner and P. Konhauser, “Cluster effect in initially homogeneous traffic flow,” Physical Review E, vol. 48, no. 4, pp. 2335–2338, 1993. [164] T. De Wolf and T. Holvoet, “Design Patterns for Decentralised Coordination in Self- organising Emergent Systems,” in Engineering Self-Organising Systems, S. A. Brueckner, S. Hassas, M. Jelasity, and D. Yamins, Eds. Berlin: Springer, 2007, pp. 28–49. [165] J. Buhl et al., “From Disorder to Order in Marching Locusts,” Science, vol. 312, no. 5778, pp. 1402–1046, 2006. [166] P. Dallard et al., “The London Millennium Footbridge,” The Structural Engineer, vol. 79, no. 22, pp. 17–33, 2001. [167] A. S. Dennis, Weather Modification by Cloud Seeding. New York City, NY: Academic Press, Inc., 1980. [168] B. Formby, “Once TxDOT opened up SH 161 shoulders, traffic started sailing,” The Dallas Morning News, Dallas, TX, 14-Apr-2016, http://transportationblog.dallasnews.com/2016/04/once-txdot-opened-up-sh-161- shoulders-traffic-started-sailing.html/. [169] D. A. Schon, “Generative metaphor: A perspective on problem-setting in social policy,” in Metaphor and Thought, 2nd ed., A. Ortony, Ed. New York City, NY: Cambridge University Press, 1993, pp. 137–163. [170] C. F. Kurtz and D. J. Snowden, “The New Dynamics of Strategy: Sense-making in a Complex-Complicated World,” IBM Systems Journal, vol. 42, no. 3, pp. 462–483, 2003. [171] Dictionary.com LLC, “Emerge,” Dictionary.com, 2018. [Online] Available: http://www.dictionary.com/browse/emerge Accessed Apr./30/2018. [172] R. V Solé and S. Valverde, “Spontaneous Emergence of Modularity in Cellular Networks,” Sante Fe, NM, 2007-6–13, 2007. [173] A. B. Haynes, W. R. Berry, and A. A. Gawande, “What Do We Know About the Safe Surgery Checklist Now?,” Annals of Surgery, vol. 261, no. 5, pp. 829–830, May 2015. [174] Office of the Deputy Secretary of Defense, “DoD Doctrine 5000.01: The Defense Acquisition System.” US Department of Defense, Washington D.C., 2003. [175] Office of the Under Secretary of Defense for Acquisition‚ Technlogy‚ and Logistics, “Department of Defense Acquisition Modeling and Simulation Master Plan.” US Department of Defense, Washington D.C., 2006. [176] P. Zimmerman, T. Gilbert, and F. Salvatore, “Digital engineering transformation across the Department of Defense,” The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, p. 154851291774705, 2017. [177] Office of the Under Secretary of Defense for Acquisition‚ Technology‚ and Logistics, “DoD Directive 5000.59: DoD Modeling and Simulation (M&S) Management.” US Department of Defense, Washington D.C., 2007. 153

[178] M. Hause, A. Morkevicius, and G. Bleakley, “OMG UML PROFILE FOR DODAF/MODAFTM (UPDMTM),” Object Management Group, 2018. [Online] Available: http://www.omg.org/updm/index.htm Accessed May/06/2018. [179] AcqNotes, “DoD Metadata Registry,” 2017. [Online] Available: http://acqnotes.com/acqnote/careerfields/dod-metadata-registry Accessed Apr./23/2018. [180] Under Secretary of Defense for Acquisition Technology and Logistics, “DoD Directive 5134.01.” US Department of Defense, Washington D.C., 2005. [181] Office of the Under Secretary of Defense for Acquistion‚ Technology‚ and Logistics, “DoD Doctrine 5000.70: Management of DoD Modeling and Simulation (M&S) Activities.” US Department of Defense, Washington D.C., 2012. [182] Office of the Department of Defense Chief Information Officer, “DoD Doctrine 8320.02: Sharing Data, Information, and Information Technology (IT) Services in the Department of Defense.” US Department of Defense, Washington D.C., 2013. [183] A. Mehta, “AT&L reorganization will take two years to complete,” DefenseNews, 2018. [Online] Available: https://www.defensenews.com/digital-show-dailies/reagan-defense- forum/2017/12/03/atampl-reorganization-will-take-two-years-to-complete/ Accessed May/02/2018. [184] Defense Modeling & Simulation Coordination Office, “Responsibilities & Governance,” 2017. [Online] Available: https://www.msco.mil/AboutDMSCO/ResponsibilitiesGovernance.aspx Accessed Apr./23/2018. [185] Office of the Under Secretary of Defense for Acquisition‚ Technology‚ and Logistics, “DoD Directive 5000.02: Operation of the Defense Acquisition System.” US Department of Defense, Washington D.C., 2017. [186] A. Mehta, “This is the Pentagon’s new acquisition structure,” DefenseNews, 2017. [Online] Available: https://www.defensenews.com/breaking-news/2017/08/02/this-is-the- pentagons-new-acquisition-structure/ Accessed May/02/2018. [187] NASA Office of the Chief Engineer, “NASA-STD-7009A: Standard for Models and Simulations.” National Aeronautics and Space Administration, Washington D.C., 2016. [188] NASA Office of the Chief Engineer, NASA Systems Engineering Handbook, 1st ed. Washington DC: National Aeronautics and Space Administration, 2007. [189] L. Dobuzinskis, “Modernist and postmodernist metaphors of the policy process: Control and stability vs. chaos and reflexive understanding,” Policy Sciences, vol. 25, no. 4, pp. 355–380, 1992. [190] E. S. Quade and G. M. Carter, Analysis for public decisions. Prentice Hall, 1989. [191] S. Ramos, Cure for chaos;: Fresh solutions to social problems through the systems approach. D. McKay Co, 1969. [192] N. Wiener, Cybernetics: Or Control and Communications in the Animal and the Machine. Cambridge, MA: MIT Press, 1948. [193] H. A. Simon, The sciences of the artificial. Cambridge, MA: MIT Press, 1996. 154

[194] R. de Neufville, “The emerging curriculum for engineering,” International Journal of Technology, Policy and Management, vol. 1, no. 2, pp. 117–127, 2001. [195] S. Koperniak, “MIT Institute for Data, Systems, and Society launches,” MIT News, 2015. [Online] Available: http://news.mit.edu/2015/mit-institute-data-systems-society-launches- 0701 Accessed Apr./29/2018. [196] Institute for Data, Systems, and Society and MIT Open Learning, “Institute for Data, Systems, and Society to launch new MicroMasters and PhD programs,” MIT News, 2018. [Online] Available: http://news.mit.edu/2018/mit-idss-launches-new-micromasters-and- phd-programs-0420 Accessed Apr./29/2018. [197] S. R. Wiseman, “The Criminal Justice Black Box,” Ohio State Law Journal, vol. 78, no. 2, pp. 349–401, 2017. [198] P. A. DonVito, “The essentials of a planning programming budgeting system.” The RAND Corporation, Santa Monica, CA, 1969. [199] S. Virani and T. Rust, “Using Model Based Systems Engineering in Policy Development : A Thought Paper,” in Conference on Systems Engineering Research, 2016. [200] R. Krishnan, S. Virani, and R. Gasoto, “Discovering toxic policies using MBSE constructs,” in Conference on Systems Engineering Research, 2017. [201] T. H. Karas, J. H. Moore, and L. K. Parrott, “Metaphors for Cyber Security,” Sandia Report. Sandia National Laboratories, Albuquerque, NM, 2008. [202] J. Wolff, “Cybersecurity as Metaphor: Policy and Defense Implications of Computer Security Metaphors,” in TPRC Conference, 2014. [203] S. Smith, “Science and Technology Intellectual Capital: A Critical US Asset,” vol. 59. Air University, Montogomery, AL, 2010. [204] O. of the D. of S. and S. Engineering, “Systems Engineering Guide for Systems of Systems.” US Department of Defense, Washington D.C., 2008. [205] J. Jensen, A. Tomaszczuk, and D. Hewitt, “DOD Guide on Intellectual Property Practices,” ShawPittman, Washington D.C., 2001, https://www.pillsburylaw.com/images/content/1/4/v2/1420/9AB089A2119C3FEB977818 959F54A43D.pdf. [206] T. and L. Office of the Under Secretary of Defense for Acquisition, “Intellectual Property: Navigating Through Commercial Waters.” US Department of Defense, Washington D.C., 2001 [Online] Available: https://www.acq.osd.mil/dpap/docs/intelprop.pdf Accessed Apr./22/2018. [207] 108th Congress, “Standards Development Organization and Advancement Act of 2004,” Public Law 108-237. 2004. [208] M. Aisenberg, D. Davis, and T. Kehoe, “Intellectual Property Rights for Digital Design and Manufacturing: Issues and Recommendations,” The MITRE Corporation, 2014. [209] “Trade Secret,” WEX. Legal Information Institute, Cornell Law School, Ithaca, NY. [210] R. Stim and F. Stephan, “What To Do If An NDA Is Violated,” NDAS For Free. .

155

[211] A. Layne-Farrar, A. J. Padilla, and R. Schmalensee, “Pricing Patents for Licensing in Standard-Setting Organizations: Making Sense of FRAND Commitments,” Antitrust Law Journal, vol. 74, no. 3, pp. 671–706, 2007. [212] P. Zimmerman, “Modularity and Open Systems: Meaningful Distinctions,” in NDIA Systems Engineering Conference, 2015. [213] B. Flyvbjerg, S. Holm, and S. Buhl, “Underestimating Costs in Public Works Projects: Error or Lie?,” Journal of the American Planning Association, vol. 68, no. 3, pp. 279–295, 2002. [214] L. D. Sturm, C. B. William, J. A. Camelio, J. White, and R. Parker, “Cyber-physical vulnerabilities in additive manufacturing systems: A case study attack on the .STL file with human subjects,” Journal of Manufacturing Systems, vol. 44, pp. 154–164, 2017. [215] D. Dvorak, “Model-Centric Engineering, Part 1: Introduction to Model-Based Systems Engineering.” NASA Engineering & Safety Center, Hampton, Virginia, 2017. [216] W. B. Smith, “Characteristics of Model-Based Systems Engineering.” Vitech Corporation, Orlando, FL, 2013. [217] R. Cloutier and G. Haun, “Concept Engineering Technologies to Advance Model-Based Systems Engineering,” in Systems Engineering Conference, 2012. [218] Manufacturing Extension Partnership, “Cyber Risk Management,” NIST, 2017. [Online] Available: https://www.nist.gov/mep/cyber-risk-management Accessed Mar./29/2018. [219] A. M. Ross, M. E. Fitzgerald, and D. H. Rhodes, “Game-based Learning for Systems Engineering Concepts,” in Conference on Systems Engineering Research, 2014, vol. 28, pp. 430–440. [220] P. T. Grogan, O. L. De Weck, A. M. Ross, and D. H. Rhodes, “Interactive models as a system design tool: Applications to system project management,” in Conference on Systems Engineering Research, 2015. [221] E. S. German and D. H. Rhodes, “Human-model interactivity: what can be learned from the experience of pilots with the glass cockpit?,” in Conference on Systems Engineering Research, 2016. [222] V. Ryckaert, “Hackers held patient data ransom, so Greenfield hospital system paid $50,000,” The Indianapolis Star, Indianapolis; IN, 2018, https://www.indystar.com/story/news/crime/2018/01/17/hancock-health-paid-50-000- hackers-who-encrypted-patient-files/1040079001/. [223] K. Zetter, “Why Hospitals are the Perfect Targets for Ransomware,” Wired, San Francisco, CA, Mar-2016, https://www.wired.com/2016/03/ransomware-why-hospitals- are-the-perfect-targets/. [224] V. Woollaston, “The NHS trusts and hospitals affected by the Wannacry cyberattack,” Wired, London, UK, May-2017, http://www.wired.co.uk/article/nhs-trusts-affected-by- cyber-attack. [225] J. M. Goldman, “Solving the Interoperability Challenge,” IEEE Pulse, Nov-2014, https://pulse.embs.org/november-2014/solving-interoperability-challenge/.

156

[226] M. Pajic, R. Mangharam, O. Sokolsky, D. Arney, and J. Goldman, “Model-driven safety analysis of closed-loop medical systems,” IEEE Transactions on Industrial Informatics, vol. 10, no. 1, pp. 3–16, 2014.

157