HANDBOOK OF ENGINEERING SYSTEMS DESIGN (PREPRINT VERSION) 1 Evaluating Engineering System Interventions Wester C.H. Schoonenberg, Amro M. Farid Abstract—Our modern life has grown to depend on many sciences. Over the past decades, engineering solutions and nearly ubiquitous large complex engineering systems. have evolved from engineering artifacts that have a Transportation, water distribution, electric power, natural single function, to systems of artifacts that optimize gas, healthcare, manufacturing and food supply are but a few. These engineering systems are characterized by an the delivery of a specific service, and then to engineer- intricate web of interactions within themselves but also ing systems that deliver services within a societal and between each other. Furthermore, they have a long-standing economic context. In order to understand engineering nature that means that any change requires an intervention systems, a holistic approach is required that assesses into a legacy system rather than a new “blank-slate” system their impact beyond technical performance. Engineering design. The interventions themselves are often costly with implications lasting many decades into the future. Conse- Systems are defined as: quently, when it comes to engineering system interventions, Definition 1. Engineering System [1] A class of systems there is a real need to “get it right”. This chapter discusses two types of engineering system interventions; namely those characterized by a high degree of technical complexity, that change system behavior and those that change system social intricacy, and elaborate processes, aimed at fulfill- structure. It then discusses the types of measurement that ing important functions in society. can be applied to evaluating such interventions. More specif- ically, it contrasts experimental, data-driven and model- Furthermore, there are a number of characteristics based approaches. It recognizes that only the last of these is that distinguish engineering systems from other systems. appropriate for interventions that change system structure. Engineering systems... Consequently, the chapter concludes with a taxonomy of engineering system models including graphical models, ... exist in the real world. They always have physical • quantitative structural models and quantitative behavioral components, but are also likely to contain informa- models. The chapter concludes with a discussion of promis- tional components. ing avenues for future research in the area; namely hetero- ... are artificial. Engineering systems are man-made, functional graph theory and hybrid dynamic systems. • but often integrate into the natural world. Index Terms—Engineering Systems, Intervention, Mea- ... have dynamic properties. Engineering systems sures, Evaluation, Performance, Effectiveness • change over time, and have a sense of temporality. ... have a hybrid state. The states of engineering • I. Introduction systems are usually both discrete and continuous. ... contain some human control. This chapter provides an overview of the background, • context, and theory for evaluating engineering system Some types of systems with these characteristics interventions. So far, the Engineering System Design include electric power grids, transportation systems, Handbook has provided: (1) background and motivation healthcare delivery systems, the energy-water nexus, etc. for the engineering systems approach, (2) theory for This list is far from exhaustive and the reader will find describing engineering systems, and (3) an overview of numerous other examples throughout the handbook. intervention design for engineering systems. This chap- The growth of engineering systems has been mostly ter concludes the third part of the handbook with the organic and incremental. Many of these systems have background and framework to support the evaluation been expanded or shrunk to match the changing (or of engineering system interventions. The first section perceived) needs over time. This has caused inefficiencies defines a point-of-departure for this chapter to enable and unforeseen dynamics within those systems. The the study of the chapter as an independent work. The successful implementation of engineering system inter- section also refers to other chapters in the book to ventions relies on rigorous evaluation for a future-proof provide context and other relevant material. design. A. The emergence of engineering systems B. The importance of evaluating engineering system inter- In the context of 21st century grand challenges, the ventions field of engineering systems has emerged at the in- tersection of engineering, management, and the social As the complexity of engineering systems has evolved, there is need for a deeper understanding of the de- Thayer School of Engineering, Dartmouth College, Hanover, New sign and operation of engineering systems. Interventions Hampshire, USA, [email protected] Thayer School of Engineering, Dartmouth College, Hanover, New were often designed and implemented with merely a Hampshire, USA, [email protected] theoretical understanding of their impact. During the HANDBOOK OF ENGINEERING SYSTEMS DESIGN (PREPRINT VERSION) 2 rise of the automobile, many transportation infrastruc- Placeholders for the editor. TBD. ture systems were overhauled to facilitate this new mode Why this handbook? • of transport [2]. However, the impact and outcomes of Why this chapter? such changes were often unforeseen by policy makers. • Today, we have the ability to much more accurately D. Chapter Outline evaluate and understand the impact of interventions in What is an Intervention? In order to evaluate engi- engineering systems. Increased data and computational • resources enable evaluation of interventions both after neering system interventions, first the word “inter- and before implementation [3]. vention” must be understood. Engineering systems The evaluation of interventions after implementation are most often legacy systems, and any change to the was especially useful when the computational resources system is inherently an intervention of some type. to predict the outcomes of interventions were limited. SectionII discusses systems and the di fferent types The results of previous interventions guided new in- of interventions. Evaluation Requirements. Artifacts, systems, and terventions and were eventually generalized as “rules- • of-thumb”. Furthermore, this type of evaluation also other things can be measured in one of two ways. (1) helped “tune” (or salvage) the intervention to get the Direct measurement and (2) indirect measurement. best results. The downside of this trial-and-error ap- These two types of measurement bring along their proach is obvious; sometimes the interventions do not own set of specific requirements in order to result perform as intended. The failure to perform may become into a holistic and appropriate evaluation of the en- immediately obvious, but can also materialize when the gineering system intervention. Section III discusses system has to function under extreme circumstances [4]. the the fundamentals of measurement and their An example of the latter scenario is the failure of the application to engineering systems. Comparing Evaluation Methods. Evaluation of en- electric rail system in New York City during hurricane • Sandy. The rain ahead of the hurricane flooded several gineering systems interventions requires a deep un- transformers and the power supply for the rail system derstanding of the impact of the intervention on was interrupted. As a result, the rail system failed and the outputs of the system. SectionIV provides an evacuation of lower Manhattan was severely interrupted. overview of three approaches to intervention evalu- It took a major hurricane to demonstrate the limitations ation methods. Model-Based Intervention Evaluation. Finally, the of the electrification of the rail system, whereas it had • been operated successfully for decades before. chapter concludes with a discussion around the im- In order to improve interventions before they are portance of data and systems theory in the evalua- implemented, evaluations are now often performed be- tion of engineering systems. Model-based evaluation fore they are implemented. The goal is to determine approaches leverage theory to enable intervention if the intervention will improve the system outcomes evaluation for engineering systems. SectionV pro- before a large investment is made and society has been vides an overview of some of the most important interrupted [5]. Furthermore, predictive evaluations can system modeling methods. be used to evaluate if the improvement is large enough to outweigh the downsides of the intervention. One of II. What is an Intervention? the challenges is to accurately represent the real-world This section introduces a holistic understanding of the system with computer models. When the system has meaning of interventions. This is realized by first summa- not been represented accurately, an intervention can rizing the description of a system to define a consistent work well in the simulations, but may under-perform framework. Based on this framework, for the purposes of in reality. In theory, the more extensive the model, the this chapter, a definition of interventions and a discussion more accurate the prediction, but in reality this is often around the types of interventions are provided. The type not feasible due to the financial constraints to build a of intervention is critical when making a decision about